 Hey, thank you guys for joining this session a Technical session at 4 p.m. The afternoon. I hope you guys are ready for that How's the day today? How's it talks and you guys awesome? Amazing. All right, so let me try to close it with like a good note. Yes Today we're gonna do Workshop on how to build an ebpf I see an eye plug-in that uses a bpf It's gonna be a tutorial before about an hour and a half I'm gonna try to make it actually shorter just for you guys to to rest a bit from this long day. All right You guys ready? Amazing, let's do it All right a bit by myself. My name is Adam. Sayah. I work at solo in The past couple years. I've been very focused on what we call application networking in general so I deal with API get ways and service mesh every day and Kind of the focus there is to secure the traffic and secure the network in general Now a thing. I'm one thing. I'm very much Interested in these days is how can I do more and? Instead of operating at l7 on the layer 7 words the application Layer, I want to see how many policies and how much we can get from Doing controlling the traffic way earlier on the network stack. That's basically where The CNI and a bpf operates, right? It's very much networking at a very low level so today Again, we are going to do a CNI a bpf workshop, but we are not gonna recreate solium, right? We're not going to do something anything complicated I think the goal of this session is mainly You know to solve this curiosity that everyone has right for example for me when I was a kid I always been asking my dad like hey how things work and he gave me like the the quick pitch, right? for example for us today is like opening a car and You know open like the hood and saying hey, this is the engine and this is the battery and You know that's pretty much it as long as if we can understand the kind of the underlying mechanism Being used in these technologies like solium and other a big project that's used ebpf Then it's a good. It's a good day. All right all right So let's get started What we are going to do today a couple things We're gonna start with a small presentation around. What's a CNI? Okay, very few slides Going on talking about the basics. We're not going to deep dive too much into it But at least just to get the main idea of how things work Now after that we are going to use our laptops I don't know if you guys are ready for that you guys can follow with me too to create our own first CNI After that we're gonna talk about ebpf and kind of the use cases we can use it with with the CNI We are not talking about a bit of more complexity there when we talk about monitoring and we're gonna end with How we can use ebpf for security? In our in our CNI and at the end we're just gonna have a small conclusion and any questions. All right You guys ready again? Awesome. Awesome. All right. So a quick question because I was very much when designing this workshop I was you know, it's trying to figure out the right technology to use especially for the code So just by a raise of hand here who writes goal link? All right. Good thing because we're gonna do this in bash. All right Right We I was looking for the kind of the simplest way to do it and I think bash helps a lot bash has a lot of abstractions and it begins just like use, you know, CLI's that are pretty much Simplifying our life goal link is definitely the language of choice when you try to deal with this situation So I'm use it. There's there's good documentation out there But obviously it brings its own complexity around error handling all these things today. We don't need that today We just need to understand the mechanisms. Okay All right Let's get going here. What is a CNI? a CNI is basically if you think about any Cuban Discusser Docker other projects used CNI is a way to Provide the wiring provide a wiring for any container, right? In our case here talking about cuban sis a pod, right? How can I allow a certain part to have a certain IP be connected be able to reach it be able to interact with it? That's the core thing Around what's a CNI is a CNI plug-in this in this example Now you can think about a CNI plug-in as the you know a plumber basically wire like creating the the pipes between kind of the network host At host network and and and your containers and pots. Okay, so that's definitely the easiest Way to to describe what the CNI is And now we're gonna talk about Basically the the spec a bit. Okay, the the spec of the CNI project is stable now It's it's been one zero zero for a while. It has a very well documented way of operating It's it's straightforward and they are definitely invite you guys to read this documentation and in this couple slides Which are gonna go through the basics, but again, there is couple nuances that I would like you guys to go and review after all So what the CNI project provide us? It's provide us three main things first is the specification of what the CNI is And how it should operate it provides us some example implementation that you can just use and reuse and At the end also provide us some libraries to kind of simplify our life We don't have to recreate the reinvent the wheel every single time. Okay, and obviously today We are reinventing the wheel just for an additional educational purpose all right So if we zoom in a bit of how actually things operate If there is a couple steps Okay, so there's always this what we call a container runtime and you can you know, there's When you use different technology means different things and if you talk about curators that's probably cubelets That's needs to Create, you know, it needs to create pod needs it needs to wire it So the way it works is you have this container runtime that needs to interact with something To wire this pause that got created in this case. We are using a CNI plug-in for this the container runtime is Going to do first is gonna go and read a certain configuration We're gonna go into this detail in a bit right right now. We don't have to very much Focus on this but now that the CNI rent on the container runtime. Sorry In this case coblets coblets for for communities will go and look for a network configuration Okay, it's a file. It's just a JSON file that describes certain things that we need to pass to a CNI And then it's gonna actually call a binary just have been passing certain environment variables and that network configuration and Expects couple things from that CNI plug-in. So the CNI plug-in gonna receive this configuration It can be invoked with certain environment environment variables the CNI plug-in based on this information gonna do What's the CNI plug-in is doing? You know creating the interface wiring everything and then returns a result That goes back to the container runtime to say well, this was a success. My container now is wired Okay in a nutshell. That's where it is. It's basically a configuration file a binary and Something that calls the binary with this configuration file. Okay All right So let's take a look at the configuration file and this is a very simple one Every CNI configuration file has couple couple fields certain are mandatory and certain are optional Definitely the thing that is very important here is the CNI version again I said there is multiple Specifications kind of the last one is one zero zero. That's the one we're using today But there's also other ones like, you know, three oh, oh four oh like zero. Sorry zero four zero and zero three zero and You know old versions Then there is the type the type is in our case if I want to simplify it is the binary name That's we are gonna invoke right in this case. It's called bridge Then we have certain keys that are called like a well-known keys Right well-known keys like in my example here ipam and and DNS This well-known keys are can be used by CNI to invoke another CNI plug-in Right. That's what you call like delegation. For example, I don't want to go too much in in in detail about this subject We're not gonna use it today But for example, if you are invoking a CNI plug-in that needs to basically assign a certain IP Sometimes you want to use another CNI plug-in that is much actually very focused on actually IP generation, right? It can be host vocal. It can be DHCP. It can be other other processes But again, that's just a small parenthesis here and again There is also this field that's our cold capabilities and this capabilities are Dynamic fields that are being that when when the container runtime invokes the CNI It dynamically change this field with certain values again. This is probably made too much detail right now We're not gonna use that today, but just to know that's exist there And I think the last one which is important in our example is that you can pass custom keys Custom key value keys. So it's gonna be any detail you want to pass to your CNI plug-in We're gonna use this today in our workshop All right, so Now let's talk about I talked about, you know the configuration file I said it's a binary we need to invoke the binary with the configuration file But I also mentioned that when we have a successful call to the CNI plug-in It returns a certain result That's the container runtime reads and be able to process and obviously update itself the container runtime Probably just you know print that at in STD out It's kind of a format here in Jason that the find the IP is the founder out. Also, that's all part of the CNI Spec and we don't have to really much dig into it here today right now But just know that there is a result and the result is in our example here for the bridge Is to return the IP that's been assigned to the certain container or a certain pod if we talk about communities All right One last quick step here. Let's talk about execution, okay execution of the CNI plug-in is Pretty again pretty straightforward. We have the container runtime They're gonna go and read a certain configuration that is by default put under slash ETC CNI net d and the name of the configuration It's gonna read that configuration and then invoke the CNI you know binary Based on that configuration that is under by default is under slash OPT CNI bin and the binary itself So in this case and it passes again, it's execute and pass certain environmental and environment variables, okay and the environment variables. We're gonna see that in a second here kind of Just a quick intro to things we're gonna use later Every invocation of the CNI plug-in has again It's invoking the binary and passing environment variables This environment variables can are the CNI command. Basically. Are we trying to add the container? Are we trying to delete a container? I was just checking if the container is good Then we're passing a container the CNI container IDs basically In communities, that's the container ID itself It's just a reference to the pod that needs to be set We are passing a CNI net NS and we're gonna get to this later that's basically to you know a way for the for for us to control where this Network namespace and I'm gonna talk about network namespace in a second where the network namespace is gonna be created a CNI if name is the interface like you know the network interface that the container runtime is expecting from the CNI to create on a specific container or pod in this case and the CNI args are all the values that we put in our configuration earlier that can be passed down to our CNI and CNI path is optional here But again if basically the CNI is not on the default path then you have to look for where is the you know the binaries All right one last thing here CNI's can you can use multiple CNI's right? We can use multiple CNI's you have to use only one so sometimes you have different use cases Yeah, you probably have an interface CNI basically the one that creates the wiring to create the interface itself but sometimes you have another CNI that just do some sort of tuning or Whatever it is right and sometimes you want to split this logic into multiple plugins in our case here CNI one, you know, let's say CNI 2 is invoked and you get the previous results From the execution of CNI 1 passed to the plug-in 2 and to plug-in 3 and again if you have 10 is going to operate the same You know, that's kind of a way to say don't think that the CNI is just a replacement You have to like completely replace the CNI you have running sometimes it's complement. They complement it themselves all right Time to start our first CNI plug-in Girls you guys all have like access to a laptop and again you guys can follow with me here But if you have a laptop and you want to do this lab like run it on your own environments That's that'd be great all right The exercise today what we are going to do is we are going to create a bridge plug-in This is a very simple example and again, there is multiple ways of creating a CNI There is you know in term of like Defining an interface. There's a point-to-point that like villain There's multiple ones, but the simplest one I think to instant the basic of CNI is the bridge plug-in So here's what we're gonna do in this session Actually before I get there if anyone wants to run this workshop on their laptop Right go on This link okay, I'm gonna pause there for a second for everyone to to copy it It it's pretty simple is just bit dot Lee slash eBPF CNI Okay, that will take you to our lab environment that we're gonna use for the exercise Are you guys good? Just let me know if you guys all okay all right awesome Okay, cool I'm gonna go back to let me actually start my environment too So because it takes about two minutes to be ready click on that just in the meantime and If you click on start you should get on this page here where it tells you that the challenges is loading and Yeah, let's talk about what we are going to do today Let's create a bridge CNI plug-in What we are going to do is Define the CNI plug-in that first creates a bridge if it's not existing We're gonna call it bridge zero We're gonna assign an IP to the bridge right the bridge is connecting is connected with the You know host network, so that's how we get the connectivity to the host and Then the bridge we're gonna define what we call V8 pairs One V8 pair is like you can think like a cord, you know So basically you're gonna put one end plugged to the bridge and the other end plugged to the container Very simple again have a bridge connected to the host network I have a V8 pair one attached to the bridge one attached to the container Again, that's what we're gonna do So let's go back to this one probably take about a minute here Yeah, so in the meantime since I have a minute here to talk about things Again, we are gonna do everything here today in bash for simplicity and we're taking a lot of abstractions but the good thing is We are going like if you want to do this for real like and just use like the right technologies There is multiple projects. For example, there is the selium ebpf Project that has good examples of like how you how you can define actually, you know the epf integration later on There is the Container network, you know networking Repo and it has a lot of good examples if you go here on The plugins that are provided actually the bridge one We're gonna do right now in bash or the exist in golang and you can just go and call plate right away Okay Let's go back here to instruct Sticking about 20 seconds If you know it has a good joke No, okay So instruct is pretty easy here. You don't have you don't have to install anything what we're gonna need is Basically this tab here. It's gonna have a panel on the left where we're gonna have all the instructions And then we're gonna have a terminal on the Sorry a terminal on the right and we have all the instruction No way saying the terminal is in the left. Sorry in instruction are on the right And it's probably just a copy-paste exercise at this point. So quick question here. Who has this the environments ready? Wow, you guys faster than me here Okay, we'll wait for me, right? Okay, keep going anytime soon. Do you guys see the the terminal? Okay, good Interesting. I'm gonna give it another try on different tab. Ah, there you go. Okay. It's been patient Okay, there you go. Let's go. Let's start So once we are going today to do today is it's pretty straightforward We are going to first Create a Kubernetes cluster a single node Kubernetes cluster that doesn't have anything much. It doesn't have any CNI Okay, for that we're gonna use cube EDM. It's it's it's very simple Kubernetes cluster here again basics so To use this platform is straightforward go and click on the command Right click on whatever you want to copy and then go to the terminal and paste all right first step is creating a Kubernetes cluster Without any CNI installed it's taking a couple seconds to get it ready and once done Now we have a current discusser. I can run the second command here. It's just to basically wire You know the cube config so now I can run commands like qtl get pod If I run the qtl get pod command You guys can see from back there Good. All right. So if I run the qtl Command I See here that I have couple pods that are running but I have two that are pending Right. So the core ones the core community spots are started. That's fine But anything that is need that needs a CNI to operate. It's impending, right? That's because we don't have any CNI installed at this point So even if you start any like hell world or you know, HTTP bin or whatever service It will not work at this point So let's let's fix that. Let's actually create our first CNI plug-in The first step we're gonna do here is to copy the skeleton of our CNI plug-in. Let's copy it and Let's open it on this tab So cop copy the command like around the command copy and then let's go to the editor here Press press refresh if you don't see it right away, you know, sometimes it takes a couple a couple times to to pop up Now that you click on the file and let's just read the first, you know What's this is the kind of the basics of our CNI plug-in? It's just a bash file and it has couple command that we need to do So we are reading this environment variable CNI command remember remember, right? The the container and time execute the binary and pass certain environment variable as part of this Environment variable we have the CNI command what we want to do in this case. There's multiple operations Actually, I'm missing one here. That doesn't really matter But in our case we want to focus on adding a pod and deleting a pod Right. So in this case We have case add delete check and version All right. This is very basic Let's keep going here and now let's implement the ad function in our CNI plug-in so the ad function is Going to do couple things The ad function is going to grab, you know First we're going to get the CRDR of the part of the of the node Like what's kind of what's kind of IPs are allowed on a certain node Then we are going to create Again a bridge like we said and then we're going to create the v8 pair One plugged to the bridge one plugged to container And then we're going to return a certain result saying well everything's done. Everything's good So let's run this command here We're going to copy the the step to add Going back to the terminal run this And now if you go to the editor just again refresh the page if you don't see the new the new code and let's go Line by line here trying to understand what we did again. This is bash. It's very much simplified for a lot of things Right didn't go if you want to do it, right? The first thing is to We are gonna pass right. We're gonna pass the pod CIDR Using the remember the configuration Jason configuration We are gonna pass in that Jason configuration the pot CRDR of the node So every single node has a pot CIDR. We're gonna pass that to the CNI plug-in Then we are going to create a bridge We are using again. Just see all I commands to do a lot of things in this case We are creating a bridge called bridge zero. Sorry. I'm just gonna remove this so Easy to read Okay, there you go. Yeah, so the first step again is creating the bridge if it doesn't exist In this case, it's called bridge zero Then we are assigning like a gateway IP and IP itself to the bridge, right? So the step here is is to assign the IP and Then we are going where we need to figure out the IP of a container, right? Again, this is very very bad code. You don't do that in general, right? Do not go and do a random to figure out what kind of IP, you know, we have But you know, you probably use the HCP instead and so on in this case We are running this command to figure out. Let's grab an IP Let's grab an IP without within the CRDR CIDR range that we passed in the configuration So we are doing a random between two and two fifty five. Okay, basically an IP there We're gonna grab this high IP put it in the variable called IP Again, remember the steps are pretty easy first create the bridge define an IP for the container Create the V8 pair plug one in the bridge plug one in the container All right, so we did we did the two first ones we created the bridge and we generated an IP randomly The step after that is to basically create the V8 pair To create the V8 pair we are using IP link add where with One the one side of the the V8 pair is the CNI if name that's been passed Through the execution of the CNI plugin again, that's configuration coming from the container runtime And The other the other end is basically the name of the interface that we are going to create for this specific pod That's the one this in this new interface is the one that we're gonna be plugged to the bridge So here's the V8 pair now. We have our extension cord. All right, but it's not plugged yet We just have that What we are going to do is to plug One end in the bridge. There you go. I pull link set The end of one of the V8 pairs is now plugged in in the bridge zero Now we have the other end of our extension cord that we need to log in the container itself so what we are going to do is to define the Contain the network namespace To do this we're gonna do a couple a couple commands here, but long story short. We are defining our Network namespace that we called in in, you know, we're just using a link here to container a CNI It's gonna be slash var run net ns slash container CNI container ID. That's basically our network namespace being defined Then what we are going to do is to basically go and Put that other end of an extension cord Within this network namespace that we just defined Okay, we're gonna plug it there The second thing we're gonna define the IP right important. We haven't used the IP yet Right now we're using the IP the one we generated randomly. We are defining that putting that in the Our network namespace so we did the two things we again We plug the other end to the container we define the IP the last thing is to define the default gateway that basically just to Carry all the traffic that goes within the network namespace out and that through the default Gatsway that which is defined we define in the bridge, right? That's kind of the way to say this is how you route traffic So once we did this steps, right we did again Probably repeating myself But just to understand the concept We created the bridge if not existing we define an IP we created the v8 pair and we plugged the both ends Now that you know you mean like we mean it's it's it's good. We are done now. We define connectivity to our container Let's report this success To our container runtime the container runtime now will get resolved to say oh, you know what that's good that's good for me and We are using obviously the net the CNI specification format to return certain results What we are going to use here is we're going to return the Certain information for example the IP of this of this specific we're going to return this IP of this specific Container that we generated we're going to return the MAC address and so on okay, which gateway is being used and so on and It's again the CNI spec is very much Very flexible and the way it operates is it's captured They you know, we don't even have to return to certain way just print it in sd out. That's all we need to do So we're doing a print right now That's the result that the container I the container runtime will capture and say hey well now that's a success You see very basic again. This is a very basic container. Yeah Good question. Good question. So the question is How we got the pot CRDR and why it's not part of as you know Part of the environment variable The the the pot CRDR. This is not part of the spec, right? This is this is basically the spec defines the way to wire a certain part It doesn't do it in the context of Kubernetes The pot CRDR in this case is very much tied to the to the pot CRDR of a node and That's why we have to pass it through the configuration and not through an environment variable. That's basically the result. Okay. Awesome. So Good the last thing we have to do here is to define the Dell that the leads command and again It's gonna see it's very hard Very complicated code to add the deletes So copy the delete code refresh There you go. Very complicated code Okay, that's all we have to do to delete A network namespace in this in this case, you know, removing the part It's just gonna go and remove the certain network namespace on that path. Very simple We don't have to define the check the version We're just gonna return basically which kind of CNI version we're using in this case Which kind of spec and is in this case We're using one oh, but this code is compatible with multiple versions all the you know We are no we're not doing anything complicated. So basically we have very much compatibility with everything Amazing you guys ready to see that in action. I don't want to see some excitement All right Okay, so before we get we get there Let's actually talk about the good point that our friend ask here is basically how to how to pass this this Pot CIDR Information in this case. I'm I'm putting that directly here 10 10 0 0 slash 24 Why just because I also control the pot CIDR during the installation self-criminators, right? We cannot I I passed the same version so I kind of guessed what kind of version I need in my CNI this is not common This is not in real world that doesn't work like this in the real world You'll need to create a certain process that goes and read, you know Calls the cube API get the get the pot. It's gonna need to do this look if if I take a look at my node It's very small. Actually, I think it should zoom in So if I do Maybe I'm in more So if I do K get node, so K for kubectl get node And take a look at what I have I have only one node again There's a single node currencies cluster in this case and if take a look and do a dash O YAML Then I'll see like if I scroll up There you go, you see the pot CIDR that's the one I actually just Put statically in my configuration file that I'm gonna use as an example But this process is not like that, you know, if you want to use a real use case You want to create a mechanism that goes and redistributes API get this I get this address Put it in the template of the come off the configuration file that gonna be set on the host so Going back to our example So we have the binary now. We good there. We're gonna create the configuration file It's very basic. You see the configuration file for us in the network configuration Very very basic. We are just putting the CNI version We're going to gonna put the type which is again, it's the binary name in our case And we are putting the this pot CIDR that there's just showed you right now through Kubernetes, right? That's it All right, let's let's let's try it If we run this So in this case, I just created two more pots Okay, just to demonstrate that this thing is broken still very much. So right now I have two pods Impending my HTTP bin Service service our pod here is impending the sleep one two and this coordinates once but basically Yeah, I don't have a CNI yet. Let's fix this What you have to do is just to copy again. I talked about kind of the default where you need to put the configuration We're gonna put our binary our bash script. It was just created under slash OPT CNI bin and I'm gonna put the configuration that we just wrote right now again under the default slash ETC CNI net D Let's do this Wait for it and then give it a couple seconds and And A couple seconds how long and I have to wait here Why should probably do that? I don't think it's aired Can't control old, you know control all the code here. What would they Did I wait saying so what do we have here we did the modification of the part Take a look. Oh There you go. What's going on here? That do I have like This let's just force the deletion just to kind of force cube. Where's that get what is that sir after the copy? Yeah, I mean, I did the copy. I don't know. Let's see. I'm just gonna force the recreation of the pods Pending still what's going on? Let's see. Sorry. Let's me double check. Okay describe Yeah Okay, okay, okay, okay. Why this is not I Did I put that? Yeah, it was intated anyway. That's crazy No, I needed to to remove the taint for me to schedule the pods on same on the master node But then the taint are not there anymore because it's schedule actually schedule the two pods Where it's going on here. What's going on? Let's see. Let's see. Let's see. Did I miss something else? Oops, where's that the nose already? Oh, yeah Yeah, I mean the node is not ready because the CNI is not there once the CNI is installed is gonna put it back to ready Okay, what's what's happening here? Yeah, I did. Okay, look, let's do this. So not block on this I'm gonna get get to that in a second I'm gonna go back to do the command quickly just creating a new environments quick, you know Do not be blocked on like try to debug this life because it's not gonna happen But question do you guys have the pods container running? Okay, that's what matters. I did that already mental times So what we are going to do. I think the way I'm gonna do it to kind of simplify this now the pods are running and all this That's awesome. So What I'm gonna do this quickly is to create Let's go back to my exercise I'm just gonna oops I'm gonna start that on Here and I'm gonna catch up. Don't worry. We'll figure this out I have to dime the should not debug life doesn't work All right, let's go back to this. Okay. Awesome. So again, let's me know how many ones have that thing running All right, that makes me happy. That's all I matters So what we are going to do next is to talk about the next step from there, okay? So now we have the the CNI being installed the notice the note is is ready to ready to take Traffic we have the you know, we have the pods networking figured out. That's great The step from there is okay. Let's talk about eppf now. Let's talk about how we can improve certain things using eppf So in our example, we did something very basic. We just created the interface but what if I want to control this interface to for example Monitor all the traffic going through or I want to secure the traffic that goes through that interface, right? For this we're going to use eppf by raise of hand here. Who knows about the ppf in general? Okay, great. Great. So who wrote any eppf code before? was it Yes, it's by definition. It's okay. So we have a couple ones. Awesome. All right. So again, this is going to be very very very Basic code eppf code. So don't be scared. We have to write some code, but it's gonna be very very straightforward So eppf what it is We can think about eppf and probably gonna skip this because I think it matters more to talk about this eppf is you can think about it as like a VM within the kernel Where you can run some code that is very much, you know in the sandbox that can do things for you Like it can attach to certain hooks. So there's hooks for everything. There's hooks for networking. There's hooks for like file access Linux define a lot of hooks for different things and eppf in our case and here we're gonna talk about networking mostly We can attach a certain code Watching for a certain hook and do something that's kind of the three part of of what we can do in eppf Now the do something do something can be Monitor the traffic. It can be secure the traffic. It can be the tech anomalies. It can be a lot of things like this so Here is an example of enough, you know, if you look at the at the stack There's multiple hooks. Oops. I'm gonna put it on the slideshow to go there If you look at there's multiple hooks, you know, you can there's hooks under The lowest level is xdp Okay from there, there's hooks that goes to traffic control and there's all the way up some some bpf Code that can watch for syscalls Who knows here was xdp or heard of before? Okay couple ones. All right Xdp you can think about it as a way like it's called express data path. It's an eppf You know, it's an eppf that allows you to Basically like connect to the lowest level level of your networking. Okay, very very low level the thing because it's low level it is doesn't have much access to a lot of the data of The traffic, but it's very fast. This is kind of the Down the pros and cons the xdp allows us to do traffic shifting fast Rewriting a full packet like for example, I want to load balance from from different points. I want to detect anomalies super fast I want to restrict traffic very fast. You know, I want to I want to build like let's say Firewalls or someone xdp is very good for that TC is a little bit up the stack It's actually have more data around the packets So you can do like, you know packets mingling we can change ports we can you know change the queue We can there's different things that we can do at the at the TC level if you look at psyllium actually psyllium Uses both psyllium would use certain parts of the of the system in you know using TC For control of like probably especially routing on on ingress because TC can be routed Routing on on on ingress where where xdp can only be on ingress ingress xdp allow you to kind of do you know load balancing again at low level in comparison If you use psyllium using xdp Here here's a You know an interesting diagram I got from the psyllium documentation Where it shows you that if you use psyllium with xdp the amount of free data like free cpu Is huge compared to like the basic queue proxy That is basically saying that's using xdp is way faster more efficient in terms of like processing of the traffic So let's go back to my Environment fast and see if I can catch up with you guys So I'm going to go here. I'm going to create my Up All right, hopefully this time works and I create my environment Taking time here great Okay There you go. Now I'm going to connect my my my uh cube config I'm going to skip straight to the final version Which is del I'm going to create my EPPF configuration Uh, sorry my my cni plugin configuration I'm going to deploy certain data like a certain application And tainted and tinted the node for master so I can schedule things on it And then I'm going to move my I'm going to move My cni configuration here There you go container creating whoo. I don't know what the last time I didn't work All right. I don't know like sometimes you do that thousand times before It works. You have to do it in front of people breaks, right? I guess that's the deal. All right Okay, so all the pods are running amazing so I can go to Basically take a look at the configuration So, yeah, if I do an exact If I do a curl I'm getting results from htp bin So basically this sleep pod this sleep pod think about as the client And htp bin has been the server. So just to simulate client server connectivity Awesome. There you go. Good. Let's keep going. So Let's go back to the slides quickly just to explain what we're going to do in the next step Here's what we're going to do. So we created our bridge plugin also Now what we want to do every time we're going to create the v8 pair and every time we're going to create the interface in our container We want to plug in and next like an ebpf code That's it. That's the goal for this first exercise. Let's write our first ebpf code very simple So in our case Or again, we're going to attach every single time ebpf at every single interface we create Let's go back to our environment Let's get started here So let's take a look at a very basic BPF code You can open the editor here and let's go through it In this example in my in all this workshop. I'm using xdp We could have been using tc because more cilium related, but let me xdp too But xdp the the good thing here is That is very simple. You can actually go online. There's a lot of xdp Information, you know, there's a lot of tutorials I can give you links at the end of this workshop to go and learn more about it So I think xdp is the right technology to first play with with ebpf in the networking space So here's the code that we are going to create You know ignore the includes that's actually we're using this library called live bpf that allows us to You know, you know, write our bpf code But the main thing that we need to be you know watching for is so This the section in our case we are putting xdp the section is defining which kind of hook we are looking for Okay, so in this case The section is saying well We are going to trigger this on xdp on the xdp hook in our network stack Then every time we we get there we're gonna process this method here called process xdp You can call it whatever it doesn't matter the name here doesn't really matter I think the section matters more And then every time again, I talked about the fact that every layer has different like More information and since it's very low level In in xdp we have we have access to this structure called xdp md So what we are going to do is straightforward The first step is we're gonna get the data so every time the packets go through this v8 Virtual like every time we we capture these packets We are going to do a couple things so we captured the data, which is the packets We're going to capture the beginning and the end right Then we're going to try to parse this data As if it's like an ethernet You know packets We're going to make sure that basically the data itself is Not bigger than the ethernet packet Basically if that happens that means that's a that's a bad packet. That's actually something you don't need on your system That's the first filtering basically you are filtering bad traffic going through your services That's what we're doing here xdp report Abort the way next dp the way it works There is codes for everything if you return a certain value It's instruct the xdp code or ebp of code to do something In our case here xdp abort just You know Stopped like it doesn't process this it doesn't process this this code It's not going to filter out. It's just going to not run on this on this particular code Then the second the second thing is We want to check actually if this is like an ethernet, you know ipv ipv4 Code like are we looking for an ipv4 structure and for that we're using you know, we're just Checking the proto on ours on our code and saying well does it match there's a predefined proto for you know value for for an ipv4 Packets we're gonna check that. Okay. That's good. If that's a packet that is ipv4 Then it's not ipv4 then just pass xdp pass says it means just like Follow don't do anything right. We don't need to do anything right now. Just Forward this packet. It's not I don't have to deal with it Now if it's if it's actually an ipv4 packet, then we can get more data And we'll follow it here. We're forwarding to we we're actually going to parse it into an ipv4 structure Right as part of this ipv4 structure first. We're gonna check. Hey, is it actually an ipv4 structure here? Right if it's not so if it's the size of it is too big Then well again, too, I don't have to run xdp in this case a board But now again if it's actually a valid packet an ipv4 packet that we parse it correctly Okay, so now I have access to the source and and destination address all the things that get that comes with the ipv4 Structure so what I can do look I can print in the kernel Right, that's the thing here. We're printing under kernel this source and the destination address We're gonna say hey, I got a packet from This address and I get the this packet is going to this address and let's print that again on the kernel side And that's it. Let's actually run this code Let's go back to the terminal Let's actually just Pull the dependency which is a little bpf in our case And then let's actually build this binary. We're building this binary using clang Which is actually our our cbuilder in this case. It's it built it into a bpf code Okay at this point we have our first bpf code compiled. We good. Let's actually now use it on our interface What to do this we we actually need to load We have to load a specific bpf program The binary that's got outputted for the previous command is now loaded under sys fs bpf bpf cni Okay, so again at this point. I have some eppf code that can be used I haven't done the wiring in my cni to every time I create an interface Attached this code to it not yet. We're gonna do this in this section So let's run this command Let's get to the editor to take a look at what we did No, sorry on the editor tab, which is a cni plugin Do some refresh. It's the exact same code as earlier. The only difference The only difference is here is after I actually output my results. I'm saying, okay, everything is good Well, actually wait a second before we finish this let's actually attach this bpf code that I just loaded And we are using bpf tool net attach xdp on the Like on the interface interface. Sorry. We just created. Okay. This command here is basically saying Run this xdp code every on every packet that goes through this interface the network interface we created for the networking in our pod Again, it seems maybe complicated, but it's very straightforward. We wrote some code We built it We loaded it and then we attached to a specific interface In this case, I'm using bpf tool for this but again If you want to do this in the real scenario And for example cilium the way they do it and other projects. There's different loaders There's actually some code it like can write it and go for example that grabs, you know They're gonna grab this this bpf code and attach it to a certain interface You know in go instead of like using a cli like I'm doing cli here is just for simplifying things Awesome. So now I'm modified my cni code. That's great. So what I'm going to do is I'm going to Yes, I'm going to basically Update you know, I'm moving this cni code to opt cni bin again when we have the cni previously So I'm overriding the old code And let's just delete the pod We're going to delete the pod just to force the recreation of the the the new network interfaces The new pods with this cni code cni plugin code attached to it So, okay, my two pods are now up and running which is awesome. So if I run, let's say I'm going to run some some traffic There you go. So I'm making a curl. What do you guys think would happen now? Now that they're on the code the package went through the v8 My xdp code it got executed and I should have logs now in the kernel that shows that this packet got processed Let's take a look We are reading the kernel logs There you go We see curl Got packets from something to something Okay, there you go our first ebpf code It just printed something it just got automatically using our cni We not only created all the networking things all the networking component for the the plugin for sorry for the the pod to be Wired correctly so we can interact with it, but we also loaded an xdp code For us to do stuff on it, right? We haven't done yet nothing. We just logged some some The the the source in the destination address, but now let's do some interesting things All right, so the next section the next exercise Is to use ebpf for different scenarios ebpf again can be used for Multiple things the main three things that we can think about in Networking is ebpf can be used for monitoring So you can have like metrics and and and stats around your traffic It can be used for security. So to block You know calls that you don't want or traffic from a to b, you know without You know interrupting basically packet that are not meant to be sent to from a to b for example and the third aspect Or or rate limiting actually or a lot of controls basically can do in security And the third aspect is routing routing for example in in selium. I showed you you know the data earlier Selium use it as a replacement of cube proxy. You don't need actually to use ipvs or You know ip tables for routing ip tables being they got created for firewalling not actually for traffic routing You can use ebpf and xtp in this case to get way faster Traffic routing in our example in today's workshop begin out probably focus on the two things monitoring and we're going to focus on Security like a small example So Before we start talking about Before we start talking about monitoring in details actually something we haven't talked about yet is the the You know, I talked about apbf running in the sandbox in the kernel code But you cannot get data out of that, you know Kernel is basically in isolation in our system There's no way to interact directly to get metrics or whatever In our user space user space is basically everything you run on on on on your node everything you create that user space Now a user space program cannot interact directly with the kernel space program They the way they interact they the way they interact and interchange data is to use what we call a bpf map A bpf map like it says here It's just a map a certain type of map you can do like a ring buffer For streaming data, you can do hash maps for key value storage You can do arrays for just like saving a random number of things Basically, you define a map and then now the kernel space the ebpf program that we wrote can write to that Can write to that map and actually the user space program can also write to that map Read write in both directions like the user space program can read and write to the map and same thing for the kernel space program So what we are going to do in the next exercise We created already our bpf code. We just demoed a log to the kernel Now let's do something a bit more interesting What we are going to do is a couple steps. We are going to capture the traffic. I mean we did that earlier same thing We are going to basically write Write the number of packets that we receive from a certain source In the map we're going to create basically a map first before all that But then we're going to create the the number of packets we receive from a certain source in a map And then we're going to use what we call a user space program to go and read a certain map And expose that as metrics that we can see in prometheus Okay, that's basically full cycle around monitoring And all that need to be loaded using our cni So let's go let's do it Um, so the first step here To click on next all right Man who's tired here? Oh, I mean 5 30 in the afternoon I get you all right, so Let's create the map The map is pretty straightforward What we have to do is define a structure in a certain way to find the values We need to store in and label it a certain way and that basically what's created a map So in our case here, we go back to the editor Take a look at the code There you go. This is the new code of basically the monitoring So what we do first is defining the structure That is basically defining the key value and we give it a name. We call that counter That's our map name. It's called counter. Okay We define that is basically storing a key size of u 32 and a value size of u 64 That's it think about it as a hash map has a key value pair Where the key is a certain size the values are certain size very simple Now same thing as all our code from earlier, so we don't really care about this The only thing actually we care about is after Now at this point since we actually have the source address of a packet Let's put that in the value and call that source ip key So the key is the source Of a certain packet so the ip address of a certain packet And every time we basically we're gonna do we're gonna run this this Method here that comes with lib bpf again. I'm using like this library called lib bpf They're gonna give us access to the map so you can just call The bpf map and look for the key if this key which is the source address exists Just increment it just say okay. Well, I got one more packets another packets every time I see the key Increment the key increment the value If it doesn't actually exist Well, just say hey, I received my first packets Right again if else very basic here and then we do return xdp pass with mean which means Forward this packet. Let it go. Let it let it continue on the networking here So we're gonna do the same exact step we did earlier. So we're gonna build our We're gonna build our Yeah, so we built our bpf code and then we Loaded it again. So we kind of did a reload here And we don't have to modify our cni because it's actually pointing the same value slash sys fs bpf ebpf cni All right, so Let's delete the pods just for us to force them to Restart with this new code which is created Take about a second here to to to create the new pods for us And we're gonna see what happens. All right, so the two pods being recreated now. Let's send let's send the packets Let's send some traffic So we're doing i'm gonna do a curl call from the sleep pod to the htp bin pod There you go. I'm getting some results. I can run this couple times. It doesn't really matter. Oops I actually deleted my my pods, but yeah, I have to Oops Where's my curl? Yeah, so let's do this Right I'm getting Some returns now if I use this bpf tool map That's gonna dump the values under the counter again. I created the map. I called it counter. Okay, so it's automatically now loaded If I run this this command There you go. I see now Two elements two values in my Bpf in my in my bpf map So every time i'm running some traffic within my v8 pair now a certain value is getting incremented in my counter You saw the power here basically we did Pretty much nothing to get certain metrics out of ebpf Actually now let's use what we call a user space program All right user space program again is is the way for us to interact with ebpf We are going to use this cool project. I found online called ebpf exporter Right, it's um, it's it's it's on on github. It allows us to automatically go and read an ebpf map We don't have to write any code So this is running. So if I do this curl local host There you go. See that Now automatically now I have prometheus metrics That is representing the bpf code the bbf metrics as as prometheus metrics and actually I can even use You know prometheus here to capture that There you go And if I restart prometheus Now if I go To Prometheus, right? Let's take a look at what we have now we can do Let's see Counter Let's see what we have here. I think it's called You see it Do you put the metric name here? Okay, I do things right sometimes. I just don't realize It just that okay, great. So you see here we have the two metrics Coming back and awesome. So here if you do like in there, I don't know like five minutes and take a look look You see that We built a cni plugin and we built some epf code that go and counts packets automatically load that And now we can see that's on on In prometheus. We did that in you know, let's say a session. So basically On epf with epf we can do things Very low level super fast. You can think about that think about things you can do here Okay, here. It was a very basic example But now you can write the ccode to do probably Anything you want with the packets And talk about anything we want. Let's talk about the next example. So we talk about monitoring Let's see what we can do next and what we couldn't do next is Basically security right so Who here use network policies? I pretty much a lot of people all right Network policies is a fundamental component of You know networking in communities. It's allowed us to Deny traffic from a to b and do a lot of fine tuning defining multi-tenancy and all of things network policies in if you want to use epf for that you Probably Would use something similar to what i'm showing right now If you want to implement that let's say from scratch and again, this is a very very basic scenario So what we are going to do for the next example is spray straightforward We are going to first create We're going to create our network policy in communities. Okay Just a super basic one simple Just for for demo. We're going to create a user space program to kind of read this network policy And actually go and create certain rules in like our bpf map Then our ebpf Code on the kernel side going to read these rules from the map and enforce something That actually if you think about communities and network policies enforcement using ebpf In a nutshell, that's how it works. Now. This is very much simplifying a lot. But yeah, you have a process They're going to watch for communities policies It's going to transform it into data that can be pushed into a bpf map on the other side You have an ebpf code that reads from this map and enforce the policy So let's do it. Let's see how this would work in communities or actually in our example here So, all right, let's keep going here. Let's get to our example. Click on next All right, let's start So for this example, we're defining a new map. We're calling that ip rules It's very straightforward. We're actually putting The kind of you know a key pair. That's the the bit more complicated thing As a key, it's not like a straightforward Like we're not using a direct like an int value or something. We're using it in a pair source destination So if we see this source in destination, that's going to be the key The value going to be a bullion like just zero or one and every time we see this key Source destination with value zero or one it will be for us if it's one allow the traffic zero Stop the traffic very simple So, yeah, let's keep going here and let's create our map Let's take a look at the ebpf code that we just updated And let's skip through kind of the first whatever we talked about when we You know, we talked about Monitoring let's focus on security. So we have this section again It's a key pair source address going to be the key. The value is bullion. It just is going to be a zero or one Now in modification of our code scroll all the way down the same logic we used to write in the map We're going to just read from the map Actually, we're going to do both what we're going to do is We're going to say hey define a key pair With the source address is the source address of the packets The destination address is the destination address of a certain packets So this is our key pair. This is our key right now And take a look at the map if you don't see any value Meaning we didn't define any rule Put one say hey, well that traffic is allowed. Put that in the map. It's fine. So for us we can see that From a to b a certain packet has been allowed Now if you see explicitly In our map that the value that we defined for this kind of interaction is zero Then just drop that packet. We don't want it. All right, let's block that Amazing that's it. That's pretty much what we need to do to define a way to enforce rules between two Like a source and destination address using ebpf So let's uh Get back to here so we Yeah, so we we updated the code. Let's build us again the same same thing we did earlier. Let's build this again and reload the new code Let's delete the pods And wait for them to be ready so we can keep going Take a couple seconds here Yeah, I mean in this this is a very simple example But yes, you can define like you can use the map as hey Well, it's our source of truth of like what should happen in my networking Right in my example, I'm populating Like one saying hey allow traffic for all the pods all the every time I see like a pair source address If I if I don't have an explicit rule Put something in the map saying hey, well, I have one means like I allowed it But if I explicitly see a zero then well, that's that's actually my but then you can structure This is an example you can structure basically your bpf map like you want It just needs to map your logic that you even have to do in your packing On xdp very very minimal because we are not We're not unpacking. It's not like we are just you know The we're just you know watching for the structure. It's not we're not we're not Modifying the packet itself, right? Like it's it's mostly like you can I mean you can modify the packet itself But when you just take a look at it in our case in this all examples, we didn't do we didn't rewrite the packets We just looked at the packets and made the decision to either monitor that and in this case We're gonna just enforce certain policy Yes, we are talking about very very xdp. It's pretty much, you know, the packet you don't you can see much from it More you go up More you can see stuff so Yeah, let's run the command here. Let's let's make a call And take take a look take a look at this map Now we have a new map that we called ip rules that has Source and destination address with certain value Basically, we are loud traffic from if you see that's split like, you know, it's kind of the same value like flipped That basically a request response we sent a request to a from sleep pod to htp bin It's a one and from htp bin to sleep when it goes back for the response. It's one two So, yes, let's actually Look at now. Let's look at a user space program that can write these rules For that I wrote something quick for for you guys to read and we can open the on the editor on the user space code I already wrote this quickly just for us to to to save time It's what what it does is very straightforward It's going to look for the binary of the bbf code that we just wrote and it's going to load it So we're going to skip all this but the interesting is it's going to get values from the kind of when we call We invoke that's that's program with three values We going to invoke it with the source address a source ip a destination ip And one or zero to say hey allow traffic or not So that's it That's all we are doing here. So and if we take a look actually I already built this for a matter of time Take a look at the binary already have it So again if we take the if we test the traffic already That would work But now let's actually put an explicit rule to stop the traffic For that i'm using the binary take a look you see ip rule This is the binary of the user space I created i'm putting the source addresses the the ip of the sleep pod The destination is the ip of http bin pod and the value i'm putting for the rule is zero meaning hey Well block this traffic Let's do this run it meaning now in our bpf map. We can take a look at the value Explicitly I see a zero here. So don't let this packet go through And now if I make a call again Voila Blocked not traffic between a to b. Okay. This is much this basically the Very simple version of like how we can enforce rules. Actually, you know what as a fun exercise I just recreated and you don't do that. Okay. That's not how it should be I actually created a bash program That just go and parse Here's the bash the the bash program I created a bash program that goes and Basically just watch for cubinetic network policies You would actually use golang and the client the cube client for that don't don't use that But in this case what i'm doing is i'm looking for any network policy and i'm actually taking a look at if if the Source and destination address are allowed then put an ip rule to allow the traffic if not deny all other traffic So by default everything's blocked but if if If this if I have a cube network policy that is explicitly allowing traffic from a to b then allow that Right, it's very simple. So if I use this code I'm going to deploy this here Basically by default is blocking all the traffic. There you go Now look, let's create this super basic network policy in communities We're defining hey the matching label actually, you know the destination label in this case is htp bin And the source address is sleep So if you if we execute this called this this uh this code again Which is actually allowing calling from sleep to Uh to htp bin it's not working right because by default i'm blocking all this traffic But if I go and copy my network policy Create that in communities See if that's got created There you go. It's got created on communities now My code is running the you know, i'm just doing a loop of like 10 seconds So it's gonna go there you go it detected the cube policy It's allowing traffic now from the the the the part that is ending with 200 and the part that's ending with 15 Which is the sleep part and the htp bin part And if I run the code again There you go We just created a very tiny tiny tiny Sillian, all right, that's basically take a community policy Um create, you know, write write a value in a map Get an ebpf code that's been attached with the cni to a first to enforce a certain security rule and with that being said This is the end of this workshop I hope you guys enjoyed it. I there's much things to talk about but All right, thank you again. This is don't recreate that I want you after this session to be able to open the hood of a car and saying this is the battery And this is the engine and you've opened like, you know You come into this cluster and look at things like, hey, well, this is probably a map and this is how things are enforced. Okay Yeah Okay, great great question. The question is is see the only language we can use for to write this program Unfortunately on the kernel side, it's c. You're gonna have to write your code in c on the kernel side for the user space Then you have more flexibility. There is uh, again If I take what I was going to show you actually this this cool project if you take a look at the cilium You know the cilium ebpf repository it has good example about like how to actually load a certain You know load a certain example like, you know kpro for example kpro will have the The back end code and you're gonna have some You know, let's say user space code leave ebpf to The rust can be an option. There is certain technologies that now is kind of doing working on What we call binding right on the user space, but the back end It's still it's still gonna be c like a I mean kernel. Yeah Yes, a rust in on the user space I mean, I'm pretty sure That's not something like I I'm I'm 100% sure that you know, let's take a look at leave ebpf actually look if you go to bpf An ebpf bootstrap. It has good examples if you go example here You can take a look. There's a rust repo that to give you a couple a couple things. Let's take a look at xdp So, yeah actually bpf Now I think the bpf itself was rated in c right, but I think the loading was in rust I think that's where it is In the kernel space you have to worry about normal like concurrency constructs and stuff like that because when you're doing the incrementing there's Yeah, I mean you you should you should probably there is I mean again I didn't like very much double like, you know talk too much on on the bf map There's bpf map that are very much isolated by by cpu and and so on so you can define You know when you're writing any resource obviously concurrency is a it's maybe a problem, but um It's most of the time you just need to design the right the right thing so if you're writing If you're writing a map just try to figure out the right use case for you for you If like something that is very much segregated for a node, for example to like limit like the interaction between multiple users It can be an option, but yeah, I think that's something that you still have to worry about but is uh But you know if you look at look if you look at lib bpf and look any lot of projects there's a lot of like It's emerging. There's a lot of new new code new technology that's kind of simplified life For you like when you write a specific code um I didn't talk about much, but ebpf is very strict very strict. You cannot just go and create any c code Once it's it detects that you are gonna, you know write something out of range or do something that is should not be doing It's not going to allow you to even build this or if it builds it It doesn't allow you to to mount it So I think you have a lot of controls there and if you look at the bpf code Look at bpf tools. It helps you like answer these questions. You have any other question Sorry to know that's actually this bpf code got bit invoked. Is that what you're saying? Yeah, so you saw what we did with uh with ip rules We set like zero and like if the rule got been blocked blocked is like a one or something That's how you should design probably your bpf code You should actually think about okay. I'm gonna define the rule The rule for me is to block a to b but also I need to define the monitoring aspect and let's let's create Let's define the risk. Let's say You can create a metric like we did earlier to say every time I'm blocking From a to b because of my code go create this metric that you can go and put that in primitives and alert on it Right, just again when you create your ebpf code think about the monitoring aspect and also the enforcement aspect Oh, so you're saying is can we can we take a look if a certain this okay, so if you if use like In in my example, I'm using bpf tool, right? bpf tool is pretty cool and you can use different things You can use bpf cool. Uh, bpf cool. That's a good name bpf tool on a certain interface It's gonna list all the ebpf code that's been attached to an interface So you can always go back and say, oh wait like what this kind of actually things not working Oh, like let's run bpf tool on a certain interface. Okay. Well, all these things are being attached. Therefore this why it's enforced was Network interface correct. That's been true. Yeah. Yeah, ebpf is is definitely again We are just seeing that in the context of networking but ebpf code can be running for let's say sys calls or like system calls or Opening a file here, you know things that are in, you know, let's say you want to monitor other aspects of Linux It doesn't have to be networking, but obviously interface comes in discussion when we're talking about networking for sure Okay, thank you very much for joining this session. Hope to see you soon and Have a good kube con