 Hey Billy are you ready for our meeting? Hey Adrian, I don't see anyone else here. Is it just us today? Yes, seems everyone else bailed If that's the case, can we hold off for just a couple of minutes? I'm trying to new deployment and I need a couple more minutes to get to a breaking point Sure, no problem. I can help if you want Yeah, I mean if you've got time, that'd be great So I've got a customer that's got some requirements I'm not real familiar with they've got a legacy application They want to containerize and require some additional interfaces to make it work properly now I know multi CNI we allow you to run multiple CNI's for a container Which will basically add multiple interfaces to the container was the first time I've really had to do it with SRV VF's I Was able to bring up the pod, but I can't tell which of the diff additional interfaces is used for what? Okay, do you have a diagram or something? Can you show me what you're trying to do? Yeah, I think I got one here All right, I'm gonna I'm gonna share my screen. Hold on just a second. I found it. All right sharing Can you see the drawing? Yes Okay, so as you can see from the picture, they need a high-speed interface to receive some video stream in on one interface The app does it's transcoding magic and then my sense of modified stream out a second interface These video streams are at high data rates and need to be on separate interfaces Okay I have a 5g case that is quite similar at least The interfaces interface requirements is similar enough In fact, this kind of setup is more common than you think Let's take a look at your cluster. Do you have A terminal around? Yeah, actually, um, I'll share but let me send you some here. Here's the login stuff and Okay, you can maybe Tmux in here. Let me share it. There we go. Yeah Yep, I mean Okay So here's what I have so far. I found some srv documentation So I've got enough that I was able to take my srv capable nick And divide it into multiple virtual functions so that multiple containers can share the same nick Hang on. I got this command and somewhere one of my notepads All right, there you go. So now you can see, um, I've got these two interfaces with four vfs each All right. So now if I show you my qtl Get pods all right, so there's my test pod and If we go and look at it So you can see here, um, I've got e0 for my default network and then a couple of net interfaces for the additional interfaces But I don't know which net net one or net two is used for what? so I'm logging I'm looking at logs and trying to hard code some values into my sample container to make this work out Okay, got you. Um, I don't think you have to hard code anything. Let me jump in real quick. Um, okay And see what versions you're running So Okay, I see that you're running multis and the device plugin so Let's start with the device plugin for instance um Okay, you're running 3.3.1, which is fairly recent. That's good. Okay. I'm gonna I'm gonna check the multis version just in case I want to make sure you have all the changes Um that we've recently added What changes are you looking for? um Okay, I'm I'm looking for some changes the device info changes the network plumbing working group recently Applied recently approved a new spec called the device info spec It specifies how additional device data can be passed into the pot. Look If everything is working the information that you're looking for should be here Yes, there it is Oh You see In the pot annotations, there is a network status field And within that field, uh, there is a JSON object for each network interface. Okay Within that object, uh There we have recently added the device info part And that is the information of your hardware device that is nicely associated with your network interface Now as how was that association was actually made? um It's cute. Let the one that allocates the device depending on the device plugins reported pools um Okay, hang on that was a little faster a lot of components. Um Maybe we can draw a picture or something with all the components and how they talk to each other Sure. Um, actually I'm working on some slides for kubecon. So let me paste A link in the chat If you can the link I have a deck on it here. Um Let's just jump to slide seven Okay here. I'll put it in the shared screen so we can see it Can you see it? Yes so Um There in that diagram shows a typical sraov Deployment So as you already know Multis is the one that allows you to call different cni plugins to attach additional networks, right? Right, right. Now that part I got yes Okay Well, uh, when it calls sraov cni It adds an extra parameter with the pc i address that it has to configure The pc i address of the vf Okay, then the sraov cni just configures the networking aspects of the vf For example mac address vlan tag, uh, the trusted mode and so on Okay, I kind of get that part. Um, I see the sraov device plugin in your drawing. Um Now what's the purpose of the device plugin again? Yeah In the case of sraov Um, we need to make sure that we properly manage the limited hardware resources that we have in the node In this case the virtual functions of our sraov cni Also, we need to modify the pods runtime spec to provide access to those hardware resources Okay, basically that is what the what the device plugin does for us It is a devon set that sits on the node. It discovers the available hardware resources Our arranges them into pools into resource pools Defined in a config map and makes these pools available to kiblet Okay, now, um, I do have a member creating my config map. Um, hang on. We'll switch back to the terminal here for a second. All right, so So here Here's my config map that I created All right, can you see that? Yes, it looks like, uh, you set up the config map correctly at least, um That config map is saying it's defining two resource pools one of the resource pools will um We'll pick up all the vfs from ps0 and the other Uh resource pool will pick up all the vfs from pf1. Is that what you wanted? Yes. Yes. So that sounds good. I think I got a little lucky there on what I copied. So, um, okay Now let's look at the network attachment definition. Did you create one of those? Yes. Yes. All right, so All right, so here's the yaml I used to create that Yeah, okay. So as you can see um Your network attachment definition references the pools that you've created in the config map So that's ctx6 pf0 and pf1. Okay. Yes, I do. I do. Okay. So, um Here, let me go back to that picture so I can look at that again. All right. Here we go. So so, um, having that reference, um properly done, uh, how it works is um when the network attachment definition references the, um The correct pool and a pot is allocated. Um, that uses one of those network attachment definitions kubelet Selects a free device from the pool and asks the device plugin to provide the device information The response contains information about linux devices That, um, have to be added to the runtime spec mount points and environmental variables Okay, so basically that is how the pot has Then access to to a specific pc i address Now the problem is that that information flows directly from kubelet to the pot and, um It gets to the pot in a very limited way. So you don't have Like the context of the cni interface with it Okay, um, and you said there was this new spec that was written. How does that tie in with what we're talking about here? Well, the spec changes this into the diagram in slide 10 10 hold on All right, there's slide 10 Yes, there you go. So as you see now the device plugin Now writes a file in the notes file system Then is that the device info file? Exactly the device info file. Okay it writes the file in a predefined path with a standardized format and uh, once the file is written Moltus will then pick it up and Write it into the network status annotation under the right interface Okay, so this device file is actually on the node So you're saying I could troubleshoot some of the device related issues just by looking at that file on my note. Is that right? Yes, that's correct All right. Hang on. Let me go back here and see if I can Take a look at you know, would you remember what the name of that file is or where it is? It's in slash bar run k8s Cni, you know tap your way. Okay. Yeah um device info Dp for device plugin Okay, you have all the yes. Oh, yeah, there's a couple of them there. All right It's a json format. So if you just if you come pipe it to jq if you have jq installed um So we can see the json Okay, on tent correct. All right That looks a lot like the device info stuff right there that was in the annotations Uh, yes, that's exactly right so, um If you look into the um annotations again, you should see that that json Part is Has been added to the uh network status annotation in the right interface. Okay. Yeah, let me go back here and see if I can figure this out Um, okay. Yeah, so I see that pc i address in the device info That we talked about and then there's that sr ov dash net zero that we're talking about in the name field Cool Correct. So um the sr ov net zero is the name of your network attachment And the uh net one is the name of the interface Seen inside the pods network namespace. All right. Cool. All right. I want to stop sharing here. That's that was good. That's good All right, so um, so we wait a minute you wrote a new spec just so we could figure out which interface belongs to which network No, that was just one of the added benefits It allows network A new accelerated network devices to be added to uh, kubernetes more recently so Recently i've been Working on this technology called vdpa It exposes accelerated free tio compatible interfaces So it supports Exposing a normal net def to be used by standard containers, but it also supports exposing a special char device called vhost or vhost ddpa device That can be used by dpdk apps for example um to do memory mapping um They basically to map to map to map the memory directly Uh from the nick to the application Okay, so so the nick can dma directly into the app So this is uh one of the use cases And as you can imagine without this pack It will be impossible for the pod to know all the extra details needed to consume these new devices Okay, so um, so it sounds like it makes um bringing up these newer network devices easier And also but more in a standardized way instead of using i don't know maybe proprietary environmental variables or something like that, right? right so we wanted to um have a clear spec to um Basically make a pot on pots and application developers lives Easier The spec supports a number of uh different network devices The strio v devices, uh like in your case, mem if ddpa and some more Just it's easily extendable It's just a json format so it's easily extendable and It should make it easy for new technologies to be integrated into kubernetes Even if they're not based on the strio v device plugin Yeah, but are there any cases where devices are created by like a cni plugin instead of the you know any of the number of device plugins? Yes, for example, the host device cni and also the user space cni that exposes virtual user devices The spec also contemplates that use case um and it supports vhost user devices Uh in that case how it works is uh, it works in a slightly different way But um, so how it works is the cni exposes a new capability And if multis detects this capability, it will pass down Pass down basically the path where the cni has to put the the devising profile that we just saw Okay, so you're saying that the cni instead of the device plugin now does all the work but it has to deal with creating this file and Making sure it exists and then writing the file and taking care of all that td stuff, right? Well, yes, uh, but Most of the functionality has already been implemented in the library It's called the network attachment definition client and so um the cni should just import that library uh to Basically write the file and once that file is written Then multis will take care of the rest basically. Oh cool. Okay, so I'll now all this information is ultimately available from with the pod, right? Right. Well good. All right. So now I just have to write a script or a small program to parse all this data in the container I'm not really looking forward to trying to parse json in a script, but I guess I can do that Well, I told you I've already been through these. Um, Let me send you a link You're gonna like it Okay Open that link. Okay. I got it. Let me let me share it back over here All right sharing All right, uh app net utility. Okay. I see wait. This is a go module, right? Um, I think the only issue is that everything I'm dealing with is going to be in c Drop down to the api section Apis okay See they are go and see bindings. Oh, yeah, there's go and see I like that. Cool. All right, and um I see there are three apis One to get the cpu data One for get retrieving huge page data And then the third one is to retrieve all the interface data Uh, so has this repo been updated to support the new device infospec? We were talking about Yep It's very useful. It hides all the complexities of parsing the network status annotation um, environmental variables and system data and it offers a nice high-level api All right. Um, it's a lot here. I see um test pods dpdk app container images Yeah, there's a lot here. Yeah, it has some sample code on how to use The apis Oh, all right. Well, that'd be useful Um, all right. So what's this srv vf development? All right, there's a lot of stuff. Um, we've been talking about here. I see the network attachment definitions and config maps So um, so even if I decide not to use it it shows all the different places in a container to collect the data and how to utilize it Um, this looks useful. I'll start playing with this. Um Do you have a list of all those repos that we talked about? I said we was a network client Or network something. Um, yeah Yeah, I got a list Uh, let me show you man. I I need to um, invite you to my cube compression patient You're just asking the right questions. So you have my um deck. Yep. Still got it So I think it's on slide 13 If you check there Okay, um, I see yeah multis and srv network device plugin some of the stuff we've been talking about. Oh, there it is network attachment client Um, I see most of these projects are under the network plumbing working group um I thought that um the network plumbing working group just wrote the multinet spec that multis was coded to And that was it Yes, that was some time ago, but They have recently approved a new open governance model They have brought together many projects um Including all of those and now it's a very healthy and diverse community Well, this is perfect. I'll try to use some of this in my work. Um, but you're gonna be out. Aren't you if I run into some issues? Yeah, for two weeks, actually You still have that cube come slide Yeah, now go to the next Uh, uh slide. Okay, and you'll see yeah There you go. Those are the upstream meetings. Uh, you have links to the meeting dogs and you have the meeting cadence there Um, all right, I like it, but you know, that's one of those things I'm still like new to all this and learning all these technologies I think if I went to one of these upstream communities, I'd just get lost and feel kind of out of place a little bit Everyone there was new to it at some point If you have any issues, just reach out to them. They are um, a very, um Very kind and they will be happy to help you. I'm sure Oh, well good. That was real helpful. I really do appreciate it. I'm sorry for delaying our original meeting What was the original meeting for? I've already forgotten Uh, it's our virtual bureau clock zoom Oh, I'm sorry to delay that for work stuff. Uh, let me go grab a beer Yeah