 Good evening, everyone. I hope you guys had a wonderful summit. My name is Mohamed Saleh. And yeah, that's my TSA-approved picture. So I'm a specialist solution architect based out of Dubai. But covering the EMEA region, you can reach me on my telegram or my company email address. I would be happy to address the questions that you have. Any help that you need, you can always ping me. So I'm actually going to talk about how to do RTKVM and DPDK on OpenStack using a triple O. Or let's say, in our case, it is actually that I had OpenStack director-based installation. Before that, you might be wondering what is OBS DPDK and what is RTKVM and stuff like that, right? So there were a couple of sessions that had happened before. So this is kind of a follow-up session, a beginner session. Actually, it is kind of a beginner session. But yeah, it is a follow-up session for these two, actually, three sessions that had happened yesterday and today. They were actually done by a couple of engineers from, I mean, solution architects and engineers from both Nokia and Red Hat. So these are the two sessions that happened for the OBS DPDK. If you want to know more about OBS DPDK, how does it work, how you can configure it, what are the troubleshooting method that you can use, please go to these links. They are not funny sites or something. I generally put it there just to make it easier for you guys. It doesn't take you to one site. So please feel free to take it. Yeah. And then that's the real-time KVM. How it works, real-time KVM, how does it work? So this was done by Tapio and he's our Nokia friend. And Eric, it's a wonderful session. You would love it. A lot of information that you can get. So please feel free to check it. So I'm getting into our agenda, actually. So it's about setting up RTKVM and DPDK using Red Hat OpenStack platform. But you can actually adopt it for any other platform as you require. I'm actually trying to base it on Red Hat OpenStack. But yeah, feel free. You can actually take the ideas, build your own stuff based on whatever OpenStack deploy that you have. So this is the minimum requirement that we had while doing the testing setup. So there was one compute node, one controller node, one director node. Director, it's undercloud. I don't want to call them undercloud. It's a confusing thing. So I'll call them director. And then there was a Lua traffic gen. It is basically a Moongen node with a Lua scripting language on top of it. It is very easy to do stuff. But I'm not going to cover the Lua part. You can actually refer to Tapio's presentation today. So it covers pretty much how things are done while doing the testing. So the important stuff. You are going to deploy RT. And there is going to be DPTK along with it. The thing is, both of them require particular configuration which complements each other. So you have to know your compute. You have to know your compute from down. I mean, you have to open up the compute node, whatever the compute node that you're going to deploy it on. You have to open it up. You should be knowing what is in there before getting into configuring the templates and things like that. So first of all, this one actually covers the BIOS part. So you have to disable the power management. You have to disable the hyperthreading. You don't want any surprises because hyperthreading uses threads, which could be when you try to configure, when you try to pin, hyperthreading plays a lot of issues. So disable it. You don't want it. You have legacy USB support. You don't want those things in there. You just disable this. If possible, disable system management interrupts. Some hardware vendors give you access to those. So if you have access to those, disable it. Otherwise, yeah, it's up to you. If you can't do it, that's good. Now, you have to know more about the compute node. Again, we are doing RT stuff. You're going to do a lot of CPU pinning. You need to know how the compute server is actually configured. So you should be having some latest processors. Does it support Noma? How many cores does it have? How many CPUs does it have? How much RAM does it have? How much RAM you have installed on it? How many memory slots you have filled in? It actually helps you figure out the memory channel that you will need it when you configure the DPDK pod. So you need to know the memory channels that are being used. Some of the hardware has those information in their manuals, or you can actually run a couple of commands, and you can get those things out. And then, how many NICs you have? Network Enterprise cards you have. Are they compatible with the DPDK? Do you have compatibility? You can actually check. The DPDK site, the link is given there. You can actually go there, check what is supported, what is not supported. And then the last thing, you have to know where your NICs are located on your motherboard. This is important because the PCIe is actually mapped to the Numa nodes. So you have, in some cases, or in other cases, it was mapped to Numa node 0. So how do you find it? You can actually use LSPCI and do some grep. And then you can actually see the PCI address. And then you can go in sysdevice and then PCI address. I guess that's the right path. But anyway, it is somewhere there. And then there is a file called Numa. You open it, and you would see the Numa node affiliation for that particular NIC. So how many of you have done a deployment using Tuplo or OpenStack director? OK, that's pretty much, yeah. So do you mind, can you teach others so that we can? I can save some time on this thing. So it would be easier for you to teach others. We are a community, right? You teach the next one next to you. Or I have two slides actually covering just basics of how do you do it. I'm not sure. I'm actually oversimplifying it, maybe. But let's go through it. So what do you do? You have the compute nodes, the controller nodes. You have the servers ready. And then one of the servers is going to be converted into a director node. What do you have to do on it? You'll install Red Hat Linux or whatever that you want to get the Tuplo up. But I'll concentrate on rel. I'm from Red Hat. They pay me, so let me concentrate on rel. So we have rel, rel7 running. And then you have to subscribe the server to Red Hat portal. Then plan. You have to plan your network. You have to plan your storage. It's very necessary. Most of the time, in my experience, it's the network that screws up. And whenever that gets clicked, the open stack installation happens within an hour. So plan it very well. Get the network relands in proper order. So that's very important. Set the hostname properly. Once all these things are done, install the Tuplo client, Python client. You can just run in a yum install, Python Tuplo client. It's very easy. It installs the necessary files. There is one more package that I missed for the images to get for the overcloud images. But yeah, it's again one more yum install. Once you have the Tuplo client installed on your machine, you have to go in and modify or create an undercloud config, which will be basically containing a couple of lines about which the internet got to use for provisioning what are the IP addresses that you're going to use, what will be the DHCP range that you are going to use, all those things. It's a couple of lines. Like 10 lines, I think max, it will be there. You configure that. Once that is done, you run an open stack undercloud install. And depending on your internet speed and things like that, it gets installed within an hour. So that's the director part. So once the director is installed, you have to have a couple of working folders, templates, images in your home folder. And then comes the fun part. You have to get the network and a couple of files, actually. The main files that we normally or in majority of the case that you will be touching at first is network environment.yaml. You can actually find it in user share, open stack, triple O heat templates. It's already there. You just have to make a copy of it, edit it the way you want, specify all the networks that you want, VLANs, all those things, put it there. Then the other one is configuring the nicks. I mean, configuring or preparing the templates for the nodes, the controller and the compute nodes. So you have multiple options there. You have multiple folders, example folders there. You just have to take one of them, modify it to your needs, again, place it in your home directory or your working directory. That's it. Once that is done, tag your compute nodes, introspect the nodes, compute, control, Chef, whatever, introspect it, tag it. Once tagging is done, you run the open stack over cloud deploy command. So we are actually calling the network environment yaml here. So that's a very brief outline of how you will be doing OpenStack install using Triple-O or Red Hat OpenStack platform. Now we have to get into the more interesting thing, which is why I'm here. So that's RTKVM and DPDK. Again, once again, I'm saying majority of the stuff, majority of the things that you do actually complements each other. So you can avoid RTKVM and have just the plain vanilla kernel. And when you have a KVM on there and then you can have DPDK, it will work fine. You add the RTKVM, it will be almost complementing each other. You have all these settings almost there. So I don't want to show you the whole network environment yaml file. It's a very big file sometimes. So I'm actually taking a couple of lines from it just to show the important things, because the session is all about showing you what are the important stuff that you have to actually care for. So here, these two lines are actually the first boot and the post-deployment yaml files. So the first one runs after the first boot and the last one runs after the complete deployment is done. So pretty straightforward, right? It's very easy. It's very easy. Believe me, it is very easy. You have to actually get to, yeah, initially, you will be struggling a bit. But yeah, it is going to be easy once you have get the hang of it. And you're going to love it, you're going to love it more than any GUI that is out there. The GUIs are very limited. You have only very few things to do within GUI. So I love the command line CLI things. So the amount of customization that you can do is enormous. Anyway, I'm not getting into that. Let's go. Now, the fun things, more fun things, more and more. The parameter defaults, it goes again into the network environment dot yaml. You can create under yaml and then add it. But for simplicity's sake, I'm adding it here. So parameter defaults. So after several number of configuration things, you have the DPDK configuration added there. So the first one, I don't know if I should walk through every single one of them. But yeah, I'll just quickly do that, network bridge mappings. I think you guys know it. It's actually mapping the physical interface and physical interface in OpenStack to the actual bridges in the OVS. I'll leave that. The Neutron DPDK codelist, two and three. This is not just two numbers. This is not just two numbers. It is actually based on the NIC. Where the NIC is located in your motherboard? In this case, it was located on Numa node zero. But then you'll be asking, why I didn't use zero and one? So zero is actually most of the time, you won't be able to tune it. You won't be able to get hold of it. It will be used by something else. So leave that one. Why I didn't use one? What could be the reason? Again, the host process, there are other processes running on the host, right? It needs processing power. So I gave it one to the host processes. Now, whatever left, it is up to me to do whatever with it. So I have actually, yeah, I'll get into that part later. So two and three is actually a pinned core. So it is not used by any of the other processes in the system. I'll get into that part in the next slide. So two and three is pinned. And it is on Numa node zero. And the NIC that we are going to use is on Numa node zero. So that's why the numbers are there. Now, neutral DPDK memory channels, that's four. The system was having four channels of memory to access the memory. So I have put in four there. You have to actually find it out the hard way. Sometimes you have to open the hood and check it. Or there are a couple of commands that you can actually run and find out DMI tool can be used. Anyway, you have to actually find it out. And it is very necessary to have a good performing DPDK RTK VM system. Now, the fourth one, DPDK socket memory. It is a huge page memory, one GB, a huge page memory. Why I have given that 1,024? It is on Numa node zero. And that's why that number is there. If it was on Numa node two, then Numa node one, then it is going to be 0, 1,024, or I don't know. Yeah, 1,024. If there were two nicks, one on Numa node zero and the other one on Numa node one, 1,024, 1,024. Yeah, it's that easy. Now, VFI OPCI, that's the driver type. That's the DPDK user space driver that is going to be used by the system. And the NOVA reserved host memory. It speaks on its own, right? It's the reserved memory that will be used by the system. Symbol, the host processes use that memory to run. Now, the NOVA CPU pin set, this is the pin set. This is actually used by NOVA while provisioning. So any new instances that are coming up will be having CPU cores from this list. So this actually says NOVA can use cores from 4 to 12, but not 8. So it's kind of a regex. I think you must have figured it out. Now, I know it is a bit too much. But yeah, that's how it is. Neutron data type, this is going to be applied on the OVS switch. It is NetDev. So whatever bridges that are going to be created on the Neutron or OVS is going to be NetDev, NetDev type. And then the next one is we host user socket directory. That's where the sockets or the interfaces for the VMs are going to be created. That's where the OVS uses or runtime folder that OVS uses. Now, the third one, NOVA scheduler filter. The important one is actually here. NUMA topology filter. You can see it, right? NUMA topology filter. So it actually gives NOVA scheduler some more things to play with. So NOVA scheduler can actually schedule our filter nodes based on the NUMA capability of the system. Then the last one, compute kernel arguments. This line is exactly as it is going to be replaced in each compute node. The grub line, you know the grub line where the kernel parameters are set up. That line is going to be placed in all the compute nodes. So what does it say? It says the huge page memory is 1GB. The default one is 1GB. The huge pages are 1GB. It has 24 huge pages. The power state is disabled. The pinned cores are from 2 to 13. We are pinning cores from 2 to 13. So the IOMMU is on. Yeah, that's pretty much it. I don't know if there is anything I'm missing there. Yeah, that's it. So this line is going to be replaced in each compute node for the kernel parameter. So yeah, watch out for this. You can do whatever you want to do with it. And then it is going to be going into the grub config. Moving on. More files. It's really tough, man. So compute firstboot.javl. Again, this is the firstboot YAML file. I'm not showing everything in it. I'm just showing a shell script. Shall I go through each and every line? It would be fun, right? I'm not going to do that. What it does is it basically sets the system CTL service file and adds a couple of parameters into it. So up. So the one that is interesting is here runtime directory mode. That is going to be 0.775. And then you have group QMO. And then you mask 0.002. What does it say? So the runtime directory will be having whatever files that is going to be created in the runtime directory is going to have group QMO. And the mask will be 0.002. And the directory mode will be 0.75. So I guess you guys know about those things. I don't have to explain it, right? OK, that's about it. And then this is continuation of it. Again, it is almost the same thing. I don't want to go through everything. It does the similar thing. But on to, ah, it keeps on. I keep on doing it. It actually edits this file, OBS CTL file, which is actually responsible for creating the bridges and the database updates and all those things. So that's a script. And then within that script, it actually replaces the UMass there. So that's what it does. And then I'll skip on to the 49th line, which actually says, which is what I was referring to, the kernel arguments that will be replaced, right? So that's what happens there. Now, I need some water. Now, thank you, man. So that's the grub line. And then we have the, yeah, due to some issues, I think there is some bug. I'm not sure if it is resolved or not. So we have a SeLinux set into a permissive that we had to do that because of some bugs or something. And then that's pretty much it. And then the rest of the things is actually part of the YAML file or the heat template that gets pushed on to the shell script. More. This is ComputePost installed. This happens after the deployment is done, after everything is done. This particular file is going to be running. So again, I'm not showing everything. This is part of it. This is the shell script that is embedded within it. You can go ahead and do whatever you want to do with it. That's the shell script. OK, what it does. It actually calculates the CPU mask and applies to OVS, what is it? OVS, that's it. So it applies these core masks to OpenVSwitch configuration. Somehow, the deployment doesn't do that. So we have to do it manually. That's what, not manually, actually. You write a script and then the script does it. So that goes there. And then it restarts the OpenVSwitch and so on and so forth. And then, yeah, the one that could be interesting to you is the Kernel RTKVM. It is actually coming from a special repo in Red Hat. There is a repo, REL, I think. Yeah, REL 7 server NFE RPMs. I think it's just a collection of RPMs. You could possibly find it somewhere else also. It's RTKVM, RTKernel, some tuned profiles. So you could use those and make your own repo and connect to it. Again, connecting to the repo, you can actually use the shell script to do the magic that you want. Here, it is actually the REL Registration YAML. It actually automatically registers the compute nodes and the controller nodes to the Red Hat portal. So that's what this thing does, this particular YAML config does. I think this is the last file that need to be edited. Yes. Sorry. OK, this is actually nickconfig.comput.yaml. Again, it's part of it. What you're seeing is Nick1, I'll go bottom up. So nick1, which whatever the first nick on the node will be converted into a dpdk interface. All these things are done automatically while deploying. So nick1 is converted to dpdk, and it will be dpdk0, and then it will be added to the br-link0 bridge. So straightforward, right? Right? Right? Come on. I know you guys might be sleeping. Too many configuration files in there. Yeah, this guy is going crazy with files. Yeah. So that summarizes everything, all the configuration that we have done. Those are the main areas that you have to concentrate on. I have actually put all these files, the original files, on a GitHub. I'll give you the links. You can actually go in and check what is happening there. What is the time, actually? OK, I have 10 minutes, right? So open stack deploy. Again, you have seen this deploy command before during the slides where I explained how the director-based installation goes on, right? So this is an extension to it. Here, we are actually adding a couple of more environment files to it. You have the network isolation, which is sometimes required. Most of the time, it is required to segregate traffic, segregate networks. I'll not get into that. Rail registration is straightforward. Then the network environment.yaml, which we created very lately. And then the next two lines, which actually says that there will be a one control node and there will be five compute nodes. Or whatever you want. If you want one node, one compute node, just replace it with one. And then it will take care of the deployment. Now, after 40 minutes, that's kind of a norm in a heat, in a open stack deployment using Red Hat Linux, or Triple-O probably. It takes 40 minutes to get a open stack running. I think it is whatever the size of the deployment, it roughly takes that much amount. The problems that could be happening is with the networks. But yeah, so far, in my experience, it gets deployed within 40 to 45 minutes. You get open stack running if there is no problem with the network. Again, open stack works good. Open stack is best. And then, come on, guys. So open stack works. Most of the time, in my experience, it is the network that makes the problem. Right? See? He knows it. Jochim knows it. So these are post-deployment tasks. Here, we are actually creating flavor that we will be using for deploying or instantiating instances of VMs. What we do, the first line is pretty straightforward. You're actually creating a flavor. The next line, the third line, which is goes to, I don't know how many lines are there, but yes, pretty much the configurations that you are seeing there actually says that this flavor is to be used for real-time and DPDK or real-time purpose. Or if you want something to have, yeah? You know, it is not necessary. It's like not everything is necessary, but you can have it. I'm just putting it there so that you can actually have a feeling of what are the things that you can actually do. Here, it is actually, this flavor is actually going to use one numanode, numanode 1. And then, it will be having a memory policy, which is preferred, which means that it is preferred to use the memory from numanode 1 or the one numanode that is assigned to it. It should be assigning memory from that thing. And then, the CPUs, the cores that need to be used are 4567. And then, the memory need to be from that numanode. And it should be 4096, which is 4GP. And then, it has real-time. It actually sets some more parameters into the lib world. So that actually gives a bit more details into the lib world. I'm not getting into that, those things. It takes a lot of time to explain those things. And I'm running out of time. And I have five minutes left. So yeah, yeah. So the whole thing is not necessary. You can choose whatever you want. But some of the things, which will set a couple of things within lib world, which will be useful for you, which will be useful for our deployment. Now, host aggregate, again, here we are actually segregating nodes with the CPU pinned so that we have some zones created or host aggregates created. So that's what it does. The lines actually explains all these things. The second line actually creates a host aggregate called pinned host. The second one actually adds a metadata to that pinned host. Pinned equals true. And then adds the compute host to the aggregate. Straightforward. Very simple. Again, we are actually creating some external network here. The one thing that we have done during the testing phase is we have actually disabled the port security. It was actually creating some issues with our testing phase. And it was dropping packets. I think it was because of some spoofing. It was trying to prevent some spoofing attack or something like that. So we disabled those port security, which basically disables all the IP table stuff meddling in between. And yeah, the first one is the network. And the second one is the subnet. Don't look at the IP address. Anybody is not going to be applied onto the machines or the instances. Now, upload image and run the instance. Everyone does that, right? I think, yeah, open stack guys. You upload the image. You run the image. The only thing the difference is, it's not difference, actually. You are actually using the flavor that we have created earlier. We'll use that thing here, M1 medium huge thing. And yeah, then it uses the DPDK nick that we have created earlier. So that's about creating and running the instance. Now, I'll try to make it quick because I'm running out of time. Measurement results. Actually, today morning, Mr. Tapio has done a wonderful presentation. It actually details about what are the tools that we have used or that he has used and how you can actually use them to do the testing, how you can actually leverage on the open source. This is all about open source tools. It's not about the proprietary tools. I'm not against proprietary tools. They do well. But sometimes, we don't have any choice. We have to use open source tools. And open source tools are not behind. They do a good job. You can actually refer to the slide. You can refer to the slide, not the slide session there. Again, it's a good site. It's actually referring to the session that happened today morning. So I have to confess something here. In this test, actually, having RT didn't give any particular advantage here. Because the thing is, the VM that we are running and the application on top of it was not using RT. It was test PMD that we were using. So it was, yeah, we ran the test. We ran the test. But the test was basically about how the DPDK is performing. So we did a couple of tests. Actually, you can see that with RT, we had a couple of performance drop in the round trip time in moving data from the package in through the OVS, through the VM, back to the traffic gen. So it was actually, with RT, it was actually taking a bit more time. It could be because of how the RT is behaving, because RT always give more priority to the prioritized task. And it might keep away whatever that is happening there. So there is a difference. I'm not sure why it happened. If anyone knows about it, please, you're welcome. You can explain to us. GPIO? IGPIO? No, it was, I think that's the driver. The driver, right? Yeah. OK. Yeah, good. Yeah, probably. Yeah, we could test that. We could test that. Yeah, it got worse when you had RT on it. But then we are not doing, we don't have an RT application on it. It was test PMD. It was not using any RT stuff. It was, yeah, probably, yeah. I don't have a mic. Yeah, actually, I have my contacts in the beginning of this slide. I'll show you the back. If you have any concerns, if you have anything that you have to know, you can always get back to me. If I don't know, I have all my colleagues here in Red Hat and outside who can always help me or help us in getting what we are trying to achieve. So we are a community. We have to do that. So without DPTK, you can see that. Without DPTK, we have an 80 microseconds, more than 80 microseconds. I think it went to some 200 microseconds or something. And then the back at loss, the throughput was like 0.2 MPPS, which is horrible. So to sum it all, you have to know your hardware well. You have to know your CPU, how much code count you have, memory channels, where the nicks are. You have to set up your BIOS properly. You have to install the director or the undercloud. Plan your environment carefully. Build your environment files. Run the deployment script with all the environment files. And then the rest of the stuff is plain open stack stuff. You have it there. And then the last one is actually the actual files, the source files that we have used for testing. You can always refer to it. It is github link. You can always go there, check it out, play with it. If you have any comments, drop a comment there. I'll definitely check it if I can solve it, if I can give you any help on it. I'll definitely do that. These are the references. A lot of references are there. Model references we can add. Yeah, basically these are the stuff that we have used. And thanks to all the guys that helped us. It was Nokia and Red Hat. It was a joint effort. Yeah, so those are the guys that helped us to get this thing running, automated, all the things. And thank you.