 Thanks for taking time to attend our session. We would definitely be able to honor you with master's degree in troubleshooting NFV issues at the end of this session. It's going to be a crash course. So my name is Adik. I'm working as a senior cloud success architect for Red Hat, focusing on cloud and open stack with targeting NFV as a primary troubleshooting area. And this is my colleague, Jason. Hi, my name is Jason. I work as a senior technical support engineer in Red Hat. I work in open stack team. I work with integration of open stack with other products. And for some time I've been working on triple low and NFV use cases with open stack. So let me first explore a brief agenda to understand how the presentation is organized. We will first explore the high level architecture of the NFV. And we will then deep dive into exploring the challenges with troubleshooting NFV issues. Then with some examples related with AVNF crash due to canal panic. And Jason will take you over through a series of issues that he had faced with OVS DPDK, I mean non-working OVS DPDK and performance issues related with OVS DPDK. And what are the some of the troubleshooting methodologies that you need to apply to overcome these challenges? So let's get into the basic NFV architecture. So as you all know, open stack is an operating system for cloud infrastructure as a service. And NFV stands for network function virtualization. Originally, open stack was developed to cater the requirements of private cloud and public cloud for enterprises and service providers. But the initial development of open stack was to satisfy or meet the NFV use cases. Later on, we had to adapt open stack to meet or cater the NFV requirements. For this, we had to introduce a lot of changes within the open stack and outside of open stack when it comes to integrating open stack with other components within the NFV. As you know, open stack alone cannot deliver NFV. Open stack is just one piece in the puzzle. So let's try to just explore the basic architecture of the NFV. We have OSS, BSS. And we do have EMS that's connected to the VNFs. And we do have VNFs. And we do have a lot of vendors who build and deliver different use cases for the VNFs. And most importantly, we do have the NFVI platform. And NFVI stands for network function virtualization infrastructure. And the open stack is going to be the primary component that is going to power the NFVI in this architecture. And we do have MANO that handles the management and orchestration for the whole NFV architecture. So from this diagram itself, you will be able to understand how challenging a troubleshooting going to be within the NFV architecture. Because there should be, in a lot of cases, there should be a lot of gay area who is wrong, who is doing things in the wrong way. The symptoms may point that there is something wrong with the NFVI open stack, but the problem may be somewhere else within the VNF or within the way it was orchestrated. So the intention of this session, as I explained earlier, to help you to understand where to focus when it comes to troubleshooting NFV issue. So let me just try to explore what is the fundamental challenge with the NFV. As I explained, we have OSS, BSS, NFVI, and MANO, and VNFs. And in most of the cases, all these components are developed by different vendors in the market. And we are going to stitch all these together to deliver NFV architecture for a telco. So the basic challenge is all these vendors have different level of support and different level of technical support people who are working on these issues. And if there is a coordination between these people, when a telco is hitting a problem, then the problem will go nowhere. So again, to make this makes it very complex when we try to split down, when we try to split each component further, further deep. So that means you use NFVI open stack as an NFVI, and NFVI, maybe the open stack is given by Red Hat or any other vendor. And if you try to go deep into the NFVI, there should be dependency on other vendors. You may be using a different storage vendor. You may be using a different SDN vendor. And you may be using a different vendor to compute services. And a lot of other add-on services are integrated into the, to create the NFVI. So this means you further need to work with different vendors to troubleshoot an issue. So this makes the NFVI very, very complex to troubleshoot. But what we have to do to ideally achieve, I mean, we need to bring all the players into a single table to basically efficiently troubleshoot an NFVI issue. I'm not trying to say a physical table, but virtually everyone should be around the table to troubleshoot an NFVI issue efficiently. So let me just give you an example. So how we applied this methodology in one issue that we heat during well-deploying an VNF on, well onboarding a VNF for a telco environment. So the problem that I'm trying to explain is Red Hat open stack was used as an NFVI. And then the telco used a different vendor to VNFs from a different vendor on top of open stack. So basically the problem was that there is a compute node one and compute node two. And one VNF is running on compute node one and another VNF is going to run on compute node two. And they can run fine. They can spawn the instances, orchestrate the instances without any issues. But whenever the VNFs start transmitting data packets, at some point, one of the VNFs kernel panics, it crashes. So this means there is no pattern. It's pretty random. So you cannot say after five minutes or something like that it crashes. No, nothing like that. It may be one hour or two hour. After two hour you keep transmitting data packets between the VNF. And this is reliably reproducible. So because you just need to transmit the packets, at some point it will crash. And we got the kernel panic message from the VNF and we also got a VM core from the VNF. From that point of view, Red Hat is unable to analyze the VM core because it's using an operating system that people in Red Hat cannot do not know the code for that. So ideally, so the VNF is going to crash on either of the compute node. It doesn't matter on which one. So ideally the VNF vendor communicated that they have tested this VNF in their internal and open stack environment multiple times. For multiple days they have kept it transmitting packets but they never hid the kernel panic. So on the other hand, the NFVI infrastructure vendor, they say the VNF is going to, is crashing. So you need to analyze the VM core and tell us if there is anything wrong with the NFVI that we need to change. So if we work this way, then definitely this is going to be a big problem and we will not find out a solution for this. At this point, we need to have a collaboration between the VNF vendor and the open stack vendor to efficiently troubleshoot this. And we managed to do this and at the VNF vendor, they tried to analyze the VM core and on the red hat side, we tried to analyze what is wrong with the open stack environment. And we got the first symptom, we got the first symptom from the VNF vendor and they are saying that the VNF, the VNF is, both VNF is configured with a 1500 MTU, MTU of the target destination VNF is also 1500. But whenever a packet arrives on the VNF, the packet is getting defragmented because whatever arrives here is a 9K packet, a jumbo frame packet. So we were wondering if this send a 1.5K packet and the packet arrives here as a 9K packet and during the deep fragmentation, the operating system cannot handle the defragmentation of too many packets that it crashes due to a bug. So this actually helped us to deep dive into how the packet is being transmitted from one VNF to another VNF within the open stack and where is it actually, the packet getting changed into a 9K MTU. So to achieve this, we ran a TCP dump on all the intermittent interfaces to see where the packet, though originally it started with 1.5K MTU, where it getting changed into a 9K MTU and we found that within this flow, either on the tap interface, either of the tap interface, the packet is getting from changed into 9K MTU. So this actually led us to explore further and understand and investigate why the packet might be getting assembled here to a 9K MTU, though originally it started with a 1.5K MTU and we found that the open stack is configured for 9,000 MTU and the VNF is configured for 1.5K MTU and due to the security groups, being security group rules being applied on the tap interface and the way the IP tables works is that to apply the IP tables rules on a packet, it would assemble the packet to the maximum possible data packet size before it applies the IP tables rules on top of that and because this has a 9K MTU and to apply the security group rules, the packet is reassembled again into a 9K and after that it is never fragmented and it goes directly into the VNF because the VNF cannot handle jumbo frames, it again need to be defragmented. So you would originally, you would really think that, oh, this is very easy to troubleshoot, right? Change the MTU of the VNF to 9,000. It's very simple, right? Why don't you just change the MTU of the VNF to 9,000 to resolve this problem? But that is a problem because the VNF is not built to support 9K jumbo frames MTU. So we cannot change the MTU of a VNF to 9K, that's not a solution. Then you would think, why don't you change the MTU of OpenStack into 1.5K to match, to sync with the MTU of the VNF? It's very easy, right? But no, because the problem is that OpenStack does not support changing the MTU per port. You need to change the MTU of the entire OpenStack deployment. That means the same environment is expected to cater a lot of other VNFs which is going to get huge network performance benefit by using the 9K MTU. So by changing the whole OpenStack into 1.5K MTU, so we are actually killing the performance of all other VNFs, so it's not a solution either here. The other important thing that we could explore as a solution is enable port security. So port security is basically a feature within the latest OpenStack to actually disable security groups per port. So that means you may guess, why don't we just disable the port security so that there will not be any IP tables rules, so the packet will not be reassembled at the tab interface to 9K. So it's a right guess, but this is not true, because with port security, we are not bypassing the IP tables layer. We are only bypassing the IP tables rules. So there should not be any rule within the IP tables layer, but there should be a single rule that is accept everything. So to apply that single rule, the IP tables kernel will again reassemble the packets. So this is not a solution either. So another solution that we explored, the VNFs, none of the VNFs in this environment requires security groups. So why don't we just set a bridge NF call IP tables to zero so that the IP tables will be completely disabled on that specific compute node. So this is an acceptable solution, but the problem with this is that we cannot import any other VNF, which may require IP table support or security group support. So we explored this option, and we originally implemented this option, and we then again explored further options. So from the VNF vendor side, they did further research, and they inspected what can be changed on the VNF to support this, and from that investigation, they found that the VNF was configured to use an E1000 emulation, and if we change this NIC card to use what IO, so then they see that it improves the performance and it again prevents the crash. So why because the crash happened because of a bug within the E1000 emulation that is within the E1000 driver that is being used within the VNF. So it gave them two benefits, either I changed the performance improvements, then also a solution for implementing the prevent the crash. So is this the ideal solution? Of course, no, because the package still gets reassembled and defragmented. It's going to affect the performance. So ideally we should, the ideal solution for this problem is, OpenStack should support per port MTU, or at least per network MTU at a minimum, so that we can specify that the port that the VNF is connected should have a 1.5K MTU and all other ports within that network environment can use jumbo frames or 9K MTU. So with this, so we have created a bug, upstream bug for this and we have internally also recommended that to efficiently support and to efficiently make the VNF working within this environment with for telcos, we must support 1.5K MTU and per port MTU within the OpenStack. So as I explained earlier, we didn't apply any of the solutions that I explained earlier, because they are not, they have its own set effects. It's not fit as a right solution for this problem. Then we came back into the two last solutions and within this, this is something need to be developed within the OpenStack, so we cannot implement this right now, but and we finally decided to, to implement this solution. So that's all from my side. So what I want to highlight here is it was a great collaboration between the OpenStack because there was nothing wrong with the BSS and OISES and the manual orchestration layer here, because the orchestration was completely successful. The problems have started after the orchestration. So we cannot, we can rule out the possibility of the OISES, BSS or other manual layers causing this problem and we came down into the two options, either the problem is with VNF and or the problem is with the NFVI infrastructure and as you know, since we collaborated, we were able to find out a solution from the VNF point of view and from the OpenStack point of view to solve this problem. So this is more important if we collaborate, we can get a solution. So and it's very critical to collaborate more effectively within the troubleshoot NFVI issues. With this, I will hand over to Jason to walk us through some of the issues that he has been working. Okay. Thanks, Sadiq. So what I'm going to talk today is OIS DPDK troubleshooting. So what I want to present to you is the methodology we use troubleshoot OIS DPDK. So OIS DPDK is a very important and very interesting technology. Why? Because you can use any hardware and a NIC that supports DPDK and you can have fast packet forwarding using this DPDK libraries, which we are using in OIS. So although it is very interesting and it's very beneficial, people who have deployed an environment with DPDK would have known there are a lot of challenges around it. Those challenges mainly come because it DPDK, OIS DPDK integration requires an understanding of the hardware some configuration on Linux system and some configuration on OBS. Something goes wrong, then it may not work well. So before I go to the methodology and then to a scenario where I explain a scenario where we troubleshoot DPDK. So let me tell what you're looking at. So you have a compute node, which is Linux system. In my case, we usually use Red Hat Enterprise Linux version seven. You have some CPU cores which is isolated and tuned for this PMD workload. You have a NIC card which is supported by DPDK, which will be associated with VFIO PCI driver. And you will have huge pages configured for your instance and OBS, how much ever that is required. You will have OpenVswitch version 2.6 or 2.5, whichever has DPDK, it's fine. You have configured OpenVswitch with one NIC card. You have enabled huge pages per socket or however you want to configure it. You have already decided the cores you want for pole mode driver thread because you need to assign some cores for DPDK. And you should also need to know the number of memory channels which are configured in OpenVswitch. So this is a classic example of your environment. You have a VM with vortio emulation via LibWord. It acts as a Vhost user socket which your OBS DPDK or your pole mode driver thread will pole and get information and it will also communicate with your actual card. And it will make things work. So I have usually come across two types of issues when troubleshooting OBS DPDK. One is OBS DPDK does not work at all. You spawn an instance and it does not, you cannot ping it, you cannot ping outside, you cannot ping inside, nothing works. Other scenarios, when you get something to work you can ping outside, you can ping inside, but it does not match the performance you are expecting. So it's like fast data, so fast packet forwarding, right? So you would expect it to be much better and you may just find maybe like 30, 40% that's a big difference 30%, 30, 40% of what you expect. It's a big degradation, right? So these are two types of performance issues that I faced with OBS and I want to explain what are the normal methodologies I developed to fix the issue fast. Fixing the issue fast is a more critical thing for us. So there are two things that I do first. So that I can fix the issue in probably five minutes. Just run OBS VSCTL show command. What will I look at? I look at the output, does it look as it is expected? I expect a DPDK bridge, a net dev bridge. I expect a DPDK port under it. Everything looks fine. There are no errors seen. That's what I expect in OBS VSCTL show output. The next step is if at all that output does not give enough information, looking at the VSWD process log is also very important. So the main thing that gives you this benefit is this open VSWD process use DPDK library. So when open VSWD process starts, that is OBS-VSWD process starts, it initialized all the source required for DPDK. That is it starts the PMD threads. It detects the NUMA architecture of your compute node. It spawns a Polymode driver threads on the right course. The Polymode driver threads access the huge pages that is configured in your system. It creates, you already have a net dev bridge for DPDK. It will create a DPDK port. It will detect the NIC which you have assigned for DPDK and add the DPDK port inside the net dev bridge. So all these information you can see just in this log file. So it's a very important log file. It's a var log open VSWD. So the next step what I do is to check configuration of open VSWD and check the hardware. Does it match? So there are multiple things inside this that needs to match. It's supposed to be NUMA where you need to have core that is optimum for use of OBS DPDK. The next point, so just by checking the configuration you have for open VSWD, open VSWD DPDK and the hardware you have is not enough because your understanding of the hardware may be wrong. So you would probably set something in OBS DPDK and it's trying to accomplish that, trying to assign those resources. If there's something wrong about it, it'll just fail. So you need a tool or you need a set of commands which will give you a real-time information of what's happening in your Linux system. What's happening to your Polymer driver? Is it able to start the right way? Is it able to use a course that you wanted to? So all these informations are important. I will not list out all the commands required. There are a lot of open VSWD-related commands which you can use to check the output of the NetDev data path, is there or not, or the DPD port is not, there are a lot of commands. But I want to just give you an example. Some of the important commands which I used to get information from the Linux system itself is driver CTL to list all the PCI devices and the driver associated with them. So DPDK use VFI or PCI driver so I can make sure that the driver I want to use is actually associated with VFI or PCI rather than the kernel driver. The second command is also very interesting. I use PS command to list all the threads on the system. So open VSWD process starts the Polymer drivers as P threads and so if you do PS-EL, capitalized for thread and you can see under OVS, you will find a PMD something, 54 or 55 or something based on the PID. These are the threads, PMD threads. So you can see the threads that are running. You will notice that they have 100% CPU utilization because it keeps on pulling on the core. On that it keeps on pulling the socket and the NIC. So you will find the CPU utilization is 100%. You get the TID, the thread ID and you can check inside your PROC file system the NUMA map for that process. So ideally in slash PROC, you don't see thread ID, you only see process ID, but although it does not show a directory, you can still put the thread ID and you can check the NUMA map and you will find which all huge pages this Polymer driver thread is using. It's a very interesting command. I will show this later. Other as to check the huge pages assigned in each of the NUMA node. So you may have 12 huge pages of 11 GB, but you can assign all the 12 on one NUMA node. Another NUMA node can have zero also. So this is a very important command. And the last is you can also run a packet capture, packet capture as you know that in OBS DPDK you cannot use TCP dump because it's a user space process, PMD is the user space process to capture that you cannot use TCP dump. You can use a tool provided by DPDK, DPDK P dump which you can compile and use it. So the next set of issues, the next set of kind of issues I've seen is performance troubleshooting. So this makes it little more tough because you've got everything to work. So a simple log file does not, may not always help. So you have to understand the scenario. So first thing is to understand the scenario. What was the test about? Is it, what is the source and destination? Are they both VNF running on a compute node with both DPDK running? How are you testing it? Which tool are you going to use to test it? What is the packet size you're using? And how much you're getting? So all these gives you a good understanding plus the network devices in between them. They gives you a good understanding of what you're exactly trying to do. And then you can come to, what is the expected performance for the hardware? And plus the percentage degradation. Percentage degradation does not give much, I mean, it does not give you a quick hint of anything, but you will just know that how serious is the issue. I mean, it's a big difference or small. And so at some point, it will give you a good clue of what could be the issue. Next is the obvious configuration complements the hardware. So it's not that the obvious configuration is wrong and DPDK does not always DPDK does not work at all. There are scenarios where you can do the configuration wrong and it will give you bad performance. For example, memory. So you configure open V-switch with memory channels. So these memory channels are something on your motherboard. Each NUMA node has some memory channels on which your memory sticks are on. So what happens is when OBS assigns huge pages, it is good, it gives you good performance if OBS can set these huge pages at, huge pages contiguous space. So if you can assign this huge pages at the start of every memory channel, when OBS PMD thread starts, it can associate the objects at the start of this, which gives best performance. So if you give wrong memory channel, it will not work well. Your motherboard can have a riser sort slot. If your vendors build your server, probably they already know and they've kept your neck card on the best PCI slot, but it's good to look at your motherboard data sheet to understand if there's any other, any riser slot and you're not using this slot or a better performing slot on your motherboard. You can enable hyper threading course. So what you can do is enable hyper threading and make the Polymer driver threads use the sibling course. So advantage is this Polymer drivers maintains a cache for each core. So as much cores that much better. So you can also set optimum receive queue. So you have an option to set receive queue for DPDK physical port. So the next is you can have the course on which the Polymer driver runs tuned to this workload. Because this is a special kind of workload where the PMD thread takes 100% of your CPU cycle and you will need to isolate it to make sure no other processes run on it. So what I do is I use a tool called tune D and then there's a profile called CPU partitioning. So what it does is it adds an option inside inside the kernel which makes sure that there are no CPU ticks on this core. So that is if in case the processor or the core has only one process. If there are multiple processes does not work. So it's necessary that it has to be isolated. It has to have this no HZ full and the cores which are going to use for Polymer driver. It gives good performance. There are a lot of things that you can look at. Avoid mixing kernel data path and net dev data path on your compute node. If you have made something like that, your packets can take both paths and can give you low performance. You also have BIOS recommendations from DPDK developers, which is also there inside the OpenVSwitch guide which I will mention it later. So you will need to apply those BIOS optimizations to make sure the CPU or the cores run with maximum benefit of your system for the Polymer driver to work pretty quickly. So the next is a little more interesting point to understand which cache it's a lot. So in your OBS DPDK for you to get the best performance you need to, your traffic needs to get all the flows resolved in EMC cache, that is exact match cache. This is a cache that is maintained by the Polymer drivers. It's a part of OBS DPDK and if your traffic frequently hits this EMC cache, you will get best performance. But if at all, so the EMC cache has something called as a hash and there's an algorithm to fetch the flows from the hash. The size of EMC cache, everything is pre-compiled when you compile the OBS with DPDK libraries. So if the EMC cache misses, then you go to the mega flow which is also something in OBS in memory. If that also misses, so there is a considerable penalty or degradation of performance in mega flow. But if mega flow also fails to resolve the flow, it will go to open flow from OpenVSwitch which gives a very bad performance as equal to using ML2 OBS by itself. It is also very important for an individual to understand what is the next version of OBS and what version of OBS you're using because there are features coming in one by one and there has been a lot of active community contributions of suggestions and code to improve the performance of communication via OBS DPDK. For example, in OBS 2.5 and 2.6 itself, there are a lot of difference like MTU or having multiple QO or so on. So having a good understanding of which version of OpenVSwitch you're using, what are the drawbacks of that version and what is the advantage of the next version would be good. Profiling is the last pointer I have. Profiling is something you only go to when you get the issue. Otherwise you cannot simply profile a component because unless you are sure this component or there's a PolmoDrive thread is not functioning at its maximum or there's some kind of saturation. So then only you can get something out of profiling or else you're just getting a lot of data without any strong hint of this is the root cause. So let me come to OBS DPDK failures. So this is a scenario which I told first that it does not work at all. So this is the symptom, it does not work at all and this is the architecture. It's a common generic architecture. You have a NUMA hardware, there's two NUMA nodes. There are some set of cores. There are some set of memory sticks on each core and there's a NIC which is in NUMA node one which I'm planning to use. So I will talk about two scenarios in this OBS DPDK failures. So first issue is it does not work. That is the issue. It does not work at all. I start the instance, ping does not work, nothing works. So first thing I do is I took a look at OBS VSTL show output. And I see everything is almost as it's supposed to be. There is a BR link zero which is my net dev bridge and there is a DPDK port. Although DPDK port there has been some error in creation. So it says cannot create PMD threads due to out of un-pinned cores on NUMA node zero. So I could probably look at the next step. Let us look at the open VSwitch log and I found the same error. I could not find anything more. So just by looking at it, I would probably guess that open VSwitch was trying to start the power mode driver. And something went wrong when it tried to start a power mode driver thread on NUMA node zero. So next, so where is OBS trying to start the power mode driver threads, PMD threads? So I had to look at the OBS open VSwitch database. There's a column or other config and I look at the mask. It's 28, 28 is a hexadecimal value. I'll just convert it to binary. So this is CPU used, CPU mask starting from right to left which is from zero to nth CPU core. So in this case, zero, zero, zero, one, zero, one. That means it started zeros, first, second, third, fourth. So second and fourth core is the one which open VSwitch is supposed to use as per my configuration. I'll just have a quick look at the NUMA topology. There are a lot of commands. There are two, three commands which you can see the NUMA topology I usually use the NUMA CTL. So a quick look says core two and core four is on node zero. So what about node zero? Why is it not able to start on node zero? But DPDK port, NIC and R and port and NIC are on node one. So I can just use that OBS, OBS it'll show. So OBS it'll list interface and I can just say check the status and you will get a lot of information in the status. I just want to concentrate on the NUMA node over here and says NUMA node one. So you would probably say so what? So what is our issue here was that we were using OBS 2.5. So in OBS 2.5, the code forces a pole mode driver to be in the same NUMA node as the NIC port. You will know, you may have read somewhere that OBS 2.6 is NUMA aware. So you do not have this feature so that that's why it failed to start the pole mode driver. So this is what it was for OBS 2.5. What we did is we just move the pole mode driver to node one. So how did we do that? We just went to the same database and set hexadecimal value to use three and five cores inside this other config PMD-CPU have in mask. And we were able to fix the issue. Similarly, another one issue where OBS DPD cannot work at all. So first thing for me was to just look at OBS CSE to show output, see what's going on. Just like before, everything is fine except the DPDK port seems to have some error while creating. And this looks something more similar. It seems like cannot allocate memory, which looks very simple thing. So if I were to guess, I would say it tried to start the pole mode driver thread and it could not allocate memory. Why is that? So what exactly gives us information about which memory to use? So I am using huge pages. And huge pages, we mentioned which socket, how much memory we have in OBS DPDK. So I just had a look at the log, OBS vswtd log. The output is little more than what I saw in previous output. Here when the OBS vswtd process started, it tried to start the PMD thread and it crashed. Segmentation fault, it gives some more ideas. Segmentation fault means it's not able to access the memory segment it wants and crashed. And later it comes to the same error which we see in OBS VTL CTL short put. Something around socket one, possibly, right? So what next? Just let's look at which NUMA node it uses. OBS DPDK uses a NIC card which is in NUMA node one. And the next output shows looks like the socket, sorry, the NUMA had 8 GB of huge pages, sorry, the OBS DPDK had 8 GB of huge pages on node zero and 4 GB of huge pages on node one. Which is no reason to crash anyways. So you have some huge page, right? So it should still start the poor driver. So OBS trying to start, it knows how much it needs to allocate but does Linux provide that huge pages or not? So before you go, that's a by default Linux, if you enable huge page on Linux system, it divides the huge page across different NUMA nodes. So if you have 16 huge pages of one, one GB and you don't have any other configuration set in sys, sorry, start up, it will divide that 16 into eight huge pages on each NUMA node. So when your system comes up, you'll find eight huge pages on node zero and eight huge pages on node one. So you can get that information from sys file system. So I just had a look at the sys file system and I noticed that for nodes zero and one when I checked. So I do find 8 GB for node zero but zero for node one. So that means the Linux does not have huge pages ready for node one. So if any application tries to grab a huge page from node one, it will fail. So that is the root cause of the issue that the system did not have huge page on node one. So just to fix this, we just enabled echo four into the sys file system. So I did not require reboot. So what exactly happened was while testing all these configurations, we made a mistake of setting zero to node one, NUMA node one. So there was no huge page on that. And we had never rebooted the system to test it again. So it always remained a zero huge page on NUMA node one. Although I don't need to reboot it, I just need to directly set huge page as how much of a required by OVS. And that fixed the issue. So these are the methodologies I use. These are the issues we faced. But it does not cover everything. There are more points to add. There are more suggestions. I've also come across a question, how to check if there's a saturation, or check the CPU caches to understand where the performance. I'm still looking at more and more tools to understand what we can do to figure out quickly in OVS DPDK to understand where there could be a bottleneck issue. So for coming across all these steps, I had gone through OpenVswitch DPDK install guide, how to guide the Intel articles, the DPDK programming guide to make up all these steps to understand how we can do things better. So that's all for me. Let me just conclude this. Jason explained some problem that you may hit with NFV, maybe deep into OpenStack and the networking stack that you can fix alone. But some problems, it may not be clear initially where the problem is actually is. So it's very important that you drive the collaboration with multiple vendors. And let this presentation be a motivation for you to drive that level of collaboration. Thank you very much. So if you have any queries, you can come over at the mic and ask them. Hi, one of his group is saying I'm from Spartan Communication, and I have a question regarding measuring latency at the virtual interface between the various virtual entities, let's say between the BNS or if you are using OVS DPDK between OVS DPDK and the BNF and things like that, are there any tools out there to measure latency between the virtual entities? I think there are different tools that comes from different BNF vendors, but I cannot pinpoint you so exactly what general tool that being used. So there are a lot of, it all depends on what's your use case for the latency and how much you need to achieve. Right, so if I'm trying to troubleshoot, let's say, latency issues, and I want to find out which particular virtual entity, which BNF, or is it the OVS switch or DPDK, wherever the additional latency is getting introduced or service degradation is happening. So exactly, so if you want to find out whether there is any latency problem within the OpenStack layer, you basically need to use a lot of tools, profiling tools that Jason recommended. Maybe he can give you more details. And if the latency is to measure whether the latency is within the BNF layer or on any other layer, you may have to use a latency measuring tool recommended by the BNF vendor. Ultimately, you need to prove where the latency problem is. Maybe you want to. Does the hard part, the hard part? Yeah, sorry, go ahead. So at the first step, maybe I was at short one PMD hyphen show command, dpi of something slash, in which I showed EMC cache. So it also shows you the CPU cycles. So it shows that polling takes 100%. There's another one section, which is processing. So maybe at the first point, you can just look at if the processing percentage comes close to 100 or something like that. So basically, you're saying the processing cycles are coming to the top or threshold. That means that that will introduce additional latency and that might be causing the problem on the OVS Yes. OK. All right, thanks. Thank you very much. Thank you. Yeah. We just have two minutes. OK, my question is if you did any tuning inside the guests, because in my experience, I activated the multi-q and for instances. But when you do the performance test, you always get it on the same CPU. My question is if you've done any tuning inside the instances for this project? I think multi-q is already available within the KVM. And I think it's still not applied within the open stack layer. And I mean, there is still work in progress to support multi-q. And on the other hand, the other tuning that you can do inside the VNF is basically implement DPD can set the VNF as well. So the packet that comes from the hypervisor, by passing the hypervisor kernel, it also bypasses the VNF kernel and directly goes into the user space mode. But this is conditional on the support from the VNF vendor to tune it. And other tuning is really specific to the VNF. I don't see any general recommendation that I can provide that can be applied for all the VNFs. Because VNFs are having different operating system and different way of handling. It handles different functions from routing DPI into a lot of other functions. There may not be a single tuning parameter that fits every VNF use cases, right? It's going to be specific. So just to add to that, the multi-q you mentioned, you'd enable that from the VNF instance? No, I enable it. On the flavor, on Nova, you can enable it on flavor. But when you start the guest and you start doing performance tests, you always get it on the same CPU in the network. So what I did, inside the instance, I activated some kernel parameters that spread the load between all the CPUs. And my question is if this is recommended or not. And in your experience, you also do that? Yeah, multi-q is definitely recommended to give you the best performance. But there is still work needed to be done to manage the multi-q efficiently within the open stack layer. It's already there within the KVM or the hypervisor layer. So if you spline LibWat and KVM, it will work without any issues. But there should be a lot of testing and everything to make sure that this is working as expected. When you put it on open stack. Thanks, everyone. So you can catch us any time if you have more questions. And thanks for listening. Thank you.