 Hello, hello Hello All right. Hello everyone. My name is Flav Leitner. I work for the network services team and today I would like to talk about the challenges to provide networking for NFV environments Just for out of my curiosity how many of you know what NFV is all about Okay, half I would say so okay good excellent I will do some quick introduction moving from one step to another trying to Introduce to the idea the problems the challenges, so you will see that I will deep dive in the end into a specific solution But to get there I need to introduce some concepts and ideas So the agenda is concepts and The ideas for NFV what NFV is all about what's the goal what we want from it and Then we are looking for the requirements After that we are going to look quickly about the 10g networks see what the challenges are there how it works Then I'm going to talk about the physical to virtual to physical the so-called PVP scenario Which is the the basic before building the NFV Then some possible network solutions, especially the ones that we are more used to they are more commonly available But there are others so I'm not covering everything here all the possible solutions especially the newest ones But we can certainly talk about after the talk or even during the Q&A session Then we will dive deep into DP DK enable open the switch Because that's the the solution that we are working mostly with it right now I will try to explain how it works internally and not going to be it's invites But giving you an overview on the how it moves packets around and then possible improvements That we think it would Help to improve the current limitations This possible improvements. I'm not covering everything possible the most probably as you you know you fix one bottleneck you most probably will find another one and There are other work around solutions or solutions that they don't Have the same bottleneck so and there are efforts in upstream to resolve for other solutions as well But that will give you I hope that you'll after the stock you learn the difficulties for virtualizing network not only for NFV, but for other user cases as well Okay, so what is NFV all about NFV stands for network pollution virtualization and what we are trying to do is actually to move from the Physical device, you know, you have a firewall. It's usually this fire will be a physical device It has a custom software. It has a custom right hardware usually very specialized the hardware Accelerated higher so asics and supports one You have load balancer, it's another physical box you have the routers or depending on the region you are from routers And what do we want is we want to have this appliances Running inside of a virtual machine So this step here is actually we are decoupling software and hardware again. The same happened with the Other things as the end for instance is decoupling the control plane from The data plane here we are decoupling the software and the hardware And the idea is that Running these appliances on the virtual machine. We add this virtualization layer obviously running on top of a Also, so-called white box this is a It the idea is that it would be a commodity hardware. So nothing is specifically not you don't need to match these hardware with the specific appliances you are running it and you You in theory you should be able to run multiple appliances Regardless of the the function itself on the the same hardware So that's basically the idea So what's the goal most of you that's how we operate today you if you want to have a new Networking in the company or whatever You need to buy an equipment you need to Order something you need to talk with the hardware vendor need to negotiate prices Then you buy something maybe they have in stock or maybe they don't you have to wait to then a truck Stop by someone has to go deploy inside of your data center. Maybe you need more space So it's a slow process until it gets delivered and up and running It also has a high cost and Less flexibility by flexibility I mean you cannot automate most of this process after that so assuming that we have Succeeded to virtualize these appliances Then you have a fast process because if you want something you can just start a virtual machine It obviously should have lower cost because most probably you have the hardware already there you don't need to Have someone to come by and what everything look for more to space, you know buy more hacks or You already have a box there that's capable of growing as you need so you have a spare room You can just start new VMs and Obviously with the VMs with you plan you saw Some presentations about Open stack and how is how is he's becoming to manage a large number of virtual machines? So we will also need a way to manage This network of appliances as well So if we succeed if you need something like load balancer to how there's some pharaohs and etc You can just click a button and suddenly your devices will be there and what are the requirements? Well in an ideal world, we would like to have this appliance if you utilize it and don't give up on anything I mean we all love performance right so but yeah this This layer here impacts a lot So it has a cost and also the hardware is not that Heavily optimized it easy that's running there So we need to make some you know Pros and cons here they might not perform at the same level, but maybe for most use cases what we have today is good enough for instance, I was asking if for the Today we have Virtualization and some of our products. I ask around and they for some use cases They really don't care about the performance because today what is provided is good enough but for an FEV Let's see So the challenge you for an FEV is that we in order to compare with the hardware We need to go with these two things low latency and high throughput How much low latency I don't know it depends on the use cases The the market that the ones that are driving this effort mostly are telco companies So they have voice traffic and voices usually is small packets Not this modest one, but small packets especially for signaling and they also need high throughput High throughput means density, right? You can put more customers on the same equipment if you don't have high throughput You most probably will have more devices and then the cost benefit might not be interesting anymore and When we do this measurements the requirement is that zero packet loss And that really means that zero packet loss if you one packet is dropping then the measurement is not valid anymore And you have to see what is the the best number you can get with zero packet loss this is the way we we saw the vendors and hardware vendors especially comparing devices and see You know to evaluate to which one is best. So we are trying to do the same thing, but we're softer this time Okay, let's look about the 10g 10g is the most probably the The speed that most data centers are Moving or already at it and it will give you the Once you understand this then you can estimate what's going to happen when we move it to 25 40 or even 100 G so in the worst case scenario that we'll are going to use this wide speed with the smallest frame Here's some explanation what the smallest frame looks Looks like so the other nets we have two pieces mostly The other net specific That's not part of the packet the the packet that you were sending but specifically to the other net This appears in the wire so 20 bytes for the the whole thing Which is 12 for the interframe gap plus eight for the Mac preamble And then the other net frame itself which is the Mac header 14 bytes plus four six bytes as a payload The total is 84 bytes So if you multiply by eight you convert by two bits and then just divide to 10g you get this packet hate 14.88 million packets per second Why this number is important because if the software solution Can process packets at that speed it means it's Reach the line hate speed. So we are equivalent to the hardware solution Okay, so now that we have this Trafficking the The peak traffic hate how much time we have to process each packet So just do a simple math here. We found that is 67 nanoseconds Well, we know that CPUs are faster But how that how much that represents on a softer solution running on by metal So if you look if you consider a three gigahertz CPU, that's about 20 hundred cycles So basically we have 20 hundred cycles to do everything we need with a packet and that means receiving Processing and either drop or send someone else But that's all we got including the if there is any processing the application That's forwarding this case the appliance To put it also in perspective one cache means takes like 32 nanoseconds and Well, this depends on the platform you are looking in the CPU So if you have something specific, this is just an approximate number, but you can see it's pretty much a half of our budget So two cache misses. We already lost a packet L2 cache heat is like 10 cycles L3 cache heat 36 so in another words, it's a very small budget to deal with this It's The amount of the cycles you have and the work you need to do it's really a tight budget budget Okay, so Going to the software stack. This is the pvp scenario the physical to virtual and then physical Traffical generator will send traffic through a physical board Going to the virtual switch or some solution that Makes the packet appears magically inside of the virtual machine Then we have a forwarding solution there or I don't know some the the the virtual function And then back to the traffic generator to through another port. This is the basic Unit the how we are testing and comparing results with the other solutions as well And this is what I will be using in the next slide as To see what are the weak spots? So basically that 200 cycle budget needs to cover all this path Okay possible solutions we have a few available today one is Linux bridge Then open the switch SROV and DP DK enable of yes Linux bridge it uses the kernel data path It is a good thing because it's integrated with the kernel so you can most probably use The facilities available in the kernel But on the other hand it relies on nappy and has an unpredictable latency nappy means unit Napster is I don't know if you are familiar with it, but it's a mix of interrupt mode and pulley mode So basically you when you start it's in interrupt mode And then when the packet comes in it generates an interruption This is a schedule then a thread will come back and will fetch this packet But switch to pulley mode. It will pull mode either until a certain budget is reached Or there is no more packets then it switch back to interrupt mode So if there are more packets, let's say that you got 64 packets, and that's the budget It will reschedule itself and then continues and pulling mode as soon as you don't have any more packets It will go back to interrupt mode Well, the good thing about nappy is that if you have Not at the highest the speed possible You don't spend CPU cycles processing the network card, but on the other hand it impacts on latency And we are looking for low latency here. So I'm interrupting mode is really not an option at this point Another thing is some predictable latency One of the reasons is the interrupt mode and another reasons could be resource allocation The kernel does a pretty good job with a resource allocation But if you we for instance get a lock in the way, or there is a memory allocation Then our 200 cycles is gone. So we need to have the this predictable somehow and It's not a CDN ready. So Linux bridge doesn't Doesn't speak like for instance open flow protocol Or any other as the end of protocol And if we measure the throughput Very simple test the fee to fee so physical to physical. Let me go back a little bit This okay one I'm short-cutting here physical to physical Okay forward Physical to physical I get one million packets per quarter and that's a very low throughput. So What we can do with the bridge, right it is used for a lot many other things and it's very hard to change the way it works or the Holy kernel to make it fit in this use case and another thing to consider is that the keyboard runs in user spaces So since the Linux bridge is in the kernel space, we need to send, you know We need some context switches to send a packet through the kernel to Kimui in user space Open the switch is pretty much the same thing as Linux Linux bridge. It runs in the kernel data path also relies on api Unpredictable latency one good thing is that it is SDN ready at least it speaks one of main protocol But the throughput since it has pretty much it leverages the same infrastructure in the kernel The Troupled is pretty much the same one million packets per quarter Also, the Kmo is in user space. So there there is a solution for the for that to speed up Which is called V host, but it's still It it has an additional cost SR of E I guess most Most use cases that depend on low latency high throughput are using this solution Because with SR of E you can bypass the host you can literally move the device To inside the guest expose the device from the host to the guest. So you bypass completely SR of E It's interesting because one physical port It appears in the PCI bus as multiple virtual ones So you don't have only one port, but you might have some up to a 256 devices they are Sort of you don't have access for the whole PC PCI device You can change you cannot change for instance things that would impact on other devices So it's quite limit, but still you have everything you need to send the received packets And you can simply move bypass the host and moving this device directly to inside of yeah It's not SDN friendly because then you don't have the virtual switch running on the whole so you don't have control You don't you cannot tell what the guest is doing But if you want to stir traffic on Certain traffic to a certain guest or another a traffic to another guest. This is not possible anymore the devices Physically exposed it to the guest and that's also a problem because there is no obstruction then you move If you do migrations for instance, you have to make sure The host you are receiving the virtual machine supports the same hardware, you know the same Everything that you had before you need to have on the the other server Although 256 devices that's a limited for the that comes from the specification seems a lot That's not exactly what the hardware provides today if you get a card you might have only 16 supported maybe 32 I get I saw some supporting 64 virtual functions. So Just because this pack limited on 256 doesn't mean you will get there. So depends on the hardware also So good thing about the solution it offers low latency. It offers high high throughput But on the other hand you have all these Cons, right? Okay, so let's go back to software one thing that we could use is the data plane development kit We call DP DK and DP DK is a set of libraries and drivers Designed to one thing in mind do fast packet processing, right? So It's an open source project Has a BSD license if you there was a talk yesterday from Maxim and Victor They went a little bit deeper on what is the PDK so in this their talk are pretty much Related to this one as well. So I would recommend to watch their talk to see what they're doing in the v-host the virtual land So what's the user job DP DK receiving sending packets within a minimum number of CPU cycles That's what we are looking for but what the PDK is not and I'm putting this here because Quite a lot of people confuse that it's not a networking stack So it doesn't replace the car the Linux kernel if your application in the socket for instance You won't get the socket here. There is no TCP processing. There is no TCP Conception control algorithm and it just gets the packet deal with the buffers deal with rings and Send out a packet on the device Okay TPDK pull ball drivers so TPDK has this pull ball drivers And what is that is a way for you to get packets from the network card directly in the user space So the buffers are all mapped in user space The next thanks to the VF IO interface we can simply get packets there or send packets out on the network card very quickly Okay, so now speaking about open this switch plus the PDK open the switch It has a kernel module you probably saw that if you already played with open the switch You saw that there is a module there There is also a module in the open the switch repository, but the kernel part of it is just a cache So the user space does all the intelligence to the user space does all the processing For instance a packet comes in you match on it and then you accumulate all the actions You need to do and once you have this big giant Rule that says I need to match it on all these fields and do all these actions This is push it to the car. So when the next part comes in the car knows what to do, but What to get to that point user space did all the processing all the processing Well, the PDK provides libraries those libraries They work like a pre-allocated memory they use huge pages It's busy pulling now interrupts You have lockless rings you have rings with a single producer multiple consumers or single consumers a single producer So we can use these libraries and and the driver to have the v-switch doing all the work And that's what we are doing TPDK enable open the switch So remember the 14.8 million packets per second if you measure Physical to physical we can get around 16 million packets per second. So that's a huge bump There is one trick here that it's not the processing is not done per packet one of the features That the PDK bring us is batching. So we are moving batches of packets But it's still if you send put on the tracker generator and measure you will get around 60 million packets per second per core But that's not for free. We need one core 100% busy running in the PMD thread Okay, let's look at about OVS DPDK for an FV We need to provide network connectivity to virtual machines came who runs in user space But now we have DPDK we have OVS everything is running user space So there is this interface called the host user It is a shared memory between chemo and the virtual switch So we can simply copy packets on this ring and it will appear on the guest No kernel at all in the in the The whole side the throw puts about three and a half million packets per second per core Default features enable in the PVP scenario. The system is not heavily done it but since this is very CPU bound and It depends very much on the the platform you have the CPU you have So if you have a high-end system, most probably you get a better number than I have here also We also support multiple Streams multiple queues, which means if you enable more cores to process more queues This is k-waps quite linear linearly. So I would say with four cores We can get line hate speed even pushing to the virtual machine But yeah, this system needs to be carefully done it. We'll talk more about that in the next slides How does it work? Basically, we have this pull mode driver thread. It wants a CPU and that means it It's nothing else should be running on that CPU if you have something interrupting the CPU You are pretty much stealing CPU cycles from it and we already know that the budget is very tight. So Any interruptions any scheduling events or anything that happens that are locking a memory allocation something like this It doesn't work. So DP DK provides us all the resources pre-allocated. We already are working in busy pulley mode But still we need to make sure that these thread Wounds the the CPU The devices can be distributed between the PMD threads. I say the device here and but this actually queues By default device provides one queue, but you can enable more queues and then distribute this queues to different PMD threads each PMD thread will be easy looping and busy loop for packets and then do the processing and the model there there is run to completion Which means once you get the thread gets the packet it will go through the whole Processing until either the packet is sent out on another port or drop it. So we don't have any scheduling in the middle and one thing that I already mentioned one of the trick to Virtually expand our number of our budget for processing is to have batching so you move Instead of a packet by packets you move like 20 32 or 64 packets at a time Okay, so let's do an x-ray of the PMD thread to see how it works It's quite simple, but due to this simplicity sometimes you have some consequence of it So basically you have the any ports in this case It's a genetic scenario. You can have any ports as any ports as you want You have the queue is represented by this this box. So this is the receiving queue Which did this box here represents the PMD thread It will pull for packets for each received queue. Once it gets the received queue It will push the batch to the forwarding data plane That's the v-switch It will do all the processing with the packet. Let's say Modify the destination IP address I don't know remove a villain tag or add a villain tag Everything anything you can do in the SDN network and then either will drop the packet or send out on TX QE for another device That's basically how it works for every I mean for any use case you want to use the PMD thread as You can see as you add more ports or QE is it could be This actually could be any QE is from the same device, but it's the same thing So as you add more QE is you will spend more cycles here pulling each port and One impacts on another of course Okay, so going back to the just to fit things together, you know what better what I'm talking about We discussed this to simulate the NFV. So here I have I don't know a firewall running Where you can I don't know use Juno as a login on Juno s or something here to establish that And here you have two logical ports the v-switch in the middle to physical parts This is represented here So you have two physical ports here and the two logical ports and this weird our arrows simulate to represent the forwarding Just to show you the equivalent sort of what I am talking about I Represent as you can see this physical port It sends traffic to the virtual switch, but it doesn't send traffic out to the traffic generator So I didn't connect the TX QE here So you only have the RX when it's relevant and TX when it's relevant We'll talk more about the second picture, but this is just to put things into places Okay, this is now with a more realistic scenario for an FV Let's talk about the packet flow So this is the feed for the physical part that receives on traffic generator. It's called number 10 It will come here the thread will pull for that will match this Data plane rule open flow rule it says how coming from port and send out on port 21 Goes to the RX of the 21 Appears in the a guest the guest does the fort to the TX QE in Inside of the gas which becomes the RX QE outside of the gas and then the the packet goes back to the pooling Then it sent out on the for data plane. It will see another rule here Oh packets coming from port 20 sent out on the Port 11 and then it sent out here on the RX TX QE and back to the traffic generator So one thing to note here The packet goes twice through the PMD thread. So remember that budget we have we actually have to Divide by two because of this Okay, how to measure to put with zero packet loss what we expect is that we have a set up on the traffic Generator a constant hate then the systems is constantly dropping packets with decreased the traffic rate and repeat until we find something that works out Okay That's actually it happens We should look at where the packets are being dropped it because there is the are the bottlenecks we should improve So in this case, there are three red stars the three weak spots where we should look more in the detail The first week spot is the receive QE for the physical device What happens that? Basically, if the PMD thread is busy doing something else and it's not pulling this device fast enough this QE will overflow And it it's a physical device. So it has a limited number of resources and There you can some network cards allow you to extend a little bit, but still It is Limited so it's a classical producer consumer problem If this is not fast enough pulling packets out of this device the queue will overflow and you will drop packets So it's really important that this runs fast The other thing is this receive QE here For the same problem if this is not running fast enough this queue will start to build up and At some point you will overflow the difference between this and the previous one is that The the drops are reported in the port start so you look at the device You'll see the drops you can identify in this case. You don't see the drops stats in the host You see the drops the stats in the side of the guest because when the application is trying to push packets to this part The port is full so it's dropping inside of the guest and it's not visible in the hosts Then the other weak point is the TX QE here on the view host user But that's not a fault in the PMD anymore. What's happening is that the guest is not running fast enough So the guest is not fetching packets is not pulling packets from the this virtual device and that So the the PMD is trying to push more packets, but there is no space so what we can do There were again the same Producer consumer problem so as you can see it's like a clock, right? So if you something goes off a little bit, there will be consequences and depending on what's off one of these three points will blow up Okay, measure throughput with zero packet loss I told you about this but in reality. I saw this happening System is stable for a period of time. You can see traffic is running just fine You go out to lunch and then you come back. You have like two packets drop it Very low throughput and the goal is to understand the drops Here is We saw the 10 gigabit budget. I put the table here with Less because that's what the virtual machines are doing anyway So this is this is the budget for the three million four million five and six Not just calling attention for this number because it will appear later, but basically That's what we are looking at To see the how in how significant are the impacts of others Remember packets go twice to the PMD. So the real budget for packet processing is this one All right, I measure how much time I spend pulling each device and how much time I spend processing the device So physical device for a processing I spend three and a 3.1 micro seconds and the view user egress device which is coming back from the virtual machine I spend two micro two point one micro seconds Others are really low processing here is zero because we are not using that and the pulling is quite fast So these are the main ones, but the total if you sum up ever all the numbers you will get 6.2 Microseconds to loop a packet batch of packets between in and out All right batch. So you saw we saw the number is six point two microseconds is 24 times more The prepackage budget I showed you before that happens because of batching. So we are virtually extend our Budget doing batching. So batch is a fundamental thing Assuming that we would have like a perfect batch of 32 packets We could have five million packets per second instead of three and a half So something is happening trying to estimate the numbers with the three and a half zero packet loss the budget will be this We can calculate or estimate the batch size to two into one in average, this is something maybe we can improve but It depends on other things right if you go there and take out all the packets You have at that moment, but it's not 32. Should we wait for more? It might introduce latency So what is wasting time internal resources is external resources internal resources if you look at how it works There is no locking a more memory is pre-allocated There is no reason for it to stop so it runs 100% of CPU busy, but we should look at the external sources of problems What they are how much significant they are we look at the budget we know the time We have so we have a way to tell if they are significant or not For instance, I was you callbacks. This is inside of the kernel So just to remind BVD processing budgets three for three million packets per second is this value. I Use this cool to call F trace It shows you the Functions that are calling in the kernel for a specific CPU you can filter that and then it has a nice time stamp And you can calculate how much it takes to run the kernel are still come back and this is the number I got without the preemption preemption cost I mean moving from the PMD thread back to the kernel run everything The run the CPU callback and then back to the user space 50 Microsoft seconds. That's roughly eight batches one solution for that Disable the RS you callbacks on a specific CPU list and No CBE pool also But if we we need to do that, otherwise, there is no way to get rid of this Okay, next slide. Another thing is time interrupt. There is a periodic time interrupt that also causes scheduling issues and task accounting we can use this parameter here to Move to a no no Hertz Reduces a lot the time interrupt but doesn't solve the problem there still will be one interrupt per second and That's unfortunate. I know that there is a patch being working in upstream to get rid of it But at this point I couldn't have tried yet. So I don't know if we could do that Okay, all the resources Other sources of problems scheduling issues We have to take out the Iraqi balance because any hardware interrupt on that CPU running PMD thread can Impact we have to use ISO CPUs, which means take the CPUs out of this scheduler We don't want any tasks running on that CPU. We had to disable Watchdog watchdog will periodically check something this has some impact as well power management and that depends on your platform you can try to use this to Limited the state of the CPU to the maximum so at least in you are sure that your CPU are running at least at a certain level Hyper threading depending on what is running on the other thread. It can also impact in terms of cash trashing false sharing and so forth so on So this is important to look at either disable or know what you what you're running and Real-time kernel real-time kernel for other stuff because as you saw if the gas is not running fast enough So we need to take care of the key mood threads as well, especially the ones running the VCPU there We real-time kernel helps to make sure the kernel will not Introduce a high latency there Okay possible improvements you saw that It cycles between the devices. So if you want devices not working or is is lower It affect others one possible solution for this is using the DPDK altred subsystem, which is kind of a Scheduler of threads in user space highly optimize it, but it adds a cost So our budget for line H speed is 200 cycles. Does that worth it? Maybe our devices are good enough. It depends on the case for most hardware that I tried The impact the the old of yes when pushing packets to a virtual machine If there was no space in the ring it would retry for 100 microseconds Yeah, 100 microsecond is really too much then we change it for trying again if it succeeds try Send more packets if it succeeds try again up to eight times It's a lot better, but still you waste time doing that. Should we just drop the packet? Then we are not with the zero packet loss. Should we try again? Then we lose time So there is a compromise there Vhost user has a feature called measurable buffers, which is Buffer in direction to support buffers above the M2 If you disable that you pretty much simplifies how it works. So it runs faster This is changing as I mentioned there was another talk yesterday They did some nice work here Increase the bed size, but again, we saw the average was about 21 There were some tests in the past that show it 32 as a good number above that It doesn't work quite well below that. It's not that efficient So this is something we could look more with as other things change Increase the virtual ring size. This is being done in upstream Bio settings. This is very very very important each vendor each hardware vendor has some features there Some are not really clear if they are related or not. So we had to talk with Our hardware vendor to see the best possible options. If you look at the DP DK side, there will be Some chips there on what to enable especially for on Intel platforms one obviously way out is a faster platform is and CPUs, right and One thing that we need to do more is to improve the CPU resolution and this is How to make the PMD thread really wants the CPU and not having any Interruption, okay Questions, maybe I went too fast, but I can scroll back the slides Well, we are mission. We are trying to push as much as many packets as possible With the very small packets. So if we can get There is no queuing in or very small queues The what do we measure the latency? It's not what the hardware can provide to you, but it's a the question was Regarding to latency if if I have numbers to show about that we are still looking at the There are measurements about that but do it to other facts like packet drops We think that first we need to get rid of them to rise the the throughput and then we'll talk more about latency But so far latency The other team that measured that Told us that this is not a problem yet. Yes, this one. Yes Yes Okay, so the question is if I try to use multiple PMD threads Kind of having this better distribution. So I could improve this by adding more PMD threads. Yes, we try that Basically, you can do two things either you have more PMD threads and then you distribute the ports or you can increase the number of queues And then distribute the queues for instance, you could have four PMD threads and For each of it processing one queue for the from the same device So the host user would provide you for queues and the traffic will split equally between them It helps to improve the it's pretty much scales linearly So as you add more PMD threads, you have more power to process traffic But on the other hand you have one more busy core that you You cannot push it to idle for instance. Yeah Yes So the question is if I try to put the two PMD threads on the siblings Running on the same core. So basically we would Leverage the the the fact that those two were running on the same core and they could share caches It helps for some use cases, but might not help in the other use cases since it's a forging data plane here and Depending on the actions you are executing you might have some cash trashing in the middle So in this case of the my data pain programming were just two rules So it most probably would help but in the real-world scenario We need to try that. Yeah Yes, Nick you mean for the network card because right now the PDD Okay, so the question is if if I could name some of the Intel features that allows the PDK to work okay, so the PDK started as an Intel project and in the beginning was limited to Intel cards, but right now we have other vendors So most cards at least if you the most common ones are supported and the feature is pretty basic I mean it depends on the other feature which is called VFIO or UIO if you don't care much about careless about security, but you the common network cards So you can use yeah Yes, it was a good server, but not high-end one. This is a Young I Don't remember exactly the models we bought as well. No, it's not as well mine is a bit older than that So see it's not even as well. Yeah, most probably a bit. Yeah Yes, basically you have to the PMD trust needs to be ISO the question was sorry the question was if I have to What can I do or if there is any recommendation to isolate these threads from other things? Yes There are documents in the DP DK for instance, and there is a guide inside of the open the switch project Install dash DP DK and there is no it's not it changed So it's DP DK and there is a DP DK advanced that guide these advanced that guide will give you Good chips to how to get there We try to keep it updated because we also want other people to look into this and help us to improve the performance Yeah, and for the the chemo Libby virt has support for CPU pinning so you can tell where the V CPUs will be running the threads that Handlers the V CPU will be handling and then you can isolate those threads in the same way You will do for the PMD, but for the chemo the problems that chemo also And yet at some point we'll need to do these calls and other stuff So that's why I mentioned in the kernel with erity real-time support Make sure that it not it doesn't get stuck without locking for instance Red Hat does provide the erity kernel You need to talk with our Organization to see how to get that. It's not I don't think it's part of the standard row Yeah, I'm sure that's not part of her But you can get that package from the upstream. I'm not sure it's really I'm not looking into that But I believe the patches is our public and you cannot compile yourself I guess there is this guy here that knows about that. Okay, so Yes, that's on our to-do list. I haven't tried yet, but it's something we should look at yes Yes. Oh Sorry, I forgot to repeat the question if we look at into cache allocation technology for improved this Yeah, you you will have to look at your season and Say how many VMs you want to run and isolate them You cannot pre-isolate them and let the CPUs available So when you start the VMs the CPU will be there, but I agree with you It's something that you it's not really easy to isolate everything right now That's one my last bullet was to help improving the kernel isolation at least for that CPU Okay, I run out of time, but I will be in the highways if you need more. Thank you Sorry about that. Oh DJ