 Hi everyone My talk will target an Audience of just system administrator or network administrator. I'm not a developer that will I believe everyone should understand this talk Not like the preview previously with lots of strange language on the screen. Okay, then I'm Olivia Koshara be I'm network engineer at orange telco French company One of my second role in orange is I'm doing the support for about 300 the firewall PF based on Second job I've got is I'm a visiting lecturer at Ren University during few days by Earth when I give Network lesson, but all based on free BSD On my spare time I am the author of funas, but I'm no more involved to this project since I've given to each system And I'm free BSD port commuter on my current playground is a BSD water project with just a simple free BSD tunes for routine stuff Then working on this project I had to to tune it for networking when I had to build some network Performance ladder I reuse The concept of my colleague at orange and I try to do it But just with software or not with very expensive XR stuff like that Then doing my talk I will present you lots of benchmark stuff benchmark result on about benchmark. There is Very interesting talk from random reg In his talk I didn't attend but I read the slides as one slide which says that all benchmark are wrong And I'm quite agree with this one on this will include all my talk to and I will explain you why First before to speaking about benchmark on tuning. We need to be agree by the unit we will use I'm a network guy. I'm working with networking stuff then the Router have two main job. The first job is forwarding. It's mean he need to Forward packets between these interface the second one is maintaining his voting table with voting protocol But I will not speak about about this Then our main unit is a packet a second unit. It's the only unit you have to use when you are a network guy Second stuff there is an RFC explaining how to correctly the methodology of benchmarking a router. I Try to follow but not perfectly this RFC and I will explain you why later Now that we are agree about the units You need to understand some reference units on a gigabit interface The maximum line rate is the one hundred forty eight frame per second We use the The term frame because frame is for a standard layer on packet just on the IP layer But in all my bench, I'm using small packets and one packet is one frame. I will use the term packet per second on another big detail that sits almost everywhere forgotten. It's a gigabit Since gigabit is a net there is two wire one wire for transmitting and one wire for receiving It's a full-duplex media then Terrorically you need to do transmit and receive at the same time This means when you are speaking about line rate gigabit router Your device need to be able to forward at three million about three million packet per second when this is But this is about a line rate router But on 10 more for 10 gigabit router, but who needs technically speaking line rate Then I need to introduce you another concept Which is how to translate packet per second with boundaries It's quite easy You just multiply your pps with the packet size, but on the internet Each packet has the same size Then for simplify Long years ago, they choose distribution a standard distribution which is for one big packet You've got Four medium packet and seven small this called a simple internet emix on using this Distribution from your packet rate. You can Estimate or give some value using a standard reference about the equivalent in bond with rate But I'm network guy. I'm not using the One with the IP layer then I'm using Bonn with as a recent layer I need to add the internet overlay only in my slide if you will see some table Referencing one with it will be as an internet level Now about who needs line rate gigabit or 10 gigabit router with these two concept packet per second on this distribution We can Estimate what is a minimum rate using the simple emix distribution? For reaching for filling your gigabit link or your 10 gigabit link at full duplex And this is the seven key recall 700,000 packet per second for a gigabit water on seven million for a 10 gigabit water on my lab I Need that all my device reach at minimum this value for be called just a simple gigabit water or 10 gigabit Not line rate, but a simple gigabit Now how to I how I did my bench We are a telco at a ranch we don't we are not we don't sales hardware then I Need to take care of my customer. We don't have any problem We have a telco. We don't care about what the customer will do with our link Then if I install a gigabit water I need to take care that this water Teorically can receive the worst case possible. It's been a line rate traffic using smallest packet size on this Why my bench are different from the earthy? My bench. I'm generating the worst case ever In one side on on the other side, I'm waiting and I'm measuring how what's the rate during this worst case It's very different of the of the unit you will see and for some material of the manufacturer Now on my lab a little more details on my hardware lives. I will use for all my talk I'm using I'm reusing all stuff I get in orange and I will do lots of test my biggest server is a Xeon a core with Hombedded a mulex card 10 gigabit card on the Chelsea o-card to give it I've got two super micro with eight core atom One with Chelsea on one with until this is great because I can compare the behavior between Chelsea one until on two small One that gate on one PC and Chinese IPU this for my gigabit World I don't have melanox. I want to test them. They are very good review I don't have for them a month sex career one circuit CPU and I don't have multi-circuit CPU Because but this is can be interesting too. I'm about to detail all the SFP 10 gigabit SFP are the same everywhere It's not a matter No before tuning a freeBSD or every kind of server There's a first concept that you need to understand is how Today a standard Network card is working on a freeBSD Today when your driver loads the kernel give the information to the network card how many CPU you've got And with this information The network card will create a multiple queue with the maximum value depending of the driver Then each time the card will receive a packet. He will use Hushing in this packet on each packet belonging to one flow will end into The same queue each time. It's a very important concept in network world none too For the whole packet belonging to the one floor need to use to cross the same path Then from this first basic concept now We can have two very important points Like I see when I read the freeBSD network mailing list that lots of problems come from here the first one is you need multiple flows if you are If you are using just for example IPR for just one IP source or one destination IP source with the same TCP It's it will not work. So you cannot use this correctly On if you are using some stuff like IPsec or GRE You will break this concept to has endpoint on your router or firewall. You will break this concept to and lots of people forgot that Has example in France. I've got lots of people using a PC Engine's IPU with a BF sense on their home fever, but the high SP which is orange For them to use PPPOE and once I did that They don't have any more multiple flow They just using one core on the performance of a very big impact. This is a very important step You need to to understand So second is what today is a CPU is a physical core logical thread or what about new map? This are some question I need to answer Then for helping you to before to try to tune You need to ticker need to check how many what the current Q How many Q's the traffic are spread between all your Q you've got this information if you check We see settle and free busy if you get the network statistical information They are not very easy to pass then how is the script from young Lex? Which gave me which a piton script who allow you to to in real time to see how many the cure are It's very important stuff for for network guy And then you need to check this step once you have you are sure that you are using multiple flows We can start to to work together second step is about What is the core today? How is an example just on my Xeon Hacker with a Shelshaw card when I boot my server is default mode You see 16 CPU eight peer to trade The Shelshaw by default driver choose to create only 16 on maximum of 16 Transmit Q one for its CPU right, but only eight with you of Q. I don't know why eight. Okay. You choose to eight then here I said, okay. Here is my first bench now How I'm doing bench I'm creating three configuration sets one of the default value a second one where I will force my Shelshaw card to use the 16 Q available CPU available and the latest one where I just simply disable it per threading My test I'm using of all my talk. I will use the freebies the 11 at one release All then I all my script are benched. I know you mind interaction then I will push a configuration set I will do a benchmark with picket again at the end of the bench I will reboot again, and I will do it five time with a reboot between each time All these benchmarks is very important for statistical analysis At the end I will have three text file for each configuration set on each Each this three five I will have five five value for my file value of my benchmark on now a little bit much With this three configuration file and put them into mini stat mini stat is a great tools If you need to do benchmark, it's a mandatory tool to use, but it's very monetary and he will analyze Your different input sets and you will check if okay if it's very you really have an impact or not On this example on the right part for example, you can see that by default. I've got the worst result It's on my left side and I When you can see with a distribution, okay, there is no arguing about it's very worst I reach about four to six million packet per second if I force my car to use a 16 CPU I've got a little improvement, but if I just these other hyper threading. I've got the biggest improvement then Okay, the first thing is okay You need to do some hyper threading because it does help you anymore on this case On this use case because you could have lots of interrupt on threading don't help you to manage all this interrupt The first the second news. It's it's a bad news This device is my biggest device when I sell to my boss. I will do I will create you a 10 gigabit water This means seven million packet per second and I've got only five that seven Okay, I need to find a solution I Got a second question once I get this result was Perhaps if I will had some core a bigger server I will solve my problem But before what's the relationships between the number of Q then of core assigned on the performance on my server then I've generated eight prefixate I forcing my Shell so far to use just for the first just one Q and one CPU on up to eight CPU On surprise with a reference blue line rate. It's a linear reference We've got a problem. I can't had CPU, but you will not solve a lot of my problem There is some stuff some big difference is does the scale today Okay, I'm a simple and users how on free BSD do you dig this kind of problem It's quite easy. You just need four common line on this kind of four common line. You generate some stuff called from graph plan graph it's Give me the How to say the hot function? How to spend how to consume the CPU time? And from this graph I can okay I see first for interesting point the first is Okay, that is a lot contention about the first function with this code IRP resolve This function is used just for adding the destination MAC address to my packet once there are forwarded So second one is a routing table Okay, I've got only two static route on my bench I try to do some stuff using a true No change then but this first two I have and users I can do nothing about it There was a second one called a random RFS queue Hmm, I will dig about it later on the last one is you see the all the part building to the Isolate pass the kernel is an ad pass on the next driver Okay, what can I do? I will try has a end user just to play with a random RF source on the free BSD as Entropy source by default interrupt on or isn't it packet I read some data are taking from this packet For using but at this very high to write this impact a little bit your performance Then I've just choose to disable Interrupt on isn't it packet frame from my configuration I've got it's just with this command line You change the input and then I generate a new bench, but in all my server in my lab And this is a result with all my server Bad news on my big one. I've got no difference at all I've got some improvement with my eight core atom a little bit my Small gigabit router see some improvement to First conclusion here is that about okay. I've got two small gigabit router Who are correctly able to do gigabit forwarding for my point of view? Second one is okay. I still have a big problem regarding my 10 gigabit water Then how can I do as a simple end user? It's very easy just call Google on you will found that this are known problem. There are already fix since few years by a crazy team from beyond X There are already have patches about that on you just have I just have to take their patches on to apply it to my Free BSD. I just backported these patches one for the two look contention. Okay, let's try Then I keep my harvest mask modified. I apply this patch to my labs. I know her is a new result. Oh surprise Almost twice twice more Performance on my router which now okay my manager will be happy Second impact is go. There is a 10 gigabit Nick into this same server, but it just reached 1.33. It's even not a gigabit line rate This card. What's this card? Okay? If you dig a little bit you see that these cards you can configure the number of q you cannot configure flow control. It's a very Bad nick Then you need really to take care about the new card you will use Shells you are very good until to I Didn't have Melanox for proving it, but they are very very good review by the Melanox one Then really try to to use good network round Oh, no with these patches. What about my linear performance rate? I did this bench again and you can see that the green line is Is lots more better than previously Now perhaps if I can add some core and perhaps I can have more performance the second one is quite funny because Between four core on a curve is a six core, but this does not scale It's like we need to have a power of two number of core. I don't know why I didn't dig this But it's a funny value. I was waiting more for my six core in this example Now, let's how can I try to improve more my informer? I need to to works of improving the performance perhaps Regarding the Network card on the driver of the network cards themselves today driver are using lots of feature Then I try to to look the main page of the driver. I said, okay, I am a bit lots of checks and stuff On there a bit Two feature that are very great for server for endpoint server Which is called TCP cementation off-load on large receive off-load But these two features are very bad concept in a routine world. I believe even in the firewall world Then This is not technical for a plus technical but more philosophical choice I don't want to have these two features Unable it in my lab when I try to read the main page then I choose to disable this feature in all my lab and I try to Generate my bench again for checking if I don't have a performance. Okay, this confirm There are no performance impact at all About this this challenge then really under firewall or router you should disable these two feature the second one on this slide You can see that's on my hate core Atom I've got a quite difference between my shelsio card on my entail card About one million packet per second Then I try to look a little more how I work entail card. What can I do in about the entail card? I did a lot of tests. I died long document about Unterip management about entail with lots of contours I'll try lots of things but Like I say all benchmark are wrong on my use case my bench is very specific With my line rate of packet when I don't want to impact just my very specific benchmark Or not a standard use case then I didn't Works more on the entail driver But at the end I found just one parameter that that matter This parameter is by default Gigabit entail card or 10 gigabit have a maximum number of packet the process Per per what per time per I need to dig this one And if you remove this limitation if you just say okay, don't Don't keep a high value about the number of packet. You can manage it in one times. You can you found a little improvement And this is okay. This is only by checking how the user point of view and the driver This is the only stuff at all I can find Then I speak about with other on the same event what about Queuing painting your multiple queue to each CPU. It's mean you you prevent the scheduler to change the CPU of your IRQ I Check the driver, but all almost lots of driver already by default Pins the queue to the to each CPU with an exception of Chelsea, which is interesting. I'm using Chelsea Is this function, but it's not by default on the generic kernel is not enabled then Again, you indexed give me a script or was there already did all this stuff And I say, oh, you can perhaps try this script. It's a versus script when you force the binding. I try it There is not lots of difference It's just you start to see benefit just with very high throughput more than 10 million packet per second For example, I might a core. I've got a little little benefit 1% It's why it's not enough to to take care about on my Atom a core. I don't see any any difference at all. Thanks. Okay. It's not Francis use case is not very important Then, okay, I believe just for rooting point of view I which has a simple user the maximum I can reach this mean, okay If you want to use freebies the other router, you really need to check the patches from the index or for this team There are online or there are people that freebies the page if you know the name or you need to disable hyper threading The harvest mask need to be fixed to And there are working progress in this direction. And if you've got yes on TSU LRO, you can do it There's no impact on the performance On an Intel card you try you can try to play a little bit Then if I raise you just for forwarding Before the tuning on my patch on after we can found some okay a Good improvement on the Xeon 8 core with a service you a little less with the other hardware On this mean, but okay. I reach my goal, which is okay I've got a Xeon 8 core which I can call 10 KB router My 8 core Atom don't reach this statues. Okay. It's a pity because it's that cost our very ship hardware On my small net gate on my small PCN times APU are great for Gigabit routing. This is it's a pretty small code machine But I speak only you but IPv4 But in the wrong you are doing IPv6 then let's just Again Yondex made for me a patches to packaging it back packaging for adding IPv6 Support on I can generate IPv6 packet in place of IPv4 surprise ouch Okay I've got a quite big impact Hopefully I'm still Over by seven million packet per second, but the impact is quite It's quite large on funny things here on the shelves in the 8 core atom There is no more difference between the shell see on the Intel because it's a kernel who who lose your all your Performance is on the canal not on the price on the driver anymore Okay, this is the impact with just IPv6 We need to work a little bit on this part Then bad customer have Use standard course setup Difference and mine one of these for setup is up lots of them are using valentagging Um, we are using at orange another setup, which I'm calling multi tenant rotor Which is using vnet jail then I want to check the impact of these two configuration set on my configuration For valent, I'm just generating okay my standard reference config without the land on the second one with valentagging It's just a very simple difference On today is a network interface if you check them they unbeat lots of valent accelerating stuff Then terrifically They should be have no impact with my pinch surprise again 17% of degradation Okay, that's a lot quite large impact on this it's but Again, it's a known problem and again young dex use already a patch. It's a long time on their server by this problem, it's it's regarding the Other free with this tack when he's got a target frame. He used a long pass for They are encapsulating this frame and once you remove the valentagging He takes a frame on he sends them back to the beginning of the ethernet for for a standard It's quite not very efficient We've got a big impact on here then okay lots of customers who are using the land tagging perhaps would not be happy But I'm still over my seven million packet per second. It's still okay The second one is my multi tenant router. I need for that by default you need Vmage It's for the virtual network stack into free BSD. It's not enabled by default Then before to create a new configuration set I need to add this option in my kernel Then if I had to add an option I bench it on the result with this option enabled is here in I lose some performance in IPv4 and I ring a little on IPv6 But it's very small if you compare to to the other bench with the valent then it's it's okay It's quite okay 6% is okay now The net jail On my server, I will move all the IP configuration When I move them into a venet jail This means has no more IP address for my on my hardware my host all the IP configuration Into the jail one I can play now I check I play on the PC on giants IP u2. I play to generate I don't remember it's between 100 on 150 jails venet jail on this small hardware. It's very fun Then what's about the impact of this setup? improvement degradation It's quite No impact at all It's quite very very very good surprise Venet now I Tried with firewall IPFW on pf because it's supported to On the in the same as no impact at all. It's quite once you enable the image. It's quite a very good good use Venet jail works very Okay, no, okay. I'm Finish all my forwarding bench. Let's start playing a little bit with firewall Okay, I will show you a graph After all this slide my first step is just to enable firewall that's we have three firewalls in free busy and I want to check How hard just enabling this firewall Unpacked my performance The next graph will I will not compare the firewall. It's very important Destination I will just compare how by the impact my routing firewalling Then okay, all bench are wrong then don't interpret this benchmark by Saying that's Pfsx in front of IPFW. It's not what this bench mean. It just means that okay On my Xeon 8 core the reference on the left is my forwarding rate at EPV4 and EPV6 I generated Stateful on stateless setup is a minimum number of rules one or perhaps two for IPF the old one And I just try okay. How about this impact? Bad news, okay If I want to sell a 10 gigabit firewall to my manager, it's start very bad Perhaps IPFW in stateless mode. Okay, but about the all the others It's the impact. I don't compare fire or just the impact is quite a bag No Let's dig a little bit about what is technically speaking a firewall Let's do a next bench with in front of my Hello all rules my unique hello rules I've had lots of denied rules with an unexisting IP and I love that try to play Okay, let's try it with one ten one thousand one hundred Rules You can see Still on Now I switch my hardware. I don't anymore use my big Xeon 8 core. I use my smallest One of my most it was a negate. It's a four core atom Why because like I said, I'll do a reboot between each my bench on the reboot and a big HP server Takes a very long time about five minutes on my bench is about one minute on its It's take weeks to do all this bench But the negate on IP are very fast to boot on I'm was I'm more interested by the impact or not by the full value Then Harry's we can see that IPFW on EPF Our linear relation between the number of rules and that break their performance very very easy Then the first rule is you need to keep the minimum number of rule, but very small at 10 You start to do to hit your performance just 10 rules The second one what's it's my pf. What are you doing? He don't care about to find my rules Then I dig a little on this tricky guys Covert all my rules into one table then all bench are wrong. This is a wrong bench I'm comparing April on pitch because once is using table or not the other one. Okay, you want to play with table Let's play with table For this next bench. I just I remove all my deny rules I just have one denied table rules on one allow all rules when I try to increase The number of IP into this into this table on okay using the same Feature we can see that we are agree Really use table if you can really it's very important. The impact is is huge Really really choose Other stuff is a good thing exact pay F It's lots of lots martyre by default because if you will do that in your place Then you can save your brain for doing other stuff Now I play a lot of with stateless Now let's play with stateful setup on my FW stateful. I try to dig a little about the default configuration setup of stateful IPF and if you check if you my packaging generate only UDP unidirectional flow and if you found one flow IPF create one Dynamic rules. It's one states on IPFW If you check this by default he accept a maximum number of 16,000 rules When you've got done it's for this size is related to the hash table size It was not very clear in the beginning when I start this I discover what is a hash table size and Then I try to increase this hash size tab to the maximum allowed by IPFW with this 65 Thousand it's a power of two and then I estimate the number of potential Keeping the same relationships The perhaps with this value. I can push my IPFW to manage four millions UDP flow without any problem Then let's try I generate lots of multiple flow by using a large source range or large destination range of my IP packet and we can see that the you can hit starting at about 10,000 you can you can hurt a little more of your performance on the one million It's totally out of Then I cannot reach four million with a IPFW Okay Just for information when I show this diagram to the young next team the next day I had three patches luck less The crazy guy, but okay, I need to do it. I didn't have to didn't have time to do it for this for today now with PF PF have a different behavior for one floor PF consume two states one for each direction On There is a linear relationships and PF have the same number of state limit like pay on the same hash table size Then on my small net gate device. I've got 8 gigabit of RAM then I try to Increase lots my hash table For using the maximum value I can which is about a 2.5 gigabit of RAM use for this hash table on this mean about okay, I can I can reach 10 million states then 5 million of UDP flow in my previous with my bench on Here is the impact with PF Okay, he does not care much about the number of states Then really if you want to use the state food firewall with complex rule, okay This why you know now we perhaps use PF It's quite a very good a behavior for this for this kind of stuff No, let's back in 2002 When I present for the first time my BSD water project I Complain that we don't have MPLS feature and has a telco In a run you are using free BSD as a firewall because it's a no go if you don't have MPLS support on Netmap was existing in 2002 and I was waiting for okay Where are the software the user on the software who on the free BSD using net map for routing stuff? now today still no MPLS but the concept of user space forwarding is Is a way to go today if you want if you check today At orange for example Preview this solution when I speak with my network Colleague if I want to install a software router, it was it were no go I cannot talk with them about a software based on Linux or free BSD. It's a no go But today they are removing all the Cisco or Johnny Perth stuff and they are using software router but based on Linux because it's Cisco fdao and dpdk server and on this if you want to compare Then we need to reach 12 million packet per second per core. I didn't even reach this with eight core on this is It's quite Very interesting gap To to reach on this why why not of people ask me? Hey, can you add to your benchmark? Can you comply to append BSD or can you comply to a Linux? It's I'm not really interested by that once because Like you see I will never compare a standard Non-tint free BSD with another stuff because it's really crazy Then I need to learn how to optimize a pen BSD and a BSD a dragonfly and I haven't half time to the for that because You need to compare April with April then if on Second tough if I want to add a Linux in my lab. What kind of Linux I will have what distribution? Then I know my my only interest is okay now We've got a value a reference value to reach on I prefer spend my time to help people to reach this unique value We will see after our free BSD there were from net gate Software called net map forward. I Don't think this is maintained in your own Github. I don't found I don't know it I didn't reach it to to use it On I don't see lots of core of lots of code updated to it This way I'm I'm saying that no production ready software on free BSD This is a shame because we are losing a role really it's At last I'm very happy to see my Cisco or Johnny Perth stuff replaced by software But it's a Linux. Okay But all is not lost if you Can help space warding is still improved and you say young decks have a lot of longer lists of patch very interesting It's an old project called project routines lots of old ideas that we need really to refresh The real talk last year about how to improve IP for adding fast past an interesting talk because reading this I discovered that User space forwarding have some problem. I was not aware about and Yes, perhaps Okay, I believe we can still perhaps place this this game again on about my All my bench script my all the script I've used for all this stuff all the configuration step I use all my result all the Flamm graph about when I generate my graph After I generate a new bench just for generating Flamm graph then for each difference It can help developer on the lots of the script you can it found on them on the busy Web page or get a page are all all online Why because I'm very afraid when a free BSD developer send you a patch Force ask you can you test this patch, please? I believe it's improved I'm very afraid of doing a mistake in my lab or giving false value to a developer Then this why I want to lots of more people reading my script on my configuration stuff or my way of doing bench because it's quite stressful to To give false information of bad information Okay, now do you have any question? Yes Is it possible that this is because of the association between course and I guess by design you will have even number of cues So if you use like an odd number of course, you will have a mismatch Is there a developer who can answer to this question? Can you bring the microphone? I Know just the very same and is actually any power of two Yeah, but until selling some six core CPU then try to avoid them on your water This is an advice I can give but I don't understand this. I need to work with the developer for that. I'm not Okay Ah It's mr. Nedgate We basically stop work on that map or because There is work underway If he's Then you are working on the user space forwarding using the PDK on free BSD Great Great, I'm waiting this very do you have a schedule? I Are the patches from your decks will be merged I Don't know if it's a Russian culture or not, but they are very Each time you ask that I say my patch is not ready. It's not clean enough There are perfect tonists person, you know then because I'm not a developer My role is to push them on to do some Marketing stuff for them for trying to push this patch. Then yes, I'm using is their patch in my protein with DRP I'm testing it on the real field or now I can push them for saying, okay, I Can give I can say okay. I use them. They're working then you need I push them to the free BSD project You can comment about what you said about TSO and our for your workload They should not matter at all because you should never receive a TCP flow in a router. You never go above the IP Yes Yeah, but you can turn them off But if you don't turn them off if you keep them It's a TCO TCP all your stuff the firewall have IPFW for example the libalias the net library does not work Then it's it's it's in the main page on user of problem with that Then my advice if you don't have it if you don't want to have any problem. Yes on the router Yes, it's it's logical. We don't have to Have negative effects on the rest of the system The other thing is that What Olivier is looking at is performance, but not always correctness So they have a cash is often need a bit of they're right They need to be fixed so that they don't just remove locking they still need the correct Oh, no, not you The problem is when I did my bench I did a third one by disabling the acceleration stuff But it's not a correct bench between before We need to fix this villain and we need to fix this villain Problem into the candle pass on the ones. I will test this patch I can now generate a new bench without the hardware acceleration for Real point of view about with or without Currently because the kernel pass already slowed on the rate You've got a worst result with this feature of this a blade but But it's not very worse because it's a kernel who already break your performance Then I will do this. Yes. I will check with the feature on with your feature off Once I will check the new patch from beyond X. Let's fix this problem Yes, and the villain and you had The number of the villain into the packet On this number when you receive a ticket from then the network have to remove this This tag for example, and this is how to on just the number of villain Is that you need to remove this this tag on use this tag for the virtual interface Holding it's a very small Action really technically bad. It's it's matter Yes, hello, hello Packets of what out of order delivery No, I'm using for the moment. I'm using that map picket a gen which is a very dumb Very simple tools just it's Daniel of service tools if you want if I want to measure Out of order packet. I need to have a more smarter tools that Will read all the Index in each package generated or we'll compare for that at all You've got X here, which is very expensive hardware But I don't have it for the moment in my lab But I need to check as I the Cisco stuff based on DPG key called T-rex and I need to check this software for For for this kind of more advanced performance test. Yes, really I need to do it Just behind you Really From the kind of space it's No, because I don't have Jim perhaps Can you get a micro to Yeah, but we need to fight against this solution Thanks for the talk, I just wanted to know about the Simplicity of begin a gen and you mentioned fighting It's actually surprising to say but you can use fighting and fighting findings for a net map to give some smartness into that map and and for example like for checking off the Of the ordering stuff like that multiply with that by site and you will give You will have a decent performance right now. Oh, what did you find this? Well, I'm from Intel and we're doing you I'm a network software engineer. I'm the guy that a zone drivers. Okay, we're talking about and yeah, we do use net map for different things like for for example for by the way flow control testing. Okay, and Yeah, we use that API to integrate into our Python written tools and they well I just you know, I thought it would be a huge performance drop when you use fighting to manipulate the reins But wasn't really it was it was decent You're creepy does your script is online? No, it's internal open source. That means that it's shame shame on you Anyway, you can here's your sleepy Thanks