 So my name is Ray Kinsla and I work on Fido C sit and I'm a software engineer working for Intel I'm also a Fido TSC member Fuck I should probably ask what a Fido TSC member is Back when I started on open source you just flipped stuff on the source forage and That that was that was opens. This is before github and now we have oversight committees We have marketing committees and we have technical steering committees and there's a technical steering committee for Fido And it's called the the Fido TSC and I'm one of the representatives on the Fido TSC And we have input into the technical direction of Fido and Fido is And a collection of projects Fido is really three key things Fido is VPP, which you heard about earlier VPP is a is a high-performance network stack Built with the same kind of optimizations as DPDK. So DPDK is a very efficient layer for IO Fido is a net Fido VPP is a series is a network stack is a series of protocol implementations on top of DPDK So that's one part of Fido a second part of Fido is the integrations that gives you so and you talked you heard Ratislav talk about it earlier and Giles talk about it earlier And then you also heard Billy talk about the same thing which are all of these Integrations that sit on top of DPDK. So with DPDK, sorry on top of Fido So with Fido, we don't just flip a network stack out the door and say you guys just kind of take care of yourselves We take care of the integrations into Kubernetes and we have so many integrations They have guys arguing with each other about which is the best way to integrate Fido with Kubernetes And there's I think Billy's presentation called out very well. There's three different ways to achieve it, right? There's integrations with open stack I think Giles talked about integrations with opens and with strong swan was it Giles strong swan And we have net Conf and Yang support. So we have all these integrations on top So that's the second part of Fido is integrations and the third piece of Fido is what I'm going to talk about today Which is benchmarking now usually I'll take a software. How many software engineers in the room? Many software engineers. How many test engineers in the room? One two three four, okay, so that's probably like a probably like it's 22 There's probably like it's probably like a 10 to 1 so Seesit is hugely important in Fido Seesit for Fido is is is how we maintain and we ensure that the Fido data planes Performance does not degrade over time So what and I and I'm not you know, it's no commentary on any other open source project But the danger when you move into open source is that when you get when patches come in the door You need a very very very tight and very elaborate CI and CD Infrastructure to make sure that as you take new patches that you're not trading off performance patch by patch by patch by patch And that's the important role sees it plays in Fido and it's a significant role and that's what I'm going to talk through Today, so what were the tree keeping tree tree parts of Fido? Three question question. I know I'm picking on you. It's like four o'clock and everybody's like we've been to the bar already Ray, you're asking this questions now. What do you do? I said there was three parts of Fido One was the data plane second piece was though up integrations third piece was the benchmarking This is the benchmarking so today. I'm going to talk about why code is not all no longer enough And this is essentially why benchmarking is important I'm going to talk a little bit about Seesit what it is and then we're going to talk dig into three interesting Problem statements with Seesit or three new problems. We're trying to solve with Seesit And then I'm going to summary have a bit of a summary So today you heard about a number of different open source projects. You heard about snab from Luke You heard about open v-switch from Kira and Bruce What else did you heard about Fido from Giles and Rastlav and Billy McFall and you heard about Sillium and who's presenting on Sillium? Yes, gentlemen there and then we have a and then there is a few others that you may not have heard of so there's a galaxy of open source projects and if you also learn more about the galaxy of open source projects I'm giving a talk a 20 to 7 and if you're not already in the pub You can come to the lightning talk room to learn about the open did the The user space networking ecosystem I'll probably just be presenting to myself, but there's a galaxy of open source projects in user space today There's never been more choice Well, how will it perform for me? you know You know, there's a whole bunch of typically vendor reports from the Ross Intel Melnox produces them. There's a lot to arm produces as well. There's a series of academic papers There's blog posters YouTube's but you always come back to the same set of questions. How are the benchmarking are the benchmarks repeatable? Are the tests quite applicable to my use case and if I had a dollar or a euro for every time somebody said Yeah, but is it a real-world use case? I'd be a very rich man You know and the difference between synthetic benchmarks versus real world use. I see Javier smiling there Our tests for platform and vendor neutral and that's kind of an important one to what extent are is vendors You know skewing the skewing the game to their own interests So these are all Important questions and they really all come back to how do you evaluate? Software data planes. How do you test and how do you ensure neutrality? well Without data and without the consistent benchmark, you know without data You're just another pen a person with the pinion and you know, this is from Edward Deming and Myself a match I chose this because I think a kind of illustrates very well our sentiments towards sees it and again That's not a criticism of anybody else But it's this is more of a statement of our intention the intention of Fido sees it is to bring volumes of data That reflect the kind of performance you're going to get from a Fido deployment in real-world use cases So to bring significant to bring that kind of clarity so that when you're asking the question about How will it work for me that you'll have the kind of data that you need to answer those questions? Now we're doing for time. I'm doing okay so Fido sees it is a sub project inside Fido. So if you go to the Fido website and You navigate through there is a whole bunch of sub projects and there's one that's called a sees it Sees it CPL sees it stands for a continuing system test system and integration testing. I think that's right. Yeah That's my first fail of the day So what is Fido sees it Fido sees it uses standard industry benchmarks and tools So we everybody's heard of RFC 2 5 4 4 I think that RFC 2 5 4 4 will be written on my epithet when I die Okay, and the metro ethernet forum standards we test ranges of packets we test 64 by packets we test All the all the packet sizes up to 15 and 18 we test I mix we test Jumbo frames so we test a large Lot just one or two different sizes. We test all the different sizes you might possibly use including jumbo frames We use only open source tools We use only open source tools our traffic generators t-rex, which just happens to be another Fido project We used everything's orchestrated and run by the test robot the robot test framework And then we use Jenkins as the CI so Jenkins takes care of running the nightly bills and running the tests and those kind of things We test multi-core scaling So, you know we test with a single core we test with two cores and we test with four cores And sometimes we test with even more and I'll talk with that about that later on we test There's a typo here. We test NDR. It is non-drak rate non drop rate Sorry when the NDR is when you don't drop packets PGR, which is partial drop rate I think we we have a tolerance of 0.5 percent packet drop and we test MRR Who knows what MRR is? Maximum receive rate is where you throw the kitchen sink and you see how much actually gets true We test the whole range of different network for the whole range of Different network functions things you might be likely to do L2 switching IPv4 IPv6 routing ACL security groups Overlays and we have a slew of unit functional and performance testing Everything's open and fully documented test environment I want to say that again open and fully documented test environment that means that and we have lots of people who do this We have lots of people who are involved in the phyto community who actually go and clone everything and Reproduce it internally because it's all open source all the test cases are open source all the tools are open source You don't need any proprietary tools so you can reproduce the whole thing internally It supports Intel and ARM architecture ARM architectures so What do what does that mean? We actually have an open lab that's hosted by the Linux foundation that gets hardware Contributions from people like Intel from people like Mel Nox from the the ARM ecosystem And we have a very sizable lab that's run inside the Linux foundation That's free for anyone involved in CC2 use where all this stuff gets tested And then that allows us to do lots of different permutations of tests We can do it same test on different generations of Intel hardware I'll talk about that later different same test with different nicks Same test on different platforms same test with an ARM platform same test with an Intel platform. It's the same test Very rigorous. It's the same test run through the same set of permutations the different packet sizes Different numbers of cores different platforms different nicks to give you a very very large Corpus of data about how this thing for info runs in the real world It's multi-platform multi-vendor. So we have it runs today on top of CentOS Suzy Ubuntu, I think I'm not sure about Fedora. I think that might be gone. Anyway, and then we also support cloud native You've heard extensively about cloud. They will open stuff also and cloud native is the hotness at the moment I know but it also supports open stack environments as well And we also have it support for some accelerators like crypto accelerators So we have an open lab Where everything gets tested in the open where patches get tested where nightly bills get tested Where release reports get tested all in an open lab in the phoenix foundation and we own use only open source tools And open source tests So if you don't believe the numbers that are produced by the open lab they can go reproduce the whole damn thing yourself and satisfy yourself that it's completely authentic and That's the idea is that we want to drive good practice and engineering disciplines at the phyto dev community So if you submit a patch To phyto and it causes a performance degradation and we'll come on with show you that later causes the performance degradation And the alarm bell goes off somewhere and somebody sees that and we can take action So the idea is that by having this elaborate CICD environment that we catch those kind of regressions Early on I'll talk more about that more in depth later that we also You know when so much of phyto and so much about user space networking is about performance are You know achieving the best possible performance that you can through software well again Phyto CICD provides you the kind of tool chains that you need to measure the performance And then it also can prevent things like performance aggressions So it's all fully automated is integrated into g it's intermay Integrated into the gerrits so that when you submit a patch at all the tests get kicked off We run functional tests performance tests. We have use case driven test definitions So if you want to do things like test things like vxlan termination those kind of things we have test cases for that and It's all get executed in the open environment. I've said all this and integrated with CI. I've said all this that's fine So what does this look like in practice now clearly? You know I did a count Last week to see how many test cases in total I could I could find and I counted 998 test cases that are currently now. These are these are performance benchmarks That are currently being run in phyto C sit on a regular basis now That's if you go through every permutation of packet size across every platform across every nick of course every microprocessor generation across every use case and you can see here that you know we actually break it out There's 144 layer 2 tests or 216 layer 3 tests 300 You know very very significant number of test cases in each area So when you go through all the different permutations the test cases across you come up with a very very large number I clearly check and show the result of a thousand test cases and it won't a you know at the same time So I just picked one and one or two in particular that I quite like So I quite like the IPv4 test get IPv4 routing test cases So and we have two test cases that we run one's called IPv4 base that runs phyto That runs a performance test of IPv4 routing With one route So you've only got one route in your lookup table and then we have IPv4 scale and IPv4 scale runs with two million routes in your test in your in your in your routing table and you can see this this is a Haswell generation microprocessor, so we have two Intel platforms in the lab We've got has well and we've got Skylake and you can see the IPv4 base with One route gets around 11 million packets per second on has well and then it gets With the two million routes it gets well about nine million packets per second on has well, okay So then we went to phyto vpp 18th. That was with 1804 we then went to 1807 on has well We did some software optimizations Came along and we also introduced the second nick So the software optimizations cause a small performance bump from about 11 million packet seconds to about 12 million packets a Seconds for the million packets a second for the single route test case And there's another small performance bump from about nine to nine and a half or something like that for the two million route test case But then we were also at that was these test cases here where we're a 10 gig nick We're able to do the same test case now with the The x7 10 and the x7 10 again is a 10 gig nick, but it's 4 by 10 gig. So it's a 40 gig nick. That's It's a 40 gig Ethernet controller that has 10 4 by 10 gig physical interfaces And you can see here that you know performance is much the same as it is with the 10 gig As with the 10 gig nick But now we have the satisfaction that we're know that we're getting roughly we're getting pretty much the same performance across a 10 gig nick I'm a 40 gig nick And then we introduced the Intel Skylake micro architecture in Fido 18.7 as well And you can see that performance again is roughly the same on Skylake. So we've also introduced a new microprocessor architecture We've introduced a new introduced a new nick and We have that warm fuzzy feeling that performance is more or less the same and then Fido VP P 1810 came along and Fido VP P 1810 made very liberal use of AVX 512 instructions now AVX 512 instructions, so I presume everybody knows in the room knows what vector instructions are I really know the vector instructions. I see nodding heads and I see hands up. So AVX 512 is the latest Is a is a laces improvement in vector instructions? So before this was AVX 2 which can't think came out and has well was it? A VX 2 is has well AVX 512 is on Skylake Which basically allows you to process even more packets Simultaneously is the idea so vector what vector instructions typically do or how we typically use vector instructions in both DPDK and Fido VP P is we use them to process packets in parallel So if you are using let's say a scalar instructions You typically do a one packet at a time if you use vector instructions You typically do four packets at a time with AVX AVX 2 we typically did four packets at a time with AVX 512 We typically do eight packets at a time if memory serves me correctly, and I hope I'm not wrong Well, what happened when we introduced all the AVX 512 optimizations into Fido VP P on Skylake You can see what happened to the baseline performance there. We went from About what 12 million packets a second on 80 and we're about 12 million packets a second up to something like 19 million packets a second Just simply by using this By using that this power inherent parallelism the microprocessor that AVX gives you Similarly at the scale test case with the 2 million routes went up from something like 9 million packets a second to something like over you know to over To over 16 million to around 16 million packets a second So you can see here the general point and I'm probably labored it now at this stage But the general point is you know you can look at the same test case across a series of nicks and across a series of microprocessor revisions and see what the impact of both of you know What the impact is without optimization the impact is with optimization across a whole range of 99 nearly a thousand test cases So this is how we bring a real breath depth and Predictability what I mean by breath is the breadth of test cases when we adapt the measurement and I mean predictability We do this for every patch. We do this nightly. We do this for every release and we generate a huge corpus of data as a result So I think I've got ten minutes left So I'm gonna talk very briefly about Yeah, there you go Let me draw draw back for a minute any questions. I'm what I've just said go ahead Funny you should ask let me come on to that Any other questions go ahead so not actually nightly Nightly nightly not every patch because that just becomes insane to your point, right? You couldn't do it every path Okay, sorry. I'm just I'm being told so and you're asked the question you're asking is how frequently we do run the entire test case Yes, so I come on to that in a bit of depth. So let me let me come to it so and we talked about this and Previously which is the continuity problem, which is that If you find performance regressions at release time, it's typically very expensive anybody here and there's lots of software engineers in the room If you have a measure performance regression at release time suddenly it's get bisect fun, right? So you're typically really get bisect and you're running the same test case and it's typically very very painful Particularly the rate of velocity of communities like DPDK and FIDO VPP the rate of velocity is huge Means that we have a huge input of patches Which means you really need is a very strong safety net to catch performance regressions So how do you maintain best in class performance and how do you address the danger of creeping normality? And I think the more common name of creeping normality is boiled frog syndrome where you're just things slowly degrade over time and you Don't notice Well, I have a pretty picture for you. So this is our performance trending Timeline so we publish this and all the developers have access to this and you can see this is a whole bunch of L2 test cases here You won't be able to read this but I can show L2 bridge domains with one Mac With one Mac address up to 100,000 Mac addresses I think we actually might do an even a million Mac addresses, you know It doesn't matter but you get the idea that we have a whole series of L2 bridge domain test cases here from a single Scaling the hallway down to a hundred thousand Mac Mac addresses in a single bridge and we can see how that test case Performs over time and not only that we're actually able to run it against what we know Where you know our good numbers and bad numbers so we can say, okay Well, what's in the green and what's in the red and we can make judgments and you can see here the red Circles are regressed where regressions have happened and green circles are where performance improvements have happened And this is a dashboard that the entire community has on a regular basis This is how you generate a warm fuzzy feeling that you're not losing performance over time and Because I said earlier that we have a very very large number of test cases We have a very nice dashboard in which people can just log on and see whether performance is increasing What's the short term percentage change? What's the long term percentage change? What's the trending number of millions packets per second and is there how many regressions you're getting a Outliers and that kind of thing So what are the challenges here and I think the two questions before outlined the challenge is very well We have a lot of tests And we have a limited number of physical platforms And so you very need to be very judicious about when you run so the entire test We definitely gets run every release and we typically run the entire test suite on a nightly basis But that consumes a consumes it consumes a lot of systems When you submit a patch we tend to be more judicious about how many test cases you we run against any individual patch Because that you know consume that can really snarl up number of tests resort sources You know, this is not an unlimited lab. We have we're not an open source project with unlimited funds far from it So we need to be very judicious about what we run and when Yeah, and one of the challenges is how you get how you can ensure you get coverage So how you know that every patch is getting the kind of coverage test case coverage it needs. Okay, I'm going to keep going Deeper diving so moving beyond the symptom so I'm gonna move past this and maybe on to Yeah, probably onto this slide So, you know, I've kind of labored the point at this stage and you know You'd be forgiven for throwing rotten fruit at me at this stage Well, I live with a point at about, you know, how many benchmarks we have But at a certain point you need to move beyond just benchmarking the problem, you know it At a certain point you need more data to try and to try and root cause the problem It's it's one thing telling you know with number of software engineers in this room You know that it's one thing telling me there's a performance Regression or there's a bug somewhere but give me a clue give me some sort of an idea as what the actual underlying problem is and You know, it's well known today that Linux ships with a whole bunch of tools that you can use for next level analysis of where Performance regressions are coming from, you know tools like Linux perf We also have things like PMU tools for okay for pulling PMU stats from the microprocessor That tells you where cycles are being born and we actually also have very good introspection actual instrumentation in phyto VPP itself to tell you where cycles are being for Borned so what we're starting to do now is we're starting to use these tools and again They're all free and open source tools to start to generate the next level of information So when you go in and you see a performance regression now You can actually drill down and get to the next level of detail and look at the graph pipeline in VPP One of the I think Giles there earlier talked about the VPP graph hierarchy And you can see in the next level where cycles are being burned and that interface and that data again is being generated and it's there So when we get a performance regression, we don't just leave you hanging and say hey guys There's a performance regression. Sorry, we give you the next level of data They actually does a performance regression in this graph node as opposed to a performance a system-wide performance regression Okay, and then what lastly we'll talk about service density and this is you know when like we talk about real-world use cases So, you know, there's a lot been a lot of discussion today about The cloud and cloud night cloud native deployments, but there isn't really a whole lot of data around how Data planes react to cloud native deployments or certainly not enough data around how cloud How did and deployments react to cloud native environments? So we've been doing some work to try and understand that better and we've been Comparing service densities for virtual machines as compared to containers There's a lot of industry discussion at the moment a lot of ops discussion in the community around Hey containers are more efficient than virtual machines hands who proves her that statement containers are more efficient Virtual machines. Okay, lots of hands But then that's well, there is that as well But there is all that's a lot of talk around that but that in my experience There isn't I haven't seen a whole lot of data to substantiate that So what we've been doing in Fido again is trying to put data behind that to understand What that looks like and then where the inflection point is So we're we have what we're benchmarking today is we're benchmarking a VNF service chain where you have a whole bunch Sorry a whole bunch of virtual machines in a chain all connected by a v-switch And then we also have the same setup for as a container service chain again a whole bunch of containers Connected by a v-switch and then we also have a container pipeline where the container where you you inject Packet into the first container and then they just pass it to each other also to understand what a point-to-point performance looks like So and in that way what we do again as we test with one Sorry one virtual machine on the switch we test with two virtual machines four virtual machines eight virtual machines We test with one container two containers four containers a containers and then we you know We typically test again You get the idea of labor the point across all the packet sizes and we produce this kind of data that you can see here that again illustrates very well where the inflection point is for you know where When you start to get a switch over cost in performance from containers versus virtual machines, I've got one minute left to go So three problem statements that we're working on ensuring that you don't lose your performance As your net as your data plane evolves also understanding where performance regressions are coming from at the next level of information And then also how cloud how data planes perform in a cloud native environment So that's the Fido sees a project. It's an open and welcoming project and Same as every other open source project There's always there's always plenty to do so we'd love to love to see you turn up any questions Okay. Thank you very much