 Okay, ready to have their beer? Okay, no hard questions, right? I'm Azmer Mohamed. I run products for Plum Grid. And thanks for joining us. Hopefully there's more coming in. Anybody was here for the Walmart session before this? Good. So low bar for me to exceed then. So I'm joined today by Tehran. Tehran just joined here from Ixia. And what we're going to do is really help you understand sort of how we see performance and scale testing for clouds. We see a lot of customers that are moving into production and asking really hard questions about scale and performance. And so we want to at least share with you what we've done at Plum Grid and the methodology and so on and so forth. So this is my contact info. You can find me on Twitter, same thing for Tehran. Tehran looks better in person, by the way. All right, so we're going to break this up into three sections. We're going to cover a quick overview of the Plum Grid O&S architecture. We're an overlay SDN vendor. So it helps to put context around how we design our products and how it impacts scale and performance. And then we'll dive in into scale. And then we'll dive into performance. And then I'm going to hand over to our technology partner Ixia to talk about the Ixia product that can be used to go validate this pure software, really compatible with testing of clouds. And then we want to make it interactive. If there are any questions, please put up your hand and let's get those answered. All right, so cloud networks, sort of what we do. SDN is probably the more common term that's used for our products of our type. And the early SDN architecture was very much a stack where the control plane or the controller was in the path of the packet. And so that was an inhibitor to scale and performance. It was great because the software defined. It was a new way of doing things. And it worked well, but it definitely didn't meet the needs that we thought was needed. So when Plum Grid came around, we wanted to make sure that we had an architecture that was distributed and scalable and performant. And the big difference, as you can see here... Can I do this? Oh, no. Sorry. Is that we took the control plane out of the data path. So all the forwarding, which is done in that purple box over there, is done without the need for the control plane to intercept. And everything is made through an API call. And so this is sort of what we built. Plum Grid's been around since 2011. And so this is what we built and we're applying to our product today. And how it looks like physically is you would have what we call the Plum Grid Directors. And that's our control plane. And that's really where we would talk to a neutron plugin. So when you think about neutron plugin, there's a Plum Grid neutron plugin, it is talking to the Plum Grid Director. Plum Grid Director, then, is the control plane for all the network functions. So you see there you've got DHCP, a router, DNS, NAT, a switch, and a bunch of other things. And it can either be a Plum Grid or a third party. And those are the virtual network functions. And the network functions get rendered inside what we call these virtual domains. And if you will, it's your network container. So if one of you, if all of you were on the cloud, each of you would have your own virtual domain. So within there you would render your virtual domain that has your virtual topology. That virtual domain gets rendered on the compute node when a new VM starts up. So you see there's VMs of three different colors, blue, orange, and black and green. And so when there's a new VM that comes up on, for the first time on a host, is only the time that the control plane will push down that virtual domain into that blue, into the Iovizer, which is our forwarding plane. And so this, at a high level, gives you a view of how we want to provide distributed forwarding, because again, we're completely taking the control plane out of the way. So when you apply this to OpenStack, there's a significant amount of benefit. So who's familiar with neutron network nodes? All right. So if you're not using Plum Grid, if this were a picture from a standard OBS setup, you will actually see agents, and you'll actually see neutron network nodes. And what that means is that your forwarding is not being done locally. So at some point during the packet forwarding, you'll have to punt your traffic over to the directors or the controllers, and then it needs to make a forwarding decision. It sends it back before it gets forwarded. We never do that. So all of our forwarding is done inside the data plane, and then the director only comes in play when you need to define your topology. Part of our demos that we do for customers is actually to kill the directors to show that you can actually run your packets and forwarding without the directors being there, you know, standing apart. So the control plane is fully redundant because we have them on the directors. We generally put the directors inside the controllers every open-site cloud. They're all in Linux containers, so they can run anywhere, but generally best practice is to put them on the controllers. Because forwarding is done on each compute node, every single time you add a compute node, the capacity for your cloud increases, the forwarding capacity for cloud increases, and then since we don't have to punt traffic to the directors, the VM-to-VM packet flow is very optimized. Any questions on your architecture? Cool. All right, so let's talk a little bit about virtual domains. So how we came out with virtual domains, we actually pioneered this technology when we started as a company, and it's gotten so popular that we've actually seen it used by other vendors to explain this concept of your private network container. We actually got inspiration from a VM. So those of you familiar with the VM, you're abstracted out from physical hardware, and what you have in there generally is the OS and an app, and you have it in there. So we wanted to apply that same construct to networking because we saw a lot of benefits there. You have the software-only container, you get a lot of benefits around portability, you're a complete decoupler from the hardware, so you can run a pure overlay network, and it's a logical representation. So this was the thing we still needed to provide a view that made sense to somebody who's a network administrator, router, bridge, firewall, low balancer, those things are still there, and you still connect them in some logical manner, but we render it in software, you're not limited by what you can do in physical. And then the benefit is you can create copy and clone, which is very much what you do in most cloud operations anyway. And so this is for us really a cornerstone of our technology and our value add. And within the virtual domain, we have these virtual network functions. And so these are what you would be familiar with in the physical world that we render in software. So all of these things here, when you put them in a virtual domain, you can connect them, you can manipulate them and whatnot. And then when you're ready to use them, is when you say, you know, render it, and then as you bring up a VM, these functions get rendered inside that data plane inside each compute node, right? Pure software. And that is where I'm showing here all the Plumgrid virtual network functions. You can also add to this any third-party virtual network functions, so you can service change things in. All right, so Iovizer, which is our programmable data plane, is in kernel. Everybody know the difference between user space versus kernel space, right? Kernel space run faster. Right, just remember that. So we spent a lot of time and effort upstreaming Iovizer into the Linux kernel, and so that's paying dividends now. And that gives us, you know, it's really ultimate software forwarding performance. Security at scale. So virtual domains, by default, are secure. Each one of us gets our own virtual domain, and that is your private network. Nobody else can tap into it. Natally modulated tenant, right? So this fits in well with OpenStack and any shared infrastructure type of environment. And then because we do everything locally inside the Iovizer, our throughput grows with each new compute node that you add. And so you'll see here in terms of the scale that we see is truly additive, highly efficient, right? There's no punting to a neutron network node. Everything is done inside the Iovizer. And then the control plane is completely decoupled from the data plane. Gives us a lot of benefits there. All right, so let's talk about what we do around scale and performance. So, you know, as a company, we've had to go through different phases where we're going through creation of the technology to begin with, and then integrating into various pieces, both in the underlay and the physical environment, VXLAN offload, for example, and also into the OpenStack distributions, right? So we're beyond that. As I mentioned earlier, we started in 2011, and now we actually have customers going into production. And so there's been a renewed emphasis at Plumgrid to really dig down in terms of scale and performance because our customers are asking us for it. So we went ahead and the lead up to Vancouver, really dedicated the team here. There's actually more than the folks that are in this page here. And really to try to do two things. One is to get the updated numbers, right? We felt that it was time for us to sort of share, be more transparent around what we can do. And then also wrap that into our standard release methodology, right? We've got to be able to maintain or exceed a certain baseline with every software release that would come out, right? So we know OpenStack revs every six months. We want to make sure that as OpenStack revs or as we rev, we're maintaining the same number or higher. And that can only happen when you're putting in investment and you have a methodology that is repeatable. So, and we've done this via capex spend on the infrastructure side. And also we do a lot of our testing, scale testing on the public cloud because it's much easier for you to spin up 1,000 nodes in the public cloud and spin up 1,000 bare-bendle servers, right? And the numbers are very clear there. So definitely focus for us. So when we look at scale, there's really three main components that sort of, you know, that's important to us. One is the number of compute nodes, right? That is a direct proxy of how big your cloud is, right? The number of instances that you can run and whatnot. The second is going to be tenants and instances, right? There's going to be a correlation to the number of businesses. So that's what we focused on in terms of how to determine scale are these three axes. And, but we also wanted to make sure that this was something that was repeatable, right? So remember when I talked about directors? So that's a cluster and a cluster starts with three and you can grow beyond that. We wanted to see how far we can push ourselves using our minimum HA director size, which is three. So everything that you're going to see in terms of scale here is sort of based around a setup of three directors and the director is just a, you know, it's a container that runs on an x86 server. We have some recommendations in terms of RAM and whatnot, but really it's fairly lightweight. So we wanted to see how they're, and then as customers exceed the numbers, then we'll decide whether we need to beef up the directors or recommend a reference architecture that increases the number of directors, right? But we wanted to start with three. All right, so what did the number say? The number said that we could, we tested up to a thousand compute nodes and what that means is that a lot of customers today are just beginning their journey on OpenStack and what we've seen with clouds is that once you get started, that explosion starts, you know, it goes fairly rapidly and so we wanted to make sure that if you start today with three directors, you can go a long, long, long way before you have to redesign your cloud, if at all. And so a thousand compute nodes is sort of what we saw and really is one of these things where, you know, back to the Neutron Network nodes, there is no need for you to think about these things, right? You're safe to go with the reference architecture of three directors and two gateways and up to a thousand compute nodes. So that's generally what we're recommending to customers and this again gets us, you know, very, very far down the road. So no need for agents, HA and network nodes to worry about as you scale up, okay? So the second thing was around, you know, tenants and instances and so again, we wanted to make sure that we could get to a number that was, you know, that gets us further out. We, you know, we frankly ran out of time. We could have gone further on this but when we reached 10,000 tenants and 20K instances, we felt that it was time for us to cap it now, but we'll continue to exceed this. Hopefully by the time we get to Tokyo, we'll get there. The, just to double click on this. So we did the testing here with, you know, fairly straightforward topologies and as you see, there's a mixture. Effectively, we had two instances per tenant. That's how we got to this number. We could have, you know, we're going to go in and have more sophisticated mixes because we know that not everybody's going to have the same topology all the time, right? But we're fairly confident that even as you slice and dice this into different mix of sizes per, virtual domains per tenants, that you'll be able to still get to the same number at the end. So that was good, but a lot of times when we run into networking, people want to know about connectivity outside of the zone. So before that, you know, if you go back, before this, it was just a number of tenants, number of businesses and whatnot. What about if I had to go outside? What if I got a floating IP? So when you get a floating IP in OpenStack, you're effectively doing a NAT. And that's both in Word and out. So source and destination. If you look at a typical OpenStack design today that uses neutron network nodes, either NAT is done in a neutron network node. So everything needs to go through the network node before it goes out, or you're going to split. Even with DVR, you split source and destination NAT. With PlumGrid, all of those are still done inside the Iovizer. And we actually did this at a customer site. So we actually went on site. We installed this on their infrastructure. And that screenshot that you see here is actually a tool that we have. So we've got this set of tools called PlumGrid Toolbox. And one of the things that we have in there is called a NAT Inspector. So you can run the tool to sort of see how many connections you're getting inside the system. So that's what we did. We actually went on site to them. I think this was maybe a week ago that we did this. And ran this on their infrastructure. And this was the goal that they set for us. They wanted to see 100,000 NAT connections running using our software. And we got this done. We haven't done a competitive analysis yet, but we feel very confident that this is much higher than any SDN vendor out there. All right, I'm going to pause there because that gets us to a summary of scale. Any questions that people have here around compute nodes, virtual domains, tenants, and instances, plus the NAT flows? You guys are easy. So what we did was we spun up the instances. We created the flows. And then once we got to 100,000 flows, we ran it for 90 minutes. And sustained. Yeah, yeah. So let's go back to the good question, by the way. So you use the word east, west, north, south, and that's a nomenclature we use also. So east, west is when you're talking VM to VM within the zone. And then north, south is when you're going in and out of the zone. So a NAT test is a proxy for north-south traffic. And then east, west is that throughput number. So we'll cover throughput. But throughput generally is a good proxy for east-west because you're not really doing any sort of IP address translation because you're within the zone, right? You're encapsulating in VXLAN and you're sending it out. So generally, we're not worried about east-west because that can run fairly fast. You can do offload. What you're more worried about is north-south. But as a percentage of traffic, depending on how you've got your apps and environment set up, you could actually have applications that never have to talk to the internet, right? It's a database server talking to a middleware. It is going east-west. Or you could have a web server that's always talking north-south, right? So for us, it's very important to sort of give you a view into both of those. So the network hops. So for the, let's see here. For all of these, the measurement was purely around can you bring up these virtual domains inside your zone, right? So it was like, bring up your IF, your interface is up and running, and you can ping packets, right? So it was not a test of performance. So that's a test of scale. I think for NAT, it was more sort of a hybrid of your testing scale and your also testing flows, right? Going in and out. So that's a mix. We do have performance coming through, but I want to make sure that we cover scale here before we transition to the next one. Good questions. Yeah, sir. I believe it was TCP. Yeah, TCP. All right. Cool. All right. So scale, and we'll do more here, but this generally when we talk to customers, people want to find out, these are the main things people want to know about how they scale. They're open to that cloud. One thing to note, what I covered here are numbers for networking. As you know, in a cloud, you have compute and you have storage. And all three of those could be a limiter for the size of your cloud, but we want to make sure that we're not the lowest common denominator for your cloud. Okay. So let's go into performance testing. So performance testing, you sort of have to take off the scale hat and put on a different hat. It's a very different view of the world. So, and Taran will cover more because he spends a lot of his working life focused on networking equipment and how it's done there, but let me just apply sort of cloud constructs here. So one is any cloud performance that you test will always be CPU bound. Plain and simple, right? Virtual machines are going to be CPU bound. And if you test on lower grade hardware, you will get lower performance, guaranteed. You test on better hardware, you'll get better performance, guaranteed. So that's the one thing. If you're going to do this and you need to measure your cloud, there's a direct correlation between the numbers that you get and the type of hardware that you use. So it is an investment. We made the investment because this is our business, but if you're going to do this at home, quote-unquote, then just make sure that your hardware is good. So what should you be looking for? So definitely, you know, so modern CPU. I think we used an E5 Intel, right? So you're using an i, i5, i7. You're using a desktop class CPU, right? You want to use something at server spec. CPU, right? The frequency, the higher gigahertz count, the better you're going to get. You can actually go in and lock. So if you're actually going to do this testing, you're going to get variances if it's not locked. And then the VM, right? So if you're using, if you're overloading the number of VMs per core, then you will get, obviously, lower performance. And when you'll see here, it's when you measure performance in the cloud, you're measuring between pairs as you go over the link. But this is probably the most important thing. The guy that helps us do this at Plumgrid, this is like, you know, this is what he does. The first four questions he asks us is about hardware. RAM and the RAM spec counts, but not as directly as CPU. So what we've seen is that, you know, so we actually load up our distributed hash table is actually loaded up inside memory. So just to play through our architecture, when you first bring up our cloud, there's a config file that's on disk, and then we load that config file into memory. And that config file pretty much is what those virtual domains we talk about. All of that is in RAM, and that's how it gets downloaded. So the more RAM you have, the bigger, you know, the bigger table you have to build this. And so that helps, and then the higher RAM spec allows us to go do this. And so this helps as you bring up the VMs how quickly we can render the apologies and whatnot. The next one, frankly, I don't know what that means, other than my engineer told me to go put it in. So if someone knows what VhostNet enable means, then let me know, but otherwise just note that as you measure. This is right. Okay, good. This is right. And then, I think in this day and age, if you want high performance and you won't expect that in software forwarding, I think that's naive. It's like using, it's like building a data center using old chips that don't have a virtualization enable, right? You just gotta leverage anything that you have in your hardware to offload this. And so with the way that you build out scale out data centers, right? If you've got a CPU that is doing processing for compute and also doing processing for networking, that means that you're going to get much lower density on the compute side because you're spending a lot of time processing network traffic. And so using VXLAN offload nakes, I love them because for that investment it pays dividends because you get more bank for the buck in terms of does it compute power. But if you end up, if you can't use a VXLAN offload then you definitely want to make sure that you turn on UDP RSS and do the measurements. All right. So let's see how we did this. We did this simple. I didn't build a thousand node compute cluster to go test performance. I just built two nodes and then I tested the performance to team them. So this is effectively the ability for you to sort of offload things onto the neck, send the packet over the wire and sort of see how far you can get done. And then we just distributed the VMs across the two nodes and saw how fast we can saturate the link. And this is what it looked like. So start with, okay, so we used 9,000 byte messages. That was the two servers that we used and we used the Melanox network card to offload. And then as you can see, the numbers grow as you get closer and closer to the number of pairs. And it's because back to that CPU you can saturate the link and the core at low numbers. So that's why with a single pair you can't get to 36, right? It's not possible today. So as you go across to this, where is additive? Is there a question, sir? Two different compute nodes. So the VMs were, so the pair would be one on one and one on two. And then we just, yep, exactly. This is how I measured it. So we did this in a single virtual domain. So this is a single tenant and we did this. We didn't have time to go do the variation where we had one virtual domain on one host and on the virtual domain on the other and do this. So this was the test that we did right here. Yep, sir. Yeah, so I brought that up because I actually had originally said packet size and my engineer said, no, it's messages. So we did, so it is effectively an MTU of 9,000, right? That's effective, but I was told the terminology is messages, not MTU. There was a question on this side? Yeah. Yeah, it was identical across and I'm trying to remember here for this server spec, it was 10 cores. So we had, we still had extra cores available but we were saturating the link anyway. One core for PBM. Yeah, that's best practice. I think if you can do more than one core per VM then you're good but we figured the baseline is that you have a single core per VM. So that was the testing that was done here. All right. So then we said, okay, so we did that with 9,000. How does it look like for smaller packet sizes? So this is what it looks like for 9,000, 1,500 and 512. You sort of see this natural, you saturate the link earlier on. It's fairly consistent. You get much better throughput obviously for the larger message sizes but it's not that far off as you see this for the lower packet sizes. It's fairly efficient. It ramps up and then we fairly saturate around the five to six pairs on this hardware and then we get there. So how should you read this? There's a lot of data here. So obviously the difference between a 9K byte and 1,500 byte after you saturate the link is minimal. It's greater with the smaller number of pairs but after that it's there. So it's fairly efficient in terms of what you can do for packet processing. Second thing is that there is still performance gains that we can reap. So I mentioned earlier that we had, we use Iovizer as our data plane and we had upstream that to the Linux kernel. So we're going to constantly try to get more optimization done to close the gap between what you see on the bar and maybe what that yellow line is. I don't know how close we'll get but that's something that as a company we pride ourselves and we want to go get that done. And then if we were to use, so this hardware here, it's not the best hardware you can find on the market. There's better ones now and so we'll get better ones and the expectation is that we'll get bigger throughput because we'll need more pairs to saturate the link. The hardware can do more. Yes, sir. This was, yes, this is east-west. So we're going from VM to VM. So you bring up a good question. Hey, what's the performance numbers for north-south? And now I have to go into the product pitch. So when you go north-south you're going through what we call our gateway and what happens in the gateway is that you would actually encapsulate and de-encapsulate VXLAN. The gateway can be a software gateway. So it's our software running on an AX86 server. Or it could be a hardware gateway. It could be a switch from Arista and QMOS. So the switches run really, really, really, really fast. That NAT that I was showing earlier, that is not done on the gateway. That NAT translation is done in Iovizer. So the performance, if you were to just measure, if I were to put a VM, let's say if I had a gateway between these two servers, I don't know what the numbers are, but it's going to be an order of magnitude or more of the NAT flow number that I showed you. So this is the theoretical max. So this tells me that today we've got still improvements that we can make inside our forwarding architecture. So we're going to do that. We're going to optimize it. There's a lot of tricks that we can use in Linux kernel that's happening. So there's more there. But if you see here, even at the larger, where about 90% of the theoretical max today? How close to that max? Obviously, we want to get to identical. We want to get to the theoretical max. But that's what we need to go work on and try to capture. But even today, it's a highly efficient forwarding engine. Good question. So I'm going to stop there. Definitely want to spend more time later on if there's specific questions you have around there. I want to pass this on to Tehran to talk about AXIA and how people use open-source tools and whatnot. That's great. But there are also tools that are very compatible to use for performance testing in the cloud. Okay, I'll try to be brief. We've got about 10 minutes. I want to leave some time for Q&A. First, a little bit about AXIA. AXIA started its journey about 18 years ago as a network-tested measurement company selling solutions to equipment manufacturers, their customers, carrier service providers, and enterprises. And fast forward to now, we're a customer advisor and technology services provider. We really focus on improving application delivery in any of the infrastructure that you have and security resilience of that entire infrastructure. So really focused on improving that experience for a customer. But today, and some of you may know AXIA as a hardware test company, and that used to be the case a long time ago, and a number of our products that were purely on hardware for the purposes of testing switches and routers at that time have been now virtualized for the modern cloud-centric infrastructure. So practically everything at AXIA that's available on hardware is actually available in virtual. But the one I want to talk briefly about a little bit in a dig deeper is IXCherry, and I'll call it Chariot once in a while. It's an all-software solution. The way it works is it's actually intention is for troubleshooting and performance testing of virtual infrastructure. So there is a web console that you deploy in your environment. It could be on the management side. It could be in-band with where your production network is going to be. So that's the console that actually is responsible for creating the test apologies. It's responsible for getting all the data from all the distributed locations and actually providing real-time stats that you can look at. The workhorse of the product is what we call performance endpoints, and I put down 980 kilobytes, and that applies to practically all the Linux distributions 32-bit or 64 that you would actually deploy on any flavor of Linux that you have. It could be a VNF that you have. It could be an instance of Linux that you prefer to use, and you would actually distribute them and install them wherever. You have servers, domains, instances that you want to actually get performance metrics from. We talked a little bit briefly about floating IP. This product is not friendly, meaning we actually initiate all the traffic from behind a firewall, so we can actually traverse one-half multiple-net boundaries very easily, and I'll talk a little bit briefly about that. And all the application traffic that we actually model in the product is state-aware, so all the data applications are TCP. They're state-full. They actually are what customers would be using. And then also streaming and UDP traffic. So a little bit more on the distributed endpoints. So as I mentioned, install on VMs or any existing VNF function. So you can imagine you can have a plain vanilla Linux system. You may have a switch as a VNF. You can actually, as long as it's running Linux, you can actually put the endpoint directly in that environment. So you can actually validate that infrastructure. It uses the actual network stack and the transport for sending and receiving all the traffic. So in effect, it's actually, you're able to validate the OS, all the parameters that it may have by default, the way the vendor shipped it, the container essentially, and the transport of it. And then it uses IP connectivity across. So what that means is that it doesn't matter where the endpoints are. You can have endpoints behind as a fixed IP. You can actually have endpoints that are actually outside the floating IP boundary. And we use dynamic pinhole technologies to actually get to those endpoints from the console and actually construct tests that actually can go east-west across domains, even if it's not east-west or north-south, even if that's not it. So it's actually meant to not have you configure any rules to be able to reach the endpoints regardless of where they are. So I kind of highlighted that here. We traverse the stacks, TCP, UDP, RTP, all of that given operating system. Any stack settings on the network that are there by default we don't touch. So you're actually testing the real thing. And it'll run on any environment. So in this case we're talking about virtual, but it'll actually run on any LAN or WAN as well. After you set up all the distributed endpoints, typically you would actually create topologies in the product and it's meant to be the way you have it. So if you wanted to have a three-tier application or you had servers in one instance in one domain talk to the other, that's what we call logical topologies. You'd create that in the product by way of defining which endpoints are on which side and then you define the type of traffic that you want to have. It could be small packets, large packets, the ones that we were just looking at. You can define how many simultaneous TCP flows are actually being generated from one to many, many to one, just based on the typological topology that you might have. So as an example, you saw 100,000 flows. That would mean maybe you want to have one endpoint that actually is responsible for all 100,000 connections to be initiated and you want to distribute them on 10,000 different destinations. You can actually model that as an example of a topology. It just is a function of how your actual antenna network is. And then once you have that set up, once you run a test it's all in real time so you actually get flow metrics for every one of them so you can actually see degradation in performance from every instance to every other instance that you have set up. A quick touch on briefly earlier, as Mir talked about, a lot of the network performance. Obviously, we test the network regardless of its virtual, if it's switching out to a top of rack infrastructure and back into another one or any combination. But we also and customers use this a lot where they actually want to max out and figure out what the limits are of X number of cores in a given physical infrastructure. So we have pretty large systems that run and I wanted to indicate a lot of times customers will run a fully loaded system. This one happens to have 24 cores, fully dedicated to multiple 40 gig interfaces on the actual hardware. And we can actually push out more than 100 gigabits of traffic through all those cores. And the idea is to see where, at what point, are all the cores busy? And at what point have you reached the boundary of the interface itself? And what does that look like from the CPU user standpoint? So we actually can let you see both at the same time by doing so. So looking ahead, as we work closely with PlumGrid, we're going to develop some test methodologies to highlight performance in various scenarios that you might have in your network, North, South, East, West. We'll create some traffic modeling that is actually based on the overlay technology that you might have. And we'll be actually creating some of the benchmark flows so that actually they are part of a solution that you get. And we're hoping to actually create these test scripts in our product so that actually they can be actually pushed out. And anyone looking to model this type of traffic has a repeatable way of doing that. I know I kind of ran through everything very quickly, just in the interest of time. I think we're towards the end here. And I think what we want to go do is really bring some repeatability around testing methodology here. So obviously for us, just plug for O&S, we really feel that we've got a highly scalable, highly performant platform. We're going to continue to update these numbers and publish them to make it available to you. But really the cornerstone of all of this is our architecture that's fully distributed. I think the ability for us to do forwarding in kernel allows us a leg up on a lot of other solutions in the space that use user space forwarding, that's not as performant. And then what we want to go do is, as we work with Ixia here, is really to provide this test for you to validate very self. One thing we want to make sure is that we're doing an Apple's Apples comparison. You can only do that when you have a well understood methodology and a tool set that is available to folks to go do. So hopefully this has been beneficial for folks. If there's any questions, we'll answer them here in the forum or if you've got one-on-one questions, we'll hang out after a presentation. Any questions here before we finish up? All right, well thank you very much. Thank you. Appreciate it.