 All righty. So my name is Robert Starmer. I am the founder and chief consultant for Communist Technologies. We're a systems consultancy. We help people build open stack environments. I'm joined by Anne McCormick. Anne is a senior software engineer at Cisco Systems, focused on network technologies and video. And the video aspect is one of the key reasons why we thought this was an important topic, video and networks. And then Alka, South Nor, sorry, did I just put your name? No, I did not. I got it. Alka is a DevOps engineer at Cisco, focused on content delivery networks, another area where quality of service is actually an important aspect in how open stack can actually deliver services. So I'm going to talk about this. We're going to talk a little bit about the traditional concept of quality of service, some of the things that are happening in open stack in terms of quality of service today and tomorrow. And then we're going to focus a little bit more on the network services and some of the other technologies that enable quality of service in open stack environments. Also looking at compute and storage bottlenecks, which, while not traditional quality of service techniques and topics, are still important. And then lastly, we'll look at a use case where quality of service, especially the next sorts of things that we want to start implementing in open stack quality of service capabilities, start to really become important, again, looking at network technologies. Now the reason that I say network a lot in this is that quality of service, this is the Wikipedia definition, is the performance of a network. People talk about this from the perspective of a network. And we actually see quality of service mostly talked about either in network technology, so network pipes, the quality of service applied to a network, or in storage, because storage actually has very similar characteristics. Most storage that we're talking about is, in a similar sense, network attached. Even if it's local storage, there's an internal network that's attaching the storage to the rest of the system. And so we think about this from that networking-like space, like advance the right direction, it helps. So if we think about this as a network-centric technology, there's a network sense of aspect to this, we're really looking at availability. So how does the network affect availability of our applications of path and technology forwarding, and also reliability? So if we're looking at being able to get a packet from point A to point B reliably, the network comes into effect there as well. And this provides a model for understanding how the network and the forwarding path of data and of services enables our forwarding plane, really. If we look at that, when we talk about quality of service, there are really four characteristics that we're talking about. We're talking about bandwidth. And again, networking-centric view, so bandwidth is our ability to move a bit from one location to another, how fast can we do that? The faster we can do that, the more data we can move between different locations. That's just our raw rate of forwarding. Now, that doesn't necessarily mean that we have good throughput or good good put, but it at least means we have an idea of how we're moving data from point A to point B, quality of service parameter. Latency is the next part of this. Latency is usually the inverse of bandwidth. The higher the bandwidth, the lower the latency. It's the general concept. But being able to measure and understand that helps us understand what kind of service we're able to apply, therefore the quality of service. So latency is an important aspect to this. Jitter is another aspect, especially when I'm starting to look at things that actually where the, not just the latency, how long does it take to get from A to B, but how consistent is that latency? If I have very low latency for one period of time and very high latency for another period of time, that may actually impact how my application is able to operate. So latency is an important aspect as is jitter. And lastly is reliability. It's great if I can get you a billion packets a second for a tenth of a second, but then if I drop every packet after that, I would also say that my quality of service is poor, right? So I wanna be able to look at all of these characteristics when I'm talking about quality and how they apply to the overall service and service characteristics. Now, even though quality of service as a general means of talking about service is a networking technology, I also think we can talk about quality of service relative to all of the open stack services. Often when we're talking about compute, we're talking about reservation, which isn't really a quality of service component, but it's a way of thinking about availability, about latency for access to service. In other words, can I turn on more capabilities? So there's a latency aspect there that we can think about. But we think of reservation. The rest of the services, whether it's network or storage or storage networking, especially since network and storage networks are often now becoming one specific or one single network rather than multiple, we're really talking more about rate limits and potentially queuing as a way of manipulating and managing those resources. In the compute space, really resource share is the fine grained way of trying to deal with quality of service, but in all cases, if I'm thinking about quality of service, I'm thinking about how I'm getting something from point A to point B. In the compute world, this map doesn't quite work as easily as the others do, because I'm not really thinking about a serial stream of data, but I still think that people talk about quality of service, service availability, service share, even in the compute world. So I think these things all fit together. Now, there are ways of also in the open stack space talking about our quality of service, how much service am I getting? The way I do that in the resource space, again, following this reservation model is to use the flavor concept. Flavors are really the only way that I, as a user, can ask for a specific quality to the service that I'm asking for. I can ask for a specific scale of system, that's one way of doing it, but I can also apply other parameters to that request. For example, I can do affinity. Affinity and anti-affinity in the compute space is one of those different ways of thinking about this. In the network space, I'm really talking about quality of service parameters. Now, there's effectively the old way of doing it, which actually mapped to a nova flavor. I'll talk a little bit more about that in just a second. And there's the new way of doing it, which is actually to pass in parameters at the network level to say I'd like to use some of these quality of service concepts, like bandwidth or latency, and apply those to the network path that I'm asking the system to establish. So that's using the quality of service parameters within the open stack neutron environment. In storage, we're talking about two different things. So there's storage networking, where effectively quality of service also applies. Although there is no storage networking interface in open stack, we're using the quality of service tools within the network space, usually the underlying infrastructure network space, to apply the same kind of quality management that we would have had in the network services space that we can manipulate directly through the open stack interfaces. The other aspect to that though is the actual storage infrastructure itself, looking at IO related limitations. So I can actually ask for different storage classes, which is really the way that I would do this in the storage space, to manipulate the IO's per seconds that are available to me. But again, this looks more like a resource share than it does a queuing manipulated management process. Does that make sense? Anybody have any questions? You guys are awake, right? I mean, I know you just had lunch. Yeah. So in the physical network, if we sort of step back and look at where the quality of service world came from, really initially this came out of the router space. So routers had this interesting job to do, which was they often had to deal with mismatched bandwidth. So we had our local networks that were running at potentially tens of megabits per second or hundreds of megabits per second or now gigabits per second. And we had our wide area networks, the things that we were routing between that were running at hundreds of kilobits per second potentially. In order to deal with that mismatch, we would take a frame of reference, a packet in the network sense, and store it and look at how long it would take to start to push out onto the slower speed network. In order to do that, routers would queue packets. So this was one of the first operations that routers did beyond just trying to figure out where to send a packet. Well, with that queuing, we then came along the idea of building potentially multiple queues, potentially adding priority to one of those queues, especially if I wanted to think about voice traffic. Maybe voice traffic is very sensitive to latency, sensitive to jitter. So I wanna try to get my voice packets onto this potentially lower bandwidth wire, lower speed wire faster so that I can reduce the jitter, reduce the latency. So this physical world was a large part of enabling quality of service across a disparate and varied network. And that same technology is being applied today in the open stack space, but not in the routed world. I guess the other thing I wanted to talk about was the ability to handle those queues. So if I have three or four or five queues and I have one of them defined as a priority queue, I also still have to deal with what happens when I start to fill them up. At some point, I'm going to have too many packets, especially if everybody's trying to jam data down a single small pipe, right? And in order to deal with that, a number of different technologies came up. The first one was sort of first in, first out. Well, I have three queues, whoever showed up first, you get on the wire next. That's sort of an easy way to do it until the queues fill up and then you start discarding things off the end. Well, if the queue is full, there's nowhere to put the packet. Guess what happens? It gets dropped, right? That's not a good thing. We'd like to find better ways of dealing with that. That's where the weighted random early discard model came into effect. If I have multiple queues that all have sort of similar value, I don't have a priority associated to those. I can actually discard some packets randomly early and that more efficiently impacts all applications rather than just some applications, right? So again, trying to figure out how to deal with lots of different talkers on a very limited bandwidth basis is a problem that we all see, right? And queuing is one method for doing this. Now, I said earlier that in the physical compute space, we don't really have the same kind of queuing mechanism because we're sort of looking more at resource reservation than we are forwarding of sort of a stream of data. But even still, when you actually look at what the, how did I told you I would blanket somehow? Didn't I say? So even though this whole model of quality of service came from this concept of data being streamed through a network, if we think about this relative to compute entities, compute has a similar problem. There are still queues even in the CPU, right? And at the end, you have one processing unit at least for some set of your compute use cases that is trying to implement something and you end up with the same sorts of things. When do I dump a cache within the CPU? All right, so these same sorts of problems exist whether we're talking about physical compute, physical or virtual compute, physical or virtual networking and storage. So with all these technologies, I'm going to get using this. We had the layer three space. In layer two networking, initially there really wasn't quite the same problem. We didn't necessarily queue up packets in networks because in layer two networks, the idea was, well, everything is of equal bandwidth. So a packet going from location A to location B should just be able to transit the network and not have a problem. Turns out that's really not true. And actually with modern networks where the layer two network is really sort of an overlay with the layer three network anyway, we end up with all of the same queuing technologies. But sometimes we have different ways of looking at what those packets look like and deciding when to place them into a priority or non-priority queue. But we have these same capabilities so we can actually start queuing traffic here as well. And, okay, now this time I didn't touch anything, so. So even still, we really need to understand how to manage and manipulate these packets. Now one of the problems is that if I start putting packets on a layer two network, I have very little information. And in the layer two network, I basically know the source and the destination on that network for a frame. Which means that I have a problem. I don't know easily how to differentiate that traffic. And as we'll see here in a moment, that's one of the nice things that is sort of coming along with the OpenStack space is that in the past, OpenStack really supported a very limited quality of service model. In the compute space, it was flavors, flavors gave you some compute level quality of service. In the storage space, there was at least storage differentiation so you could have fast storage or slow storage. But didn't really look at the quality of that service that you were getting from those resources. In the networking space, we had RXTX factor, which was actually tied to compute. It wasn't even tied to the network that we were using. So it was just a simple way to say well, I could provide potentially better forwarding for some of my compute resources, which you would choose based on the flavor that you chose, and poorer resources for other flavors. So I could have a cheap flavor and expensive flavor and I could sort of do that with this RXTX factor model of trying to share our network bandwidth. And it was really sort of a rate limiting approach to managing our quality of service. With the more recent versions, and especially even up to the Mitaka version, we had ML2 extensions that supported a more direct connections into the network, things like SRIOV. So now the virtual machines that we deployed have direct access to the network and potentially bypass some of the internal latency of the compute resources that we're working with. But more importantly, they had more direct rate limiting at the network description layer. Now, using something like SRIOV or some of the other interfaces, we're still actually manipulating that rate limiting against the interface of the compute resource. So we're still tying it to the compute resource. But again, we're just limiting it based on rate. How much data are you putting through the system? With Newton though, we now add what I think is one of the more important aspects of quality of service, which is the ability to tell the system what our packets are. So this is this concept of DSCP marking, which is, I've forgotten what DSCP, huh? Yeah, differentiated service code point. Of course, we all know that one, right? So DSCP marking is really important because it allows the device itself to start telling the network what it is, what traffic it's sending. And especially in a Layer 2 network, where all I really know is the source and destination, adding this additional mark is very important. So rate limiting seems like a good idea, right? To begin with, if I just limit rate, I can say, well, I have X amount of bandwidth and I can differentiate based on different systems that I'm deploying. Maybe it'd be nicer to be able to apply that rate limit to different tenants rather than to specific, to sort of generic flavors of things. And that was sort of what we had initially. The problem with that is that rate limiting is non-discriminatory in that it will do this to you regardless of what you're trying to accomplish. So if you have a really bursty application, something that only for a short period of time needs to communicate, get a chunk of data sent out or received, rate limiting might impact that much more highly than you might otherwise expect. And this is this concept of good put versus throughput. Throughput is how many packets per second can I get across the network, right? And I can limit that rate, that's fine. But on a network, we're not often just spewing packets. We're often spewing packets as a part of an actual session, a TCP session specifically. And TCP has this very interesting thing called early or slow start, restart process, right? And that whole process looks at trying to make sure that I'm equally sharing the network and being a good network neighbor, partially because of the routed world that this came from. And in doing so, if I constantly drop packets because I've hit some rate limit that won't let me send another packet on the wire, I actually restart that process of starting slowly and sort of creeping up to the overall available actual bandwidth that I have, which means that my transmissions are going to be impacted just because I'm limiting the rate of my system, right? Especially for bursty apps. There are actually tools within the physical network world to deal with this. You can say, well, I can rate limit, but I'll give you a burst capability. Well, there's no way to control that in the open stack space. So instead, we just say, hey, you have 100 kilobits per second. That's the limit. If you send one packet, even if you're on a 10 gigabit per second link, and that packet exceeds the one kilobit per second rate over a period of time, well, then that packet's going to get dropped, right? So a very inefficient method, but it is a method. And in some cases, this is an important component of how you do this. With DSCP marking, we can take this a step further. First off, we can actually start using the physical network, which has much better control over how it manages these packets. Things like including the ability to support bursting against rate limiting. But I still want to know what those packets are, where they're coming from, where they're going. And in this case, I have two options for managing this marking concept. What I want to do is I want to take the packet, which has a source and a destination, and add one other piece of information, which is the mark. What class of service am I expecting this traffic to get? All right, and that mark is really important. I can do this either at the application level. So if I trust my application, which I'm deploying the virtual machine, it's my application, I should be able to trust myself here. I can actually apply this mark at the application and have the application, or at least the operating system that my application is running in, apply a marked packet and pass those into the network. Now if I accept those into the network, then that means that that mark is available. The network can now look at that mark and decide, hey, what class of service does this belong into? Am I going to give this priority? Am I going to give this higher bandwidth? Am I going to do something else special with this packet? So from that perspective, DSCP marking is incredibly important for actually making quality of service a useful tool in these sorts of environments, right? The other thing I can do is I can say, well look, any traffic of a certain type coming into my network on a port is going to get the mark. So I can actually do the marking for the upstream system as well. And both of these models are supported. If I'm using something like SRIOV, which is effectively one of the virtualization bypass technologies, basically drops the packet directly onto the hardware, well even in that case, I still would like to be able to use this marking capability. And SRIOV, I can either mark at the physical network port, which is where I'd have to do it because I've kind of bypassed the entire virtual switching layer, or I can do it in the host as well. So all these things work with this SRIOV model and with OVS and Linux bridge-based ML2 interface based services, right? So this is the networking quality of service space. And this is the end of my section. And you're up. So that's the network. Again, compute, network and storage all have quality of service. But when we really think about it, it's the network that's the important part here. No? Hello, I'm Ann McCormick. I work for Cisco as mentioned. And now let's get beyond the network. So traditionally when we talk about quality of service, as Robert had mentioned, we think about networking type things. But just because you've gotten the traffic there faster doesn't mean that it's going to be processed faster or stored faster. So what are the things that could potentially happen? You could compute and storage bottlenecks and that's what I'm going to be talking about today. So first of all, compute bottlenecks and how to alleviate them. So on an OpenStack network, you've got a controller running a scheduler and you've got a bank of compute resources on which you can schedule a VM. And it can choose amongst those in order to schedule it. Once a compute host has been selected to run your VM on, you spin up your very important VM. And I think one of the most obvious things that can happen in terms of compute contention is that something else is running on the same server. In this case, I've called it a compute hog. When that happens, it can cause your very important VM to start context switching amongst CPUs on the node. Generally it's not a problem if they're within the same NUMA node. However, in the worst case, your very important VM can get context switched over to a different NUMA node, meaning that the cache needs to be repopulated and you're going to take a performance hit. Less obviously is that if your very important VM is running all by itself on a compute host, it can still context switch and it can still take the same hit by going to a different NUMA node and having to repopulate its cache. The reasons why it would do that is eventually it's going to depend maybe on network traffic, maybe on IO. And so let's talk about the cost of CPU sharing and context switching. I ran a quick open stack test using IPERF with multicast traffic. I come from a video background so in my case, I was thinking all multicast. This was on a network that was highly optimized for multicast. It's got all the bells and whistles, SRIOV, multiple RXQs with maxed out Q sizes, RSS, receive side scaling, ARFS, accelerated receive flow steering, and QUAS, in this case, preference was given to the multicast traffic. And my IPERF receiver on my test was running on a tenant VM receiving a steady stream of 800 megabits per second multicast. And then I started context switching, forcing it to context switch. And if it was switching within the same NUMA node, like I said, the performance did not take a huge hit, everything seemed fine. When I forced it to switch across NUMA nodes, my tenant experienced a 0.2% packet loss. And you can say 0.2%, it's less than one, it's not really so bad. But things like video are actually extremely sensitive to packet loss and to jitter. So this is kind of a big deal. So what are some of the things you can do to differentiate or prioritize your compute resources? Well, at the highest level, you can create host aggregates, basically groups of compute nodes, which you can reserve for your important applications that are running, and the less important stuff can run somewhere else. You can also get a little more fine grained. And I think Robert mentioned this in your flavors. You can define the types of hardware resources you're gonna need, things like how many cores, CPU capabilities and limits, affinity and anti-affinity. One thing I wanted to mention with anti-affinity is I believe you can use that to promote host isolation, so that is the only thing running your VM. And at the lowest level, once you're on a compute host running your VM, you can do CPU pinning and NUMA awareness, basically saying I want this application to run on this set of CPUs, and I don't want it to go across NUMA nodes. So let's jump in now to storage bottlenecks and how to alleviate them. So we've got our compute hosts on the left, we've got some storage servers on the right. Let's say that they're in a cluster of some sort. We'll spin up our very important VM, and in this case, let's say it's producing, reading and writing lots of IO traffic. It will have an IO hog running at the same time, pointing IO traffic at the same cluster. The points of contention that can be experienced here really depend on what the storage backend is. If it's a Swift cluster, the first thing, and let me see if I can get my handy-dandy laser to work, the first point of contention is gonna be right here because Swift has a proxy server that sits in front of the nodes. So if the traffic has to share a server, that's an obvious point of contention before they even hit the storage servers themselves. If this is a Cinder cluster, once a volume is assigned to a VM, the traffic is gonna go directly from the VM to the storage server. And the way that you can hit contention there is if our data hog down here, or IO hog, happens to have a volume assigned on the very same storage server. And if this is a Sef backend, things are very distributed in Sef, so you could argue that that distribution makes contention less likely, but it is still possible. Let's talk about the cost of storage contention. I ran a quick test to see if I could actually get things to contend for resources and get a noticeable problem, and I actually did. I ran an OpenStack Read-Write IO test. I had two VMs running on the same host, in my case. They had different volumes. I had a three-node Sef cluster running an active, active, active, meaning that the content was getting distributed across all three. And when I had both VMs reading simultaneously, they both experienced an 80 megabyte per second drop in read rate. When they were writing simultaneously, they experienced 100 megabyte per second drop in write rate. So this is rather significant, and it wasn't hard to make it do this. So what are some of the things you can do to differentiate and prioritize your storage resources? Again, at the top level, you can do host aggregates. You could define a set of storage servers that's dedicated for your more important traffic, and that way you're not gonna contend with anything else. You can also use flavors again here to your advantage by defining IO bandwidth limits for your outbound traffic, your outbound IO. What that would look like, your very important VM would have full bandwidth to be able to write out the IO. A less important one might have a bandwidth restriction, so it can't put out as much IO traffic on the network. And one thing I wanted to say here is that's great and all, but once the traffic is actually out on the network, and getting to the storage cluster, it's kind of all bets are off. They're equal citizens, and there's no way to differentiate the traffic, except for the fact that there's just less of the non, or less useful traffic, if that makes sense. And finally, a little more finer grained, you can actually differentiate at the storage backend, and the way that you do that depends again on what your storage cluster is made of. If it's Cinder, you have things like Quas specs, volume types, and priority, which can give more IOPS to particular volumes. If it's a Ceph storage backend, there's things like storage types and the ability to limit IOPS to certain spindles using crush maps. And as far as I know, and you can call me out if I'm wrong, but I don't believe Swift today has any way to differentiate or prioritize at the storage backend. So in conclusion for my section, network Quas is really only a partial solution. The only way to guarantee resources for your mission critical applications and data is to look at a solution all across your cloud resources. Network, obviously, but also compute and storage. And it's gonna be complicated to do this and to get it right, but it's something you really need to think about upfront. And that's it for me. And on to Alka with a real world use case. Hi everyone, my name is Alka Sapnor and I'm from Cisco Systems as well. I'm a member of the DevOps team and the team that I work with is involved in deploying, maintaining and operating private content delivery networks for service providers. That said, when Anne hit me up to come up with a real world use case for this talk, the first thing that popped into my mind is some of her customers are looking to migrate from the bare metal edge cache and mid cache servers that we're currently using in their CDNs on to virtual caches in OpenStack. So that said, based on what we've heard from Anne and Robert, what are the things we need to take into consideration to meet our performance in the CDN that we're looking to provide for these service providers in the future. But before we start looking into the future, let's take a look at what's deployed today in the customer environment. So today we have these origin servers which host the content in the CDN. We have two levels of bare metal mid caches and edge caches which basically provide the content to the end clients. And on the left hand side here, we have this gray cloud that hosts the CDN's control node, the CDN's monitoring nodes and the CDN's analytics nodes. Now today all of these nodes in the gray cloud are on virtual machines. So what does the control node do? The control node basically pushes the configuration down to all of the caches and other servers within the network. It also pulls the stats from these cache nodes to keep track of the performance and capacity of the CDN. The CDN monitoring system pulls the servers in the network for system stats and it's also our alert management system. And finally CDN analytics basically collects all of the raw logs and does the big data analysis. Now that said, when we're talking about transitioning all of these bare metal caches onto OpenStack, as with any CDN, the three important things to keep in mind are performance, availability and capacity. So how do we go about doing this migration to ensure that we don't compromise on these three goals? To start off with, all of our cache nodes now are host aggregates, all writing to a dedicated storage cluster going into the future. But we also heard Ann and Robert talk about the congestion in compute storage and network. So what are the things we need to consider to ensure that we meet our performance goals? Well, we need to make sure that we employ anti-affinity on the compute node such that all of the available resources on the compute node is dedicated to the VM that's hosting the edge cache and the mid-cache servers. We also need to employ CPU pinning and NUMA awareness to ensure that we reduce the cost of context switching. And then from a storage perspective, we need to deploy a dedicated storage cluster to avoid storage congestion. And last but not least, from a network cost perspective, today we already enabled DSCP marking on the egress traffic that's going out of the edge cache to ensure that the video traffic gets priority to reach the end clients. So how does content actually make it down to the end client? So let's say that you as an end user have your iPads and your Android mobile devices and you want to binge watch Game of Thrones day. So you select the app that's provided by your service provider. You click on the play button of the content that you want to watch. The request now comes to any one of the edge caches in the network. And the edge cache looks to see if it has the content locally. If it does, it serves it up from the local cache. If not, it goes up to its immediate parent, which is the mid-cache server. The same logic applies here. If the content is available locally, it serves it up. If not, it goes up to the origin server, requests the content, caches it along the way, and sends it down to the end client. So all of this addresses our performance goals. But what about availability and capacity? Now this is where we introduce another component in our orchestration layer known as the director. The director will have to monitor not only the CDN, but also the open stack, collect the stats, and make an informed decision to dynamically bring up an edge cache or an edge cache group, thereby addressing our capacity demands. However, from an availability perspective, we have an inbuilt redundancy within the CDN. This, along with the capability of dynamically spinning up an edge cache, addresses both our availability and capacity goals. So in conclusion, it is absolutely possible to dynamically expand a content delivery network. However, the orchestrator has to ensure that network, compute, and storage give top priority to the application traffic. Thereby, we meet the three goals that we started off with performance, availability, and capacity. That brings me to the end of my segment of the talk and back to you, Robert. Well, great. So we've talked about quality of service from a real-world use case, right? Using DSCP marking to make sure that traffic is delivered and making sure that our storage actually has the right availability to ensure that the application gets its best access to the storage resources and pinning our resources onto specific CPUs so that the CPU doesn't actually add additional jitter or latency to the forwarding of our traffic. We talked about CPU, network, and storage quality of service capabilities, the technologies that are available through OpenStack today. To that end, we're done. But for me, I like to answer questions. So if there are any, I'm happy to do that. Yes. So the question was, is there any work going on between Cinder and the networking teams, the neutron organizations, to support? I'm assuming you're thinking like the network quality of service resources. I haven't heard of any. I don't know if anybody else, anybody from the Cinder or neutron teams here that have heard of any quality of service integration work going on. No. Yeah, so that's basically taking that scheduling. Oh, do we have a mic somewhere? There's a stand-up mic. There's a stand-up mic. Oh, yes. Yeah, anyway, so I'll just finish that. So what you were saying is that the work that it does exist currently is between Nova and Neutron. I think what's actually going on with that? OK. Yeah, but that actually makes sense. It's effectively like the RXTX Factor component that was a part of the Nova flavor that implemented capability quality of service in the Neutron domain. I had a quick question for the second presenter. How quickly were you switching between the two NUMA nodes to get the 0.2% packet loss? I was just curious. Let's see. I didn't measure in terms of time how fast, how long it took. I measured in terms of how much was actually dropped. Yeah, I mean, is it like you were switching every second? Like how many times? Like how fast? Oh, I see. Like I said, it was a very simple test. I just basically aimed the traffic at it and saw how much it dropped. So I only had to do it once in order to see that drop. Oh, wow. OK. Yeah. Thanks. Other questions? When you're doing a real course on the network, how do you deal with unexpected L2 broadcasts? When a non-unicast traffic or some bogus broadcast traffic, it can seriously reduce performance. Sure. So this is where the physical network platforms, and actually even the capabilities that exist within OBS and Linux bridge kinds of networks, become important. Even in the layer 2 space, you have potentially multiple queues. There's at least a priority queue and a regular queue in most environments, or at least the ability to establish a model that looks like that. So dealing with unknown or unexpected traffic that drops into the generic queue in most layer 2 environments, establishing a quality of service mapping is actually an important part of understanding the applications that are running in your environment and then mapping that to both the physical and virtual spaces. Unfortunately, there is no easy way of dealing with unexpected traffic. So it drops into that one queue, potentially impacts that one queue. And if you don't implement any quality of service, all of your service is going to get impacted by that. The models that we built, actually some of the models that I was helping Ann with a while back, were looking at separating out multicast traffic. So this is traffic that has jitter and latency sensitivity. More important than just available bandwidth. To a certain extent, you have to have enough bandwidth available to deal with your traffic needs. But it's the latency and jitter sensitivity that you're usually trying to use quality of service to help you implement. The first place I ever saw quality of service was with voice traffic. We had IP phones, the early IP phones. The first ones I had, you'd be on a call and suddenly the entire call would drop because your jitter got too high. Your latency just got too high. The connection just disappeared. That was the first thing we were trying to address because the rest of the traffic was still an important enterprise critical traffic. It wasn't like I had some massive burst that caused that. Just somebody went to a web page and that was enough to cause that kind of jitter and latency in the 100 megabit networking days. So those are the sorts of things that you have to deal with but you have to then map out how quality of service is going to be implemented in, I think probably more importantly, the physical network because that's where you start aggregating more and more and more bandwidth. So if you have a proper class map in the layer two space, or even look at the queuing types of capabilities that you have in the layer three services space, that's how you can deal with the fact that you're going to end up eventually with a broadcast storm potentially. How do you deal with that? That's what the physical network hardware is really good at, right? Great. Any other questions? If not, well thank you very much. We'll still be here if you have anything I'm going to ask that away. Thank you.