 So I think it's time. So let's get started for the last presentation for today. Welcome, everyone, on this Extending Network Quality of Service presentation. We have Lajosz Katana on stage. He's the neutron PtL, this cycle. And my name is Boyaj Gibizar. I am one of a core developer. Let's see what we'll talk about today. Lajosz will talk us through what is NetworkUS in OpenStack and why you need it. Then I will give a brief history of what we did since last time we were here in Berlin in 2018. And then at the end, we will show a some live demo about the minimum packet rate feature, the last part of this set of features. So let's get started. Lajosz, please. So the first question is, what is NetworkUS and why you need it? So actually, Quality of Service is the ability to guarantee for you network requirements, like jitter, reliability, things like that. Generally, in networking, that is the task of switching or switches or routers to guarantee make this requirement work for your applications. In OpenStack Neutron, there are several Quality of Service types which you can select for your applications. So the API provides five QS rule types as of now. I try to list them in more or less historical order. So the first one, that is the bandwidth limit to limit the bandwidth of your ports, networks, or floating IPs. And your applications can use that limited bandwidth only. The second one is the DSCP or differentiated service code point marking. That is to mark the traffic on your ports or networks or floating IPs, and other network devices can treat that traffic with higher or lower priority and forward those packets based on that. Minimum bandwidth is the next one. That is to offer minimum bandwidth for your ports or networks, which will be guaranteed on your host. The next one is a packet rate limit. That is to limit the amount of packets which your ports or networks can use. And the last one is the guaranteed minimum packet rate feature. That is similar to minimum bandwidth to guarantee that your applications has always the required packet rate capacity on your host. Actually, I would like to highlight two types from these quality of service rules. The first one is the guaranteed minimum bandwidth. Tibi, could you go back on, please? So this was the first feature which actually made it possible for the users to make the scheduling of your virtual machines on a host where there is always enough bandwidth available. So why you need it? So it's always possible that network heavy applications, they always need the amount of bandwidth during their lifecycle on the host. So it's not just for an or whatever. They must be sure that they are placed on a host where there is the bandwidth and they can use it during their lifecycle. So we have in Neutron now with this feature, we have two types of enforcement. The first one is the data plane enforcement. That is, let's say that is the traditional networking way of making the quality of service guaranteed for the applications. That is when in case of Neutron, Neutron will make sure that your packets are in a queue and your application have the minimum amount of bandwidth they need. The other type of guarantee which actually was the first with this feature, that is to have scheduling or placement enforcement. In this case, Nova will place your virtual machine to a host where there is a guarantee that the bandwidth is available. So it's not just a networking guarantee, but even the host will be guaranteed that has the necessary bandwidth in kilobit per second. And why we need placement? So yeah, placement in OpenStack is the service that has all these informations collected for scheduling. So it has everything now. How many vCPUs we have on the host? How many memory? And now with this feature, we have the bandwidth also. So you can make placement know about with your configuration on your host to know what is the maximum amount of bandwidth in kilobit per second which your virtual machines can use on that specific host. The second feature, which actually really similar to the previous one, that is guaranteed minimum packet rate. That is to guarantee not the bandwidth, but the minimum amount of packets which are packet rate that is available for your virtual machine on the given host. This feature is really strange from neutron perspective because there is no data plane enforcement for it. So it's a scheduling enforcement only which we have finished for this. So your virtual machine will be placed to a host where there is enough packet processing capacity. And it is guaranteed that it can use that for all the time during its lifetime. Thank you, Laos. So as I mentioned, I would like to give some history of this. 2018, we were also in Berlin. And the time we showed the first work in progress presentation about the minimum guaranteed bandwidth. And since then, each OpenStack release contains something extra, top of that. So let's quickly go through that, what happened when. So Stein was the first release after the previous Berlin summit. And there we released support for basically creating and deleting VMs with guaranteed minimum bandwidth QS ports. Then we move forward and added support for more complex lifecycle operations. We added cold migration and resize support for in-train. Then we added live migration allocation on-shelf support in Ussuri that's completed all the move operations for QS features. And then one thing left, adding support for attaching and detaching interfaces from running VMs with QS policies on those ports. But for that first, we had to step back and give or implement the generic support for attaching and detaching SRIOV interfaces without QS. That was done in Victoria. So then in Wallaby, we went back and added the QS support for the SRIOV interface attach and detach. This completed the minimum guaranteed bandwidth feature. It took a couple of releases, as you see. And then in Xena, we started looking into how to do the same thing for the minimum guaranteed packet rate. We basically was able to implement everything in the Xena time frame. Just we wasn't able to merge it in time. So in Xena, you only see the specification of the minimum guaranteed packet rate. But in yoga, you got the whole thing, old lifecycle operation support for the minimum packet rate as well. This also shows that when we did the minimum bandwidth guarantee, we build up a framework that now it makes it easy to add the next one, next QS support in OpenStack. Because, as you see, it takes six releases to give the first one and basically one and a half to give the second one. You can imagine that this was a multi-year effort, but you can also imagine that this was a multi-team cooperation. So I would like to take the chance to say thank you for everybody who helped make this happen. Special thanks goes to Benzer, who helped a lot in the networking area, and also Przemek, who cannot be here today, who also helped a lot in the networking area. These, of course, couldn't happen without the Nova and Neutron core team to give us feedback and actually make sure that what we merge is working. Just to give a perspective how big this work was, during the last four years, we merged 40,000 lines of code for this feature altogether in Nova and Neutron in more than 200 commits. If I have to guess, that means there was at least 1,000 rounds of reviews on those codes. So again, thank you, everybody who helped this. This raised the question, OK, if you do all these, what's next? Pajos, please help us. Actually, for minimum packet rate, there are small things still merged. So actually, those are the CLI support and the hot template support. So the patches are there. The review is missing. So that's some organization. So that's not actually code writing, but just to push over the gate, these things. And there is a bigger thing that is actually a quite new feature for this. That is the supporting bandwidth, quality of service in multi-segment networks. So for that, the first step was done in placement to have any trade support that's finished. And the next step is to have a Nova and Neutron specification for that. And that will pave the road for the next steps. OK, so if anybody wants to help, that's a good place to join to this work. Contact us, and we can help how to continue this. As we promised, there will be some live demo. So let's see if I can make this work. Just a second. OK, you see a terminal. Good. You can see that's font size. I have a single node desktop here on my laptop. Nothing special. Let's get into it. I pre-configured these both with bandwidth and packet rate. Let's pray for the demo gods. Bit. Come on. Yeah, it's up. It was just slower because this is demo time. It's a lot slower. Come on. OK. So as I said, this is a single node desktop. So we have one compute node. I don't know what's happening. OK. So I think that will be slower than I expected. It worked this morning. Sorry for that. Then I will just go through what I would show as a plan B. So this desktop was configured with both bandwidth and packet rate configuration. If you look at the OVS configuration file for Neutron, OVS agent configuration for Neutron, you will see bandwidth configuration there and the packet rate there. Yeah, it's still there. I configured basically five megabits per second in both direction for bandwidth. And I specified 100 packet per second for the OVS. And then if you have this configuration, then the Neutron OVS agent will report these to the Neutron server. That will report this forward to the placement API where basic resource providers and resource inventories will be created based on your configuration, where basically there you specify how much bandwidth your compute host has. And that is based on, in case of bandwidth, it's based on a bridge in OVS bridge. In case of packet rate, it's based on the whole OVS. If you are using OVS on the hypervisor, if you are using OVS on a smartNIC, then you can specify packet rate per direction of that OVS. There are separate configurations for that. So if you did this, then there will be a corresponding placement resource inventory there. And then you can start creating the API pieces to consume those bandwidth and packet rate resources. So one thing you need to create is a QS policy in Neutron. And then you can add QS policy rules to that. You can specify how much bandwidth you want per direction in that QS policy. Also, you can add a rule for the packet rate. As Laos mentioned, the CLI patch hasn't merged yet. So I plan to use Curl to create that rule. And there you can specify how much kilo packet per second you want. And there are two ways to request either a directionless packet rate if you are using OVS on the hypervisor or direction-aware OVS if you are using OVS on a smartNIC. If you have this QS policy, then you can connect that to a Neutron port. When you clear the port, you can specify the QS policy. And then this will express that this port requires bandwidth and packet rate to be guaranteed. Then when you boot a VM with that port, Nova will first detect that the port has a resource request like bandwidth and packet rate. And Nova will incorporate that resource request into the placement query that Nova scheduler runs to find the place where those resources are still available. Of course, it's also combines in other resource requests like what you specified in the flavor. So placement will then, based on the inventories that was reported by Neutron server, finds those compute hosts where they are still. That amount of free resource is available. And then the Nova filter scheduler still filter down to that a single host selected by other means like pneumatopology and PCI devices. And then when the Nova scheduler selected a compute host, Nova will also start allocating those resources in placement that will complete the resource view in placement. So placement will know the resource inventory your compute source has and the resource usage based on what the scheduler selected, what host the scheduler selected for your VM. And then Nova continues booting that VM as usual. So that was what I planned. We have still a bit of time, so I will try to reboot this. But actually see what's happened. In the meantime, if you have questions, then I think we have time. Yes, please. OK, so the question was that is there a way, because it's visible from the talk that we are talking about database rows, basically numbers in the database when we say that this host has some amount of bandwidth and some amount of bandwidth from that is used. And the question was can we somehow correlate this with the reality on the compute host? That compute still really has that amount of bandwidth available. And the answer there is so the placement enforcement part of the feature doesn't handle that. It's basically you have to, I mean placement has to trust the operator what the operator says. But on the other hand, for the minimum guaranteed bandwidth there is data plane enforcement, where your VM is basically throttled when it's trying to use more than what would make possible the other VMs to use their guarantee. So in that way, you have some of the guarantees. Still, this doesn't solve the problem when the operator defines that this compute has 10 gigabytes of bandwidth. But in reality, it only have five on the line. That's something that is outside of this feature that's probably need to be handled by monitoring, open stack monitoring or something like that. There is definitely possible. It's not implemented in neutral nova. But you can build up a monitoring system that can try, look into the placement API to see what placement thinks, what is available, and try to correlate that with reality on the hypervisor. That's possible but not implemented. Actually, when we started the feature for guaranteed minimum bandwidth, we thought about such feature to add kind of script or whatever for the operators to help them automatically feel these values in the config. But the decision was that it's not possible to have a full picture of what you have on the host. So it will not provide you a full picture for it. So it's better to keep the human mind in this and do it with the human thinking. Yeah, so one thing we could do there is let some code in Neutron auto-detect the size of the link and configure the available bandwidth based on that. That's one way to do that. We choose to not do that and let it done by some configuration management to configure Neutron based on that information. That simplified our work. OK, let's see if I can actually do a quick demo in six minutes because it seems not working. So with the repetition, this DevStack is configured with resource provider bandwidth on the BR test OVS bridge, five mag in each direction in the Neutron OVS configuration. And that BR test bridge is mapped to FizzNet zero in my configuration. It's just an information that we use to create a network for this FizzNet. And I also configured the resource provider packet processing without direction configuration for 100 packets per second for the packet rate feature. OK, so if this is configured and an OVS agent is restarted, then the OVS agent will actually report it to the Neutron server. And then in the Neutron API on the Neutron agent show, you will see similar configuration appearing in the object. These are the same configuration I did, but the Neutron agent also extended that with some default values, which are also configurable like reserved values for each resources. If we have this and Neutron server was able to communicate with the placement API, then placement will have resource providers. You see three big boxes. I will zoom in shortly. This is the placement view about the single computers I have in this setup. The three big boxes are resource providers in a tree-like structure. The top of that is the regular NOVA root provider which providing memory disk and VCP resources, reporting memory disk and VCP resources. But under it, we introduced two child providers. Directly under the root provider, we have the OVS agent provider that provides packet processing capacity as configured. As I said previously, if you use OVS on the hypervisor, you will have a single resource pool per OVS. If you use OVS on a smart link, you probably want to separate packet processing capacity per direction, and you can then configure that differently. And under the agent provider, you have the bandwidth per bridge. As you saw in the configuration, BR test has configured bandwidth. OK, so this is the placement view without any allocations. I just configured the compute host with these configurations. But now I can start consuming that. First, I will create some network on that fizznet zero and some subnet that's just because I need ports and ports need leaves in networks. And I will create a QS policy. First, that will be empty. And then I add three QS policy rules. Bandwidth is per direction. So I will request one max per each direction with this QS policy. And I will also request 100 max. Just a note for this. So you can have a QS policy for networks, minimum bandwidth. But this feature works only for ports. So you have to create a port and associate the quality of service with that port. And you have to boot your VM with that port to make it really scheduled based on that. Yeah, the generic rule here is if you want to use any odd ones networking feature with your VMs, then please pre-create the port and pass the port to Nova in the VM boot. Don't expect that Nova will figure out what kind of port you need by just passing a network. OK, so I created a QS policy with three rules. Two bandwidth and one packet rate rules. So now I can create, yeah, that was the QS policy. I can create a port with that QS policy. Then we will quickly look into the port just to show that this is connected to the QS policy I created and one extra details. Basically, Neutron translates the QS policy rules into basement terminology. So the Neutron port has a new attribute called resource request. And that's a collection of resources the port requires. And that's already formulated in the placement terminology. So it's already talks about resource classes and traits that is what placement understood. And this is the information that Nova will read out and combine into the placement query when we boot a VM. So let's boot that VM on this port with this port. Then after it's booted up, we can look at the placement again and we will see that there are now resource consumption as well, not just resource inventories. Let's hope it will boot. Yep, it's booted and now looking into placement. So the same picture as before, three big boxes, the resource providers. But now there is a third, the fourth box in the bottom that represents the virtual machine we just booted and it's connected to all the three resource providers and allocating different resources from the different resource providers based on what they are providing. And if we look at, for example, the bandwidth provider, then it now has not just resource inventories like five mags total in egress direction, but also there is usage one mag which is basically used by the VM, I just booted. So this was what I wanted to show and thanks to the demo guys, second time it worked. So thank you for the patience. And if you still have questions, we are here to the week, but I think we are out of time right now. Thank you.