 Yeah, it works. So hello everyone on this presentation about guaranteed minimum bandwidth feature. We are three of us on the stage, Miguel. So I'm Miguel Avalle, I've been working with Neutron since 2013 and over the past three cycles I have had the honor to be the Neutron PTL and it's been a great honor this past three cycles. And I am Benzer Omsic, I'm a software developer for Ericsson and in the past in OpenStack I worked for a short time with Heat and Horizon and nowadays I'm mostly developing Neutron. Yeah and I'm Balazs Gibiser, Gibi on IRC and I'm a core developer working on this feature on the Nova side. So before we continue there's another two people that we want to mention here. The first one is Lajos, please stand up. He's been one of the key persons developing this functionality and the other person that we want to mention is Slavic Kaplonsky. Slavic is one of the Neutron core team members and he's also been one of the key persons developing this functionality. Slavic we miss you. Okay so what you will see today we will try to give you an overview about this feature. Why do you need this, how this works and then we plan to show you a real live demo. We have some technical issues with the presentation but let's see if you can solve in the meantime. If not then you will see a non-live demo and then we go back to the slides to talk about where we are with the actual implementation of this feature and then most probably you will have time for questions. Okay let's dive in. Why do you need minimum guaranteed bandwidth? First of all if you have network heavy applications then the definition of network heavy tends to suggest that you need something like a guaranteed pipe coming out of your server or going into your server. A basic example of that is when you have a streaming application where each video stream has its own well known bandwidth need and each application server has a configured number of streams it can handle then you easily calculate how much bandwidth your server at least needs to serve those streams. Another example is if you have SRIOV devices in your in your cloud then most probably you are giving virtual functions to your servers and those virtual functions will compete of the bandwidth of the physical functions they are coming from and if those virtual functions are used by different servers then those servers will compete on that bandwidth and you most probably want to avoid fight over bandwidth and instead you want to split that bandwidth properly beforehand. So these are the use cases and then we can go a bit more details about how this works. This feature works the cooperation of three open stack components Neutron, Nova and Placement and we are assuming that you're a bit familiar with Nova and Neutron because those are well established open stack components but placements are pretty new so we would like to talk a bit more about why we need placement for this and what is placement. Bandwidth is a resource that is naturally belongs to Neutron to handle and in the open stack baseline when you want to place a VM to certain compute host Nova decides the place of the VM based on its resource needs and in the past this decision was impossible to make based on resources that are not owned by and managed by Nova. So one reason we introduce placement in open stack is to allow placing VMs based on resource needs that are not managed by Nova. And as in this case the bandwidth is something that managed by Neutron we need placement for this feature. What is placement? Placement is a database and the REST API and placement maintains a quite even a quantity resource view of an open stack managed cluster and we will dive a bit into that two views of the cluster. Besides all these things placement gives you a lot of benefits during scheduling it solves REST conditions and makes sure that the resource allocation could be done in an atomic way. So what's inside placement? So placement stores the quite the concrete view of the resource view of the cluster and the basic resource of resources is the compute host. So compute host is represented in placement as a resource provider as simple as the name and the UID and as we go forward as a compute host provide resources provide resources you can have placement can store resource inventory for you this compute host might provide disk memory and VCP resources and therefore in placement for this compute host resource provider there will be inventory of these resource classes and the total amount of value for each. Of course placement will store extra information top of the total available resources for each resource class but we don't dive into that right now if you are interested this morning was a good presentation from Kevin about these and also in Vancouver from Eric and Ed you can find that video online. This is the quite the view of the system so these are resources that can be consumed but there is another view of the system where you describe the quantitative part for example if your compute host provider tells placement that it provides 10 gigabyte of gigabyte of disk it might want also describe that this disk is coming from solid state disk or a spinning disk because that affects your quality of service. This information can be described and stored in placement as straight like this this is the disk is coming from a solid state hardware can be described with a storage disk SSD trait. This completes the picture so you have quantity view with resources and you have quality view with traits. Let's go forward. So this is half of the picture you can store resource inventories and traits in placement but you also consume those resources so when you go to OpenStack and boot some servers NOVA will eventually create consumers for your VMs in placement that are consuming resources from resource providers and this way placement will have a view not just about the total available resources but what is used and what is still free for new allocations. These are the simple things and if you go forward then there are complications your hardware already complicated than that your compute host will have more than one physical network device and if each device provides some bandwidth some resources like bandwidth then you want to express them as separate resource providers because you cannot consume from both at the same time for the same virtual nick so those will those providers those physical nicks will be represented as two separate providers but you still want to express that they are belong to the same compute host because your server cannot consume from two compute host at the same time so this this creates a tree in placement of resource providers where the compute host is a root resource provider and there could be many layers of child providers under them 100. This is all you have to know from placement so that now we can talk about how we model bandwidth in placement Mikuel we talk about that. So before implementing this feature we were really faced in the between Neutron and Nova with a dilemma on one hand in Neutron we have had a minimum bandwidth quality of service rules since many cycles ago so we were so with minimum bandwidth you were able to say okay I want this port and I want this port to consume to be given at least this minimum bandwidth in this nick so we go and we wire the virtual interface and we wire the bridge or we wire the nick in Neutron in such a way that we deliver in the data plane we deliver that minimum bandwidth to the nick to the virtual interface of the instance but then what happens if Nova comes and schedules such a big number of instances in that compute host and oversubscribe the capability of the nick of the physical nick of the host then I mean we made a promise with the minimum bandwidth rule in the quality of service policy that we cannot deliver on because Nova doesn't have any knowledge of the promises we made to the user through the quality of service so what we are doing is essentially having Nova and Neutron cooperate using the placement the the placement API in such a way that we Neutron will let placement we create resource provider trees of resource providers in placement in such a way that when Nova schedules an instance it's going to take into consideration the promises that we made in minimum bandwidth to the ports of that instance so as you can see here we have we have the we have the root of the so we have the root of the tree which is resource provider and that's and that's the compute host and then underneath that we have a couple of agents that are also represented as nested resource provider of that of that compute host and underneath that we have virtual nicks or bridges that belong to that to either one of those agents and that's another nested resource provider inside the structure if we go to the next slide please we can see in a little more detail so this is the second and third level of the of the of the resource provider tree you can see that we have the nested resource provider for the agent and underneath that we have a specific nick in this case it's it's the nick ENS5 in the compute host and what we are saying here is that this nick has an inventory of that many kilobits per seconds in in egress and and that many kilobits per second in in the ingress direction and the other thing that we are saying there is that that that nick has those traits meaning that that that that nick is connected to that physical to that fiss net to that physical network in this case fiss net zero and that we can connect to that nick we can connect uh those uh virtual nick uh types we can connect direct we can connect mac v tab and we can connect that direct physical uh for those of you who don't know what a vnick type is essentially when when you when you create a port and you want that port to be realized in the compute host you specify the vnick type for that port in other words you're telling you're telling neutron when you when you wire this port in a compute host i want that that port to be connected directly to a to a physical network in a physical interface and that would correspond to the vnick type direct so so so you as a user when you create your your your a port you specify that at the at the binding process and the binding process is nothing more than the moment where neutron the the binding process is the moment where neutron realizes your port physically in whatever in whatever compute host that port is going to leave so so so now we have we have we we want to create this tree of uh nested resource providers for for for nova to use during the scheduling of the vm's and what we do is that that we communicate the the um the inventories of bandwidth and the and the and the types of nicks that we have in the different compute uh uh uh host all the way from the compute host to the placement api and essentially this is a three-step process as you can see here we have a we have config files and in that in those config files we declare uh we declare the different bandwidths uh physical bandwidths that we have available in different different vnicks in the in the different physical uh network interfaces the neutron agent starts up uh it reads it reads its configuration file and it communicates that that information to the to the neutron server via the rpc channel and it essentially it it says i have i have these many interfaces i have these many many bridges and in in each one of these bridges and in this one in in each one of these interfaces i have this much uh bandwidth inventory and i have this these traits and then the neutron server what is going to do is using the placement service api the rest api is going to uh create the tree of uh resource providers that i explained in the in the in the previous two slides okay so this completes the first step of of actually using this feature but as an end user you you most probably want to have your server getting guaranteed uh uh bandwidth and for that you need to do some extra steps so as a first step as miguel described neutron will periodically and continuously report resource inventories to placement and when you as an end user want to have a server uh with a port that has guaranteed bandwidth you go to neutron create a port as usual but you also select qs policy rule for that port that might be pre-created uh the administrator pre-created for you or you might also be able to create yourself it depends on the configuration of the of the cloud but that will describe uh that how much bandwidth that port needs and then neutron will return return back to you the you idea of the port as usual now you go to nova and ask a server from nova and you specify that nova please plug this port as well uh to my server so nova will know that he needs it's need to plug a port then uh nova needs to figure out what is the resource needs over our resource needs of your server and that's coming from different places one place where you specify resource needs is the flavor you're the built at the vm width uh there you specify cpu memory and disk but now with this feature a port will have a resource needs as well nova only sees the you idea of the port so nova needs to go to neutron and ask neutron what is the resource needs of this port neutron will based on the port's qs policy answer this question answer this question to to to nova then as nova now gathered all the resource needs the server needs nova can go to placement and dust placement please placement tell me where are those compute hosts where are those resource providers representing a compute host uh are in the system that are capable of support this over our resource needs of the server uh then placement will go into the inventories it knows and search database for for allocation candidates basically resource provider trees that still have available resources for your request so now i will get back a list of allocation candidates from from placement and each allocation candidate is capable of supporting uh this server so now i will now use the basic the the the normal filter scheduling process to filter those down further based on uh extra logic that nova has like numa and and and uh and others to select one single compute host one single allocation candidate that will be used for your server when it's done nova will go back to placement and tell placement now placement these allocation candidate becomes a consumer nova creates a consumer that consumes all the resources that the server needs uh from those resource providers that are selected by placement first by providing allocation candidates and then the final decision made by nova by filter scheduler to to set a single resource provider tree so now nova knows which compute host the server will will be booted on and actually nova knows which resource providers providing the resource needs of the server but as this resource is not just coming from the flavor but also coming from the port now nova needs to tell neutron what was the decision it's not new nova previously did port binding to tell neutron which compute host the port will be bound to to let neutron do some extra preparation work on the networking side but now with this feature nova needs to also tell neutron which resource provider which physical network device will provide the bandwidth uh resource needs of each given port so that will be an extension of the port binding here we also need to mention let me well like talk about the port binding a bit because that's that's something we changed so what i want to mention here what i would like you guys to remember is that in this process that we are showing here steps three and step step three and step seven were already happening anyway i mean that's that's the normal that's part of the normal process of booting an instance so nova when nova is creating the instance one of the things it does it it requests from from from the neutron rest api the characteristics of the ports that that are going to be connected to that to that instance and it also it also once once the decision is made to skate to schedule the the instance in a certain host that was the moment of binding the port in other words is the moment of telling neutron neutron please that this port is not is not a an entry in the database anymore go and and create why this port instantiate this port realize this port in this compute hole so steps three and the step seven were already happening anyway the only thing that we did is that we added some extra information some extra information in the in the in the dictionaries that are exchanged in the in that in those two steps okay so if the port binding succeeds then nova goes forward with the normal server boot processes as before and that completes the process completes the whole process and usl end user will get your server booted with the ports with guaranteed minimum bandwidth okay i think miguel we talk about two small so remember remember remember step three so in the step three always we have always when when when when nova requests the port the port information we send back that dictionary through the rest api in a response to the through the rest api the only thing that we are adding is that the port the response in the port has a new attribute which is resource request and and that that resource request includes two other attributes inside it's a dictionary with two other attributes number one the resources and what there we are saying okay this port has has a minimum bandwidth a qr rule associated with it so that's the that's the that's the that's the bandwidth that we promise to that port and the reason we need this that this that way is that we didn't want nova to have to learn to talk to the quality of neutral quality of service api we wanted nova to mostly continue doing more or less the same thing it's been doing for years so we just started this request and the other thing that we added is and by the way we since we are we are this this port has to be connected to this physical network and with this veneq type and this is a this is this required attribute sub attribute is what is going to be used against the traits of the resource providers that we created in the in the placement service here now we are looking at what new nova is sending to neutron in the binding process in the in the step number seven in the in the in the graphic again this this dictionary was already it was anyway being sent by nova it's it's a it's a it's a put request it's an update request to the port from nova to neutron saying neutron please bind this port in this host and that that was happening anyways the only thing that we are adding is in the binding profile and binding profile it's a it's an attribute that we already existed in in in in neutron that it's that was specifically created for to send this kind of additional information for the binding process the all we are saying is we are adding the resource provider uuid in other words the uuid of the resource provider that represents the veneq that that placement selected and and that way neutron has now all the information necessary to bind the port in the host selected by nova and in the interface that can provide the the the minimum the minimum bandwidth promised to the to the port so this is the place where we would show you our demo yep so basically what is going on is that we have a working live demo on another laptop which we cannot connect the projector so i'm sorry for that but i will not be able to show you that as it is but we have uh back up slides which oh shall we give it a try yes okay let me just until then for the sake of simplicity so so bear with us just a couple of minutes we're going to give this a try if not we come back to for the back up slides yep excuse me excuse me uh can i ask a question so uh you spoke about s r i o v uh network card so let's say you have one physical port and you have multiple uh virtual functions on top of it and uh nova requests let's take two virtual functions so when you make allocation uh two out of uh let's say 16 how much of uh bandwidth are you going to reduce on the physical port what did i so you promise a minimum bandwidth now an allocation has happened and uh two virtual functions have been consumed on the physical uh function so how do you partition do you like equally partition the total bandwidth across all the virtual functions or you have a mechanism to control it per virtual function so i'm not i'm not absolutely sure i understood the question but let me let me give it a try okay um so so so remember that that in in the in the response we sent to to to nova from neutron with the port characteristics we say we say this port requires this this much this much bandwidth and that that it's a consequence of associating a a a minimum bandwidth rule qs rule to to that port with that information what what between nova and placement they go and they they they select the host and the within that host the inventory that can fulfill the bandwidth available in that in that in one of the nicks of one of the hosts once no one placement take that that decision that that the the bandwidth consumed by that port is deducted from from the resource provider so the next time when you schedule another another instance against that host against that nick uh the the inventory the bandwidth inventory of of of that that that of that nick is already updated it reflects the fact that now you have the first port consuming bandwidth from from that nick is that was that the question no sometimes when you have virtual functions how do you know how much of bandwidth to reduce once the allocation is done you actually actually ask for that bandwidth so when you create the port in neutron you tell neutron how much this port will use that is at the physical level right no no that's on the neutron port logical level you will tell neutron that this neutron logical port needs one gigabyte of minimum guaranteed bandwidth and then uh this will be deduced from the inventory of the physical function that actually create that that is uh uh used to create the virtual function for your vm so you're deducting from the physical function so that's what you're saying yes and that's and that's why that's why this allows us to actually fulfill the promise if you want this minimum bandwidth we we commit to deliver this minimum bandwidth to you because now we are we are deducting whatever your port is going to consume from from from that inventory we're deducting that bandwidth from that inventory so we don't oversubscribe anymore that's exactly the point of this picture thank you okay so i think we can show the actual live demo yeah sorry for the colors sorry for the color space that's the best we could produce now but this way it it's at least live uh so let me first jump into what the demo will show you uh i want to show you uh what you need to expect as the user of open stack from your environment how you can discover that the version of open stack you're using is supporting this feature or not so we will be talking about a few neutron api extensions and placement api micro versions that you will need to use this feature as we probably already mentioned at the beginning this is a work in progress feature uh so you can reproduce all what i'm showing if you take the master branches of nova nutrient placement and a few patches that are still under review then i will show you what kind of configuration the cloud admin uh we'll need to produce to set up uh what kind of resources are available in the system these will be mostly neutron configuration files and then i will try to show you three scenarios uh which will show the feature working and also what kind of guarantees it gives you and when it cannot satisfy what is asking what what you're asking from the system uh i may have some uh maybe three colorful figures later i will hope work with this strange uh color space but let's see uh and here now i don't need the slides anymore but uh don't care about this these are just my uh own own uh notes uh this will be the one uh where you should see what is happening and i hope this is uh readable at this point so the first thing is that this demo is running in a desktop this is just the desktop directory i am using uh i already built a stack i don't want you to wait for that now and uh for the sake of simplicity so i don't have to switch between an end user an unprivileged user and an admin user i will be doing everything as an admin but i will be mentioning what kind of operations are usually decent uh usually that uh so the two api extensions you will need the two api extensions you will need from neutron and which will be hopefully merged in uh the staining release are the uh qs bandwidth minimum ingress extension which makes sure that you can create a qs policy rule which uh describes the ingress direction of the minimum uh guaranteed bandwidth ingress and digress in this case always uh are meant from the perspective of the vm so they are practically egresses upload and ingress is download then uh the other neutron extension you will need is the port resource request extension uh this refers back to what miguel already explained uh this is the machine to machine interface between neutron and nova where neutron uh tells nova what kind of resources are needed for this port uh so this is usually not seen by an unprivileged user then from placement uh what you need is at least 1.29 of the placement micro version here in this uh system we have 1.30 so this is more than enough then let me jump to the configuration that is usually preset up by your cloud admin the first and simplest thing is that since neutron server is a client of the new placement service you have to tell neutron server where the placement service is available and what kind of credentials uh you have to access it so that is what you are putting to neutron conf and uh then here comes the real configuration which describes uh your resources so what kind of bandwidth you have on your physical network interface cards uh since now we are working on support for the neutron ovs agent and the neutron sri ov agent i will show you both configurations in the ovs agent the kind of physical resources uh that we are talking about on the bridges so here the configuration describes that on the bridge belonging to fizznet zero you have an upload of 10 gigabytes and a download of also 10 gigabytes and just for the sake of an example on this other bridge you only specify uh an egress uh value and the ingress value is undefined here just just a quick observation here uh in in this uh first implementation we are we are requiring the admins to specify the physical bandwidth uh we are thinking of in future iterations to have this to have a auto discovery feature so you don't have to but even when we will have the auto discovery this will probably stay here so the admin if uh here she wants to or can override what was auto discovered uh and practically the same configuration exists uh for the sri ov agent here in this example on this enus five uh physical function we did configure that a total available of 40 gigabits I guess as as up and and down then let me move on to the actual uh scenarios uh proving that this feature works and and how it works so this will be the tricky thing I sorry I hope the colors are somewhat yeah somewhat working so uh the three pictures from left to right are the three scenarios I'm trying to show you uh the different colors are for different VMs so I will be booting all together three VMs in three scenarios the first scenario is a single VM uh the fact that that VM will boot proves that you can boot a VM which has a minimum bandwidth guarantee and uh boot is booting is still working uh with a new feature merge uh then the second scenario uh will add an extra VM on top but in such a way that all resources should stay under what is the total available so both all the CPU RAM and bandwidth resources uh should be sufficient to boot the second VM and in the third we a third scenario I will be replacing the kind of orange uh VM to the brown VM and as you can see the brown VM uses the same flavor as the orange VM so and the only change from scenario two to three is that we will be raising uh the bandwidth request the minimum guaranteed bandwidth request of uh the brown VM uh in such a way that it would try to go over the hundred percent of total available resources and of course that's impossible so what we expect is that the first boot will succeed the second boot will succeed but the third boot will fail and Nova will tell us that it did not find a compute host to uh to do this job for us so uh let me just do these then uh and before we jump into the scenarios I have to show you a little bit of an environment setup so these are the uh resource providers uh that uh formed the uh resource provider tree uh we already learned about uh and then I can show you that on one of these resource providers we actually have an inventory reported uh does it work for you if I still reduce this then it's still readable okay uh so what we can see here is that on the brfisnet zero uh resource provider we have an inventory uh both for egress and ingress uh bandwidth minimum guaranteed bandwidth with the values of 10 gigabytes up and down just to just to be sure that we're on the same page in these last two commands uh pens is talking placement api yep which by the way is the same api that neutron uses to create the resource providers yep usually these values are only visible for an admin so an ng server we'll not see what is the total available uh in the system in an usual configuration and uh then I must show you that the network we will be using for all three VMs is a provider network so it is a vlan type uh provider network using vlan id 100 and as in many other examples before uh we did show this is attached to fisnet zero and uh what I uh still want to show you is that we have a qs policy again usually pre-created by the cloud admin and in this qs policy we will have sorry we have we have two qs policies one providing at least one gigabit of guaranteed bandwidth and another providing at least six gigabits of uh uh guaranteed bandwidth so for some ports we will be using the using the one and the other ports we'll be using the other and that's how we can uh make the difference that I did show you in the figures in the beginning uh so let me show you that I already pre-created three ports port one two and three four VM one two and three in the three scenarios uh the first VM will be using six gigabits of uh guaranteed bandwidth the second will be using one and the third again six so the first two together should be using a total of seven that's under the 10 what we have available but the first and the third together should be using 12 that's above the 10 we have we have available uh okay so finally I can really jump into uh the demo and hopefully if I don't have a demo effect I can show you the O at the beginning yep thank you if I don't have a demo effect I should be able to show you that I am just booting a VM that requested six gigabits of uh guaranteed bandwidth yep and because the VM went to active uh uh the request must have gone through and let me prove that to you by asking for what kind of resource allocations are now present in the placement database and what we see here this is only part of the actual resource allocation so the resource allocation is made for the uh whole VM and but this part only shows uh that was made for the uh port of the VM uh so uh we did not have just the VM booted but the resource allocation actually happened now uh I can move on to the second scenario as a reminder the second scenario adds the orange VM on top but still staying under the what's available uh so I should be able to boot VM 2 adding one gigabits of guaranteed bandwidth on top and this again should succeed yep it went to active so it succeeded and now what I wanted to show you is uh that previously I did show you that some allocations were made uh proving that something happened in placement this was the allocation of the six gigabits of the first VM and now I can also go back to placement and ask for not just what kind of allocations were made but all together how much resource was used up so far so I can show you that uh seven gigabits is now used up from the 10 we have and then we can jump ahead to uh the third scenario in which uh uh sorry not there in which we would be trying to go over the 100% mark so we expect this to fail yep and in order to do that I'm first deleting VM 2 I'm replacing that so I'm also waiting for VM 2 to go away so the resource allocations get freed and uh creating VM 3 instead which again uses the same flavor so the CPU and RAM uh requirements are the same but now we were requesting six gigabits of uh uh bandwidth so the boot fired the system couldn't place uh uh the VM as expected uh let me just show you as a final part of the demo is that yep is this the one that the error message was actually know well it was was found so this was really a failure coming back from the placement of the VM and that concludes the demo and if we still have time let me just jump back to the slides I think I think I only only use like three sentences to close this up uh one thing is uh what you saw is working progress uh most of the things that are in this demo is or all the things that is in this demo is available either as merged code or a carried review uh you have the links there uh but there are things that are not working yet uh like you cannot move VMs around having minimum guaranteed bandwidth or you uh cannot use multi-segment uh neutron networks uh with this feature we will be working on that uh as as we progress uh and on the last slide you will see uh a lot of references uh that uh where you can read more about these features specifications and Ben's already published the blog post about the demo technicalities so you can reproduce the demo in your own environment and look at the details okay thank you so I just I just want to say that as the neutron PTL it it fills my heart of joy that it's Thursday at 6 p.m. and you guys are still here watching a CLI presentation thank you so much thanks a lot thank you