 morning everyone and sorry for let you waiting so long because my computer was stuck. Welcome to this session and in this session I would like to tell something about how can placement help you to achieve. I'd rather change the title to more customized scheduling by integrates with different services. And I can also see some key contributors to the placement service and please point out if I write something wrong so that we don't send wrong messages to the audience. And I will also share some example with current implementations in NOAA, Neutron and Cyborg and this can be expanded to any other similar services if you want to use placement. And first please let me introduce myself. My name is Zhen Yuicheng and I'm a software engineer from Huawei and I have been contributing to OpenStack for several releases and below are my two colleagues and sadly due to some last-minute arrangement change they cannot join us here today so I'll be the only one talking here. And let's have a preview of what we are going to cover in this session today. First we will have a brief introduction about what is placement and why do we need it and together with some must-know concepts about placement. Then we will have a look at the most important use case currently about placement is how NOAA uses placement and what's the problem did they solve. And after that we will have a quick peek at some ongoing implementations about how we use placements with other services to interact with NOAA for some more customized scheduling process and then so anyone else wants similar features can like see how it works and do its your own implementation. And finally we will briefly introduce what the team is working on in the current release and what kind of interesting features can user expect it for Stein. So let's start the first topic is the placement in a nutshell. Let's have a little bit of the placement history as the start. Placement is introduced in Newton and it first come up with as a part of NOAA and the goal is to enable more efficient accounting of resources in placement deployments and to achieve a better scheduling of virus entities in the cloud. And it is a separate REST for API stack and the data model used to track resource provider and inventory of and usages. It also has some different classes of resources. As we have just released the Rocky release so placement have been already gone through four releases and our contributors have added a lot of wonderful features in the past releases. This made the placement much more powerful and easier to use. Here I just highlighted some like highlight features and I have added the list in the reference so if anyone interested you can check and for example we are allowing users to define customer resources classes start from Okata and this makes user easy to use the placement service to their own projects as they can easily model their own resources as some customer resources classes in placement. And the impact we added this trace API and we later allow user to query resource providers by tracing twins and this again gives very much more convenience to user when modeling their own resource providers. We will cover the concept of trace later in this session. And in the last release, Rocky, the concept of aggregates was added and if you are familiar with NOAA, then you must know that it is a group of resource providers. So it may have some similar features and user can set up aggregates and query allocation entities by aggregates to have some more customized scheduling process. And with the growing of maturity, more and more projects starts to consider adopt placement as their resource management service such as Newton and we will give an example about it. And Cinder, I heard from the forum yesterday that they will consider to use this to model some shared storage thing and Blazer. I know they are working on some features and also Cyborg is planning to use placement to interact with NOAA. I will cover the neutral on Cyborg as an example later in this session and as they are the very first project interact with NOAA using placement and have more clear road maps comparing to others. And some of you guys may know that placement is not going through some extractions and there will be a forum about this in the afternoon. So if you are interested, you can go there and find out what's going on. And before we go any further, I would like to point out that although placement have simple architecture and very clear design, and there are still tons of details to cover if you want to fully understand it. And in this topic, I will only focus on the high-level resource abstractions and workflows. And by understanding this, you will at least know what things you should do if you want to adopt placement as your resource management service. And then you can get to more detailed reference about the specific parts that you have problem with. For example, Eric and Ed have some good sessions in Vancouver about the implementation details of placement and they come up with a very interesting example that you can easily very easily understanding what's going on and the link is below. Okay, let's have a look at the placement architecture. As I just mentioned, the architecture itself is very simple. And as we show in the graph on the right, it is a whisky application that sends and receives JSON requests and relational database for data persistence is just this simple. And as the status managed solely in the database, scaling placement is simply done by increasing the number of whisky application instances and scaling the database using traditional database scaling techniques is easy and efficient. And here is a very good example about how to scale the placement in real deployments. This is an example. Deployments are shared at Vancouver Summit and placement is developed together with, deployed together with Salesforce architecture. And as we can see from the graph on the left, they have deployed over our 16 placement services across 70 cells. And they have 200 compute nodes for each cell. So it's over our 14,000 compute nodes in one region. That's really pretty huge. And it's also showed that placement can serve this kind of large deployments. If you are interested in more details about this example, you can go to reference two for more details. I added the link at the end of this file. And yesterday, they just shared some of their new upgrades to the deployments. And if you are interested, you can also check the videos from yesterday. And now let me introduce some important concepts of placement. And they are very critical to understanding the placement. And if I make some mistake here, please point out so we don't send out wrong messages to the audience. First one is result providers. It's just an abstraction of data model to representing the object that provides certain type or number of resources tracked by the placement service, such as a compute node or storage pool. This is a very basic concept for the placement areas. The main resource is the resource provider. And we also have resource classes. It's the type of resource that we are, the resource provider can provide. We have standard resource providers, standard resource classes such as disk GB, memory, MB, and VCPU. Those are hard coded into the code. And as I mentioned before, users can now set their own customer resource classes. It will be prefixed with this custom underscore. And another important thing is the inventory. As I see in the board, it's a quantity of the different resource classes that each resource provider can provide. For example, resource provider one has 100 units of disk GB and 2048 units of memory, MB, and its VCPUs. And this is the quantity resources. And we also have this newly added trace. It describes the qualitative aspect of the resource providers. For example, in the previous example, we will have 100 disk GB. But we might want to describe that this 100 gigabytes disk is solid disk drive. So we can set the SSD trace to resource provider one to indicate that this compute node provides SSD disk. And when you want your instance to have this kind of feature, you just set it in a request and the scheduler will schedule your instance to the host that has this kind of feature. And it's the consumer. It's the user who occupied resources from the resource providers. For example, an instance is a consumer for resource provider one. And it's consumed 10 disk GB and 1,024 memory, MB, and 4 CPUs. And another one is the allocation. It's the model that we used to store the resource occupation relationship between the resource providers and consumers. A typical allocation records will be like consumer one occupied for your resource one and from resource provider one. And another thing is the allocation candidates. It's the placement will provide a group of resource providers that are suitable for your request and they are called the allocation candidates. And the caller like NOAA and other services may use these candidates as their input of the filter and start process as NOAA has and they can then select the best candidates they think it is. And the last one I have is the most important but also a little bit complicated concept. Now in NOAA it's the nested resource providers and in Okata release we added this to let users to be able to assign hierarchical relationships between resource providers. This could be very important for users to represent resources like NUMA, nodes, and SROV networks. I have a graph on the right which gives a simple example of the nested resource providers and as we can see in the top we have a root resource provider and we use this parent and root field to indicate the hierarchical relationship between the resource providers and we can see the top one has ID with zero and it stands for the root resource, computer node resource provider and it has like 100 GB inventories and I had SSD trace marked and it has two children with ID 1 and ID 2 each represent a NUMA cell and you can see the parent is 0 and the root is also 0 and these two NUMA cells also have children's they will with ID 3 and ID 4 and we can see the result provider ID 3 we have a parent is 1 which means its parent is 1 and the root is 0 which means the root node is 0 and we can use this field to indicate the tree architecture and this is actually very good as it's perfectly reflects that it perfectly reflects the physical architecture of the computer node so user can use this kind of structure to achieve more better scheduling and now after noting the basic concepts we are I think we are go to go to the how placement how NUMA uses placement and what kind of problem did it solve let me start with the what kind of problem did we have in NUMA before we have placement the first one is the incorrect resource usage reporting due to legacy reasons NUMA consider resources are being reported by computer nodes but only by computer nodes and when reporting NUMA calculates the resource usage and availability by simply summing across all the computer nodes in the database this causes some problems for example if some of my computer nodes uses shared storage for availability reasons what I mean here is not not that instance is using Cinder and some storage pullback and as the instance volume what I mean here is the computer node itself also uses some storage pools too for their storage or for their local disk so when the instance created by image will be using this kind of storage shared storage and for example if we have 10 computer nodes and we connected to a pool that has only one gigabytes of free disk and as each of our 10 computer nodes will see that it has one free disk and when reporting each of them will report they have one gigabytes of disk so overall the user will see it has 10 gigabytes of disk but it is wrong because in fact we have only one gigabytes of disk before the before placement came and the second one is a large-scale placement problem when scheduling NUMA scheduler will retrieve a list of all computer nodes in the entire deployments and it will loop through them across all filters enabled in the place in a deployment it is extremely useful and its inefficiency gets worse with larger deployments and the other one is the cross-project scheduling it was very hard to leverage NUMA scheduling and I had to see some other customized scheduling features by other service like routed network functionality by Neutron at the same time it to achieve more advanced scheduling process using more generic resource management services across all open stack services will make this much easier and let's get to the workflow details in order to use placements there are a few steps to follow and the first steps is to model your resource classes to the placement service and then to make your resource provider report these inventories about the resource classes as we are using NUMA compute node as an example and the resource classes reported by NUMA compute node is currently our standard resource classes so we will skip the resource class modeling part and jump to the report part directly in order to report the resource inventories to placement logic was cited in the resource tracker it is a component in for NUMA compute that collects and vertically report them to the scheduler or in this case it will report to placement as well and we added a logic to make its report to placements about the available resources and currently we only reports vcpu disk gb memory mb and vgpu and we also started to report some cpu features as traits from last release and this could be very useful too and if you are interested you can go to the reference tree for some more details and as mentioned the report process will be running periodically or whenever we perform an action that can trigger resource changes like instance creation deletion and migration and it is simply done by calling this put requests to placements for example our resource provider have 16 vcpu as inventories and I also want to point out that you can see this resource provider generation here is 66 this is added by in order to avoid risk condition and we will placement will check this version to see if any anywhere else has updated that and if it does not match we will have to try again and for more details about this I suggest you to watch the Vancouver video I mentioned before it has some more detailed information about this field and after we have reported our resource to placements the next step is to make our scheduling process use these resources with records the in NOAA case NOAA scheduler will gather all the scheduling related parameters such as vcpu memory mb and disk for user request from user request and also user may specify some specific special needs about using placement using flower extra spec and image properties this will be translated to resource provider trace if you are using placement after we gather all the information NOAA will call placement using these kind of requests as we can see from the example we will want to query resource allocation candidates that has one vcpu and 1044 memory megabytes of memory and 100 gigabytes of disk and in this example I added this required equals to SSD to indicate that we have we want to our instance to use SSD host and another thing I want to point out that placement uses micro version to control the API changes and for this kind of request you have to specify the open stack slash api slash version placement equal to 1.10 to make a success call and after this kind of request we will have a response like this the fonts is smaller but the response is long and I want to show the original thing to avoid misunderstanding and as we can see actually this request is also for micro version 1.10 I guess and so if you are using different micro version the response might be different as we can see we will have a provider summary field in this response dict and it will be keyed by the resource provider or allocation candidate uuid and we will have for in this example we will have our resource provider have this kind of trace and we will have a capacity of 256 vcpu's and six of them are used and such like this and we will also see our what kind of request did we make in the location request field and after getting the response from the placement we can now go through you know you know our case we will go through our original filter and weight process and to select the best host for our instance creation and another thing to point out that we have a resource claim process in NOAA and it was done in NOAA compute before because in order to avoid risk conditions for the scheduling process and now when we are using placement we have shifted the claim process to our earlier states by calling this put request to placement and together with all the claiming resources as its request body this way we maintain the only source of the resource and we can avoid conflicts in scheduling process even when we are using multiple schedulers and another thing I want to point out that in current NOAA sales v2 is used by scheduling and it can only work within one cells the scheduling process can only work in one cell so the scheduler will select a best host and then it will select some alternatives from the same cell with the best host and even though placement have helped a lot in the large scale problems and we still have faced some challenges and bugs for example if we are using a very large and sparse cloud for example 1000 empty compute nodes for very fresh deployments a request with very small instances or just very normal instances I mean that we don't want any special features we just want CPU RAM and disk and this request will return always return like all the compute nodes in the records because everyone can fulfill this kind of request and this has implication of memory and performance on both placements and NOAA scheduler so in order to solve this two configure options are added in QINs the max the first one is the under the scheduler section is the max placement results it's just simply as a limit to the placement called by scheduler another one is under placement sector is the randomized allocation candidates config option is equal to false and with this one why do we add this one is because if you are limiting your resources your responses what you you admit end up with that you always got the first first 1000 records by default thought and this is problematic and so we would like to run to return randomized response using this kind of config option and also in rocky a pre placement filter functionality is added and it will be function the function will work in the early phase of scheduling process and user can get improved overall scheduling process currently it supports filter resource providers aggregates by project ID and availability zone and this method helps the all the mentioned method helps a lot in improving the overall performance in real life deployments you can consider using a similar process in your own projects if you want to adopt placement uh so uh let's go through some more generic use case about placements to interact with noa and i will just simply introduce some uh noa newtron and noa cyborg interaction uh the first example is noa newtron uh the goal is of this particular case is a feature people wanted for very long period of time uh we want to be able to scheduling instances based on the network bandwidth available in the uh compute node uh after a very long time design the implementation i have to say that it is now in uh good hands and it is in the final step of implementation and uh i think we can have it in this cycle and let's have a look at how it works uh the feature is actually used uh the result necessary resource provider architecture so the first step is noa creates the root resource provider of the compute nodes uh resource provider tree and then newtron will be in charge of creating the network resource provider tree of the compute node under the no compute node root resource provider and report the bandwidth inventories and when creating instances with bandwidth requirements newtron will also provide the resource request uh as a uh a part of the part uh in newtron api and noa then will take the parts resource uh request included in the uh get uh request to placement and uh uh then as a normal scheduling process noa will select the uh placement with user response the uh suitable allocation candidates and then uh as the normal scheduling process noa will select the best host and its alternative from the allocation candidates returned by placement and the claim the resource allocations and last thing noa will also pass the allocation information to newtron during the part binding process and we will then band the part and let's have another look at the uh a cyborg example and as mentioned in yesterday's keynote cyborg is an open stack project that aims to provide a general management framework for acceleration resources uh such fpga acac and gpu uh et cetera it was launched in pike and and growing fast and for more information of the project you can go to the following the reference five and here i'm just introducing how it's planned to interact with noa uh for the current most common use cases at present uh the acceleration resources will be attached to instances as a device for users uh thus we need to uh noa and cyborg to provide a joint scheduling process to make this work so uh the best way they can find is to going through uh placement let's see how it works uh it's as usual cyborg will be in charge of discovery and managing the accelerator resources and abstract them as a resource classes imprisonment uh then like newtron cyborg will have to report the accelerator resources as a child resource provider or some inventory of a computer node resource providers uh then the user will have to specify their accelerator requests in flavor or trespack or image properties to when they're booting the instance and noa will then use uh this uh special request to placement query parameters and then get the allocation candidates uh for this particular use case they will add a new os acc library that can be used for attaching and detaching those devices to the instance uh this is like the os brick and os weave flips uh as we can see from the above two mentioned examples uh you can see that even though the details behind the placement uh concepts like nested resource providers and other uh placement implementation details is complicated but when you try to use it it's actually not that hard uh you just model your specific resource types as a custom resource uh classes and report it as an inventory or child of the existing resource providers and then query it during the scheduling process then you are go to go so I would say uh our developers have done uh very good job by making it simple to use for the users and uh don't be uh don't worry if you uh the current workflow does not suit your use case I think you can ask ask them in RRC or mail list I believe the those brilliant engineers will be able to figure out a solution for you and speaking of our brilliant developers let's have a look uh on what they are working on in this release and what can we expect it for the starting release uh first placement is going through extraction and you might end up with a separate project after this release you can get details for this in the forum in this afternoon and the second one is that I used as an example to illustrate the nested resource providers uh new metapology with the resource providers by having this we will be able to accurately model uh the physical architecture of our uh computer host and achieve a better scheduling uh the third one is the neutral example as I just mentioned it's now in good hands and as the final stage of implementation and we will likely have it in this cycle and the last two uh could be very interesting for developers who want to integrate the placements in their own projects uh the one is to support using this any syntax when querying resource provider with trace and the other one is to allow filtering by forbidden aggregates meaning user can use this exclusion mark uh number of syntax in the query to illustrate that we want candidates that are not a member of any uh specified group and uh have added all the design details about those features in the reference list and if you are interested in those you can go uh to the specific specs for details and these are all the reference and uh that's all what what I got here so thank you so any questions excellent presentation and definitely you people have brought uh in four releases something really valuable for the service providers now uh one question to you is on the database model that you are using and uh what is uh how is it going to be is it a operational database and how is it different from NOAA plus since you are also bringing the hierarchical inventory from different modules how will the migration uh what do you think of migration and what impact will be there in the next release let's say Stein to uh train or Rocky to uh Stein any comment will that will be helpful or any pointers will be helpful I think oh the when you do not data migration when you do a upgrade there is none normal upgrade you mean the can you explain what do you mean by normal I am trying to understand okay so you don't see any impact of this on the production environments if you do some upgrade rolling upgrades yeah because you remember the synchronization between the operational databases and the which we have been facing in ODL we had with same thing with the neutron at some point all those you don't expect anything to come up so operational databases are different and whatever you maintain inventory is different so you can take an inventory have the schemas and all update and all that but if you have a running platform or running infrastructure as a service instance and you want to do an upgrade don't you think it will have impact maybe we can talk offline but I certainly want to follow up on that thank you