 If I would start my presentation in Chinese, I will give the presentation in English. And of course my English is much better than English. First of all, welcome to attend this session. And I wish you also the continuation of the management of the system into the future. I have prepared the presentation and I have accepted the presentations of the Hong Kong company. And many of you know the company, but maybe some foreigners would not know the company. So I would give a brief introduction to the company. As you all know me, I would like to give a brief introduction to the company. The continuation of the system in Tindong actually started quite early in 2005. In 2016, when the cabinet became open on the source. So I would just return that to the cabinet. So maybe we are the first user to practice the cabinet. And for now, we have already performed the continuation through the state and non-state applications. And I would like to clarify that I am only responsible for the internal transaction data. It could be the IT and the business department. The data is quite huge. For one of the challenges, the database says they have different states. And first, we need to cut the line extension of that. As you all know, the stability is quite important for the system. In order to practice, we have another open source project called the database. It has a new STL approach to offer the dynamic standard like speaking and merging so that we can have the dynamic monitoring of the system. That is the foundation of the continuation. So we can have the flexible management of the resource. And we can offer the very basic mechanism. And we have a very new approach. For one of the challenges, it actually appears out of the characterization utilizing the resource management as such as other services and microservices and line services. So all the resources should be allocated to that. Because the data has different states. And it's not easy to adjust to that. And many systems rely on the database. So it's quite difficult to manage that. And we have a really high demand and dynamic demand for these services. Another challenge may be the online service and non-state application which is the most focused on CPU and memory. But for the database, we want to have a long-term storage room. So this room must be managed. Besides, we want to pay more attention to I and O bandwidth, which is automatic. And for the last on the list, the stateful application, it has quite a huge amount of data. I know it's huge cost. So it becomes another challenge. And this is an actual situation. And later, we have another approach. The person needs to understand the workload. And this instance is constructed from the actual system. It is a system of six over six hundred controls. Both of the pictures are right now. It shows the CPU utilization distribution of the different containers. And for the P19, P15, and above the P19, we can see the overall distribution of that. But the basic of that is the maximum value because it has to be much higher than the P19 value or the main value. If we want to optimize the system, we have to find an appropriate approach. Very classic approach with select P19 value. Since the difference of the P19 value is not that huge, but that of the P80 value, the P19 value is much more classic. And what if the remaining time to come to the maximum value comes to the resource management? For the first, we need to solve the sign-in. We want to give the right sign-in for the container on the database. And first, we need to have an knowledge of awareness of that. Previously, in practical practices, we may say that about 80% will be too much or do not meet that much resources. But now we have a basic approach. When we start a new business, we can adjust the amount of data like this, business type, and we can have a business profile and draw some characteristics of that. After a long time, we can draw some characteristics according to the historical data statistics and prediction. And then we can struggle it to different hosts. And we want to take our situation of affinity and stability. And we consider the container with all the resources for the P19 value. What happened if the remaining 10% appeared? So if you're quite familiar with the Kubernetes subcontinent, we have two parameters. One is requesting its need to guarantee the resources must be allocated to each drop. For now, most optimization focuses on the request of this parameter. This is the maximum value for the lowest value. Well, for the basic idea or concept of that, we can see the picture on the right side for the blue part. This is just allocated a resource according to the historical data statistics and prediction. We take the 90% value as a whole. And then we have the continental limit and that is a maximum utilization. We can set the maximum value. This part cannot be guaranteed if the competition happens. And we have another innovation in that. It is simply just a scheduling. The role of the resources of the physical service while for the cloud secretiveness allocating, it all needs to allocate the blue part to the subpar parameter not to limit the capacity of the container. It is also a guarantee to ensure that every job can have the resources. And when we do that, in this case, we have two processes. First, not all the request of resources will not be here. And for the second, the maximum value requested by each job will not be achieved at the same time. Just like the flight ticket booking, we may book the ticket, but not all the ticket will be used. And in case some do the sum of the maximum value of each job for the superior capacity to be over the three times the capacity of the container. So this is a basic concept that we need to guarantee the part but this is not the peak value. And we will also get improved to the different jobs. And it is called the shared help room from the physical service. It can be shared on physical service and we just assume that we will not reach the peak now. And as you can see from the forum shown on the slide, the help room will not adjust over the limit. And it will guarantee the basic resources can be allocated to each job well for when we can make some flexible part which can be shared by each job well for the scheduling. And we just increase the cumulative. Well, if we do the sum of the part the value will not limit over the limit of the capacity. Since we have some room to do flexible managing since the resource scheduling is not too many. The main approach for resource management it may take much more aggressive approach the performance may not be enough. So how can we guarantee the peak value and how can you calculate the specific room? Well, this is important. Actually, we'll say they will not achieve the peak value at the same time. But we can see the picture on the right side on the slide. This is a correlation analysis and it contains 60 components of the completion. We can see that the completion there's 60 components and the dark color is the combination. There's 60 components and the dark color is the combination between the two components. So if the color is not that dark it could quite align but it doesn't have to be involved with each other. This is a affinity and this is a very simple way of estimating the relation to what is continuous. And this is a formula which we can have continuous if these continuous strongly correlates with each other so that we say that they may achieve the peak value at the same time. So we'll give a certain resources so that I don't miss the remaining capacity while for those who are not strongly correlated with a part of the sum of each one or each two or each seven will not connect them together. Since this is an external value that is to say each one to each four obviously each one is lower and the value of each one to each nine so theoretically the maximum on sum must be lower than the sum of the maximum because first of all all the resources will be used if the second not all jobs will reach the peak value at the same time. This is a very simple way of estimating that this is a very simple way that comes from the practice and sort of fix that actually in the state of mind. I don't want to understand the resources I think that 90% of the money is quite good if you want a 5CPU at the maximum if you want a 5CPU and here I would just take the best line as a 99% to value if you want to do the 99% value as a best line we can find that it will be allocated with much more resources than that of the 90% but the remaining 99% if we have a lot of resources we'll just take that and go control the money and then I think that much bigger than the value of 90% value and the real money will tell us that more and more for the lower of this value but for the critical of our money we're going to the container manager to have the basic the condition is to meet the performance at least that shows the performance in a problem may occur in a way defined as quality of service violation if resources are not sufficient to have some problems and we can see that the value is about 90% and violation may reach to 6% or 7% for which we have to call 4 blocks but basically all of the right sources are coming to the maximum request and this shows that this is the same place with the resource allocation and for the size of the database we'll just take a few more basic approach one thing that I want to mention is that you can adjust it according to your requirements for example if you need it to be higher or if you need it to be 95% so that you can see it is quite elastic and flexible and another point to make is actually this has already been proved and for the current data for example the proportion 1, 2, 2 or 1, 2, 4 so the first we can save some resources because this is quite conservative or sometimes if it is quite aggressive then it will be problems in its performance so on the one hand we need to save resources but on the other hand we also need to guarantee the performance but occasionally sometimes there will be a gap but this is just occasionally and the second aspect to mention is how to manage our database first is we need to give it a right sizing and another point to make is I want to emphasise this is not just one time because it is a cycle, a life cycle and it is not real time so it is a life cycle and we will observe its changes if we find that there is an incremental trend that we will adjust it about its limit and its request so after the adjustment first maybe on our local host and we have some practices in Jingdong for example its CPU memory or its request and we do not need to restart its container because for some questions it needs to be restarted but for that there will be a high cost of the state but if we have incremental businesses and as you know that the database when it is increased to a certain degree and there will be not enough space in your disk or your IO bandwidth have some problems so that you cannot resize through your local host so that you need to pre-scheduling to migrate or auto-stamming horizontally so under that circumstances we need to have such a Vitas and this image can show you and you can divide it into one or two or three or you can just simply migrate it on this platform and to combine it with Vitas and all the challenges just as I have mentioned the database has states, it is stateful so during your migration first maybe there will be some impact but the impact will be little first you need to you need to build a replica on your host but during this process the data sometimes it may take several hours or it needs some configuration so we need to optimize this process and we need to be very careful about it so during our auto-scaling for example this container, my circle if it does not have enough space maybe you do not need to migrate it sometimes it is also available on your local host so here mainly we can do these three things first is migration because sometimes when the database is too large so we can split and merge about the host selection optimization and another aspect is addition or removal other containers so how to do it or whether to do it or which to migrate so mainly we will take some three aspects into consideration the first is where do you migrate it to for example the affinity and about your maximum value maybe you will have high resource cost adding your headroom and the second is the migration cost we will take the size of the data into consideration and the last point to make is after you migrate it we should also think about the objective host or the match between the resources so we will take these three aspects into consideration and to optimize the host selection whether to add or remove and how large was the size addition this is another aspect for optimization and as you can see the equation on the slide so during the migration of resources we have a metric that is multi-resource balance because you want to avoid the physical avoid the situation that there is no space of the memory of the CPU so you need to balance the multi-resource and balance the workload so currently it is open source and you have the node, you put it on the container and there will be a number if you put this container on this physical host how much CPU has been left for example 20% CPU or 20% memory and so that you get zero when the first one CPU remaining fraction minus memory remaining fraction that enters zero but it does not take the feature of the workload into consideration because it is when you give 5% of CPU and 5% of memory remaining fraction but however the workload does not work like this because sometimes the memory maybe 10% of the memory and 5% of the CPU so that on such a circumstance it does not work and that is why we come up with an optimization it is a simple idea and it is a multi-resource at present we have CPU and memory and we regard them as some factors when we have 70% CPU and 50% memory and the second is 40% CPU and 80% memory and we want to migrate the containers that we can realize 10% CPU and 18% memory and we will calculate it so traditionally maybe we can put it on host one and it will be better and it will be more balanced but our idea is that if the remaining fraction can match the requested remaining fraction then it will be better what does mean by matching for example if we have 20% CPU and 40% memory then it can completely match it that is why we need to take these two factors memory and CPU into consideration and how similar they are the similarity so you can see the equation on the slide and we need to evaluate the similarity between the request source and available resources even 2, 3 dimensions or 4 dimensional then it can also work so they call sign if it is 1 then it means it is completely similar to each other because these two are quite similar to each other and if sometimes the CPU remaining fraction is large sometimes the memory remaining fraction is large so this is according to the feature of the workload to schedule and to scale and that is why we need to take the workload into consideration and this is quite scalable no matter its performance or its function it can work quite well so if it is good enough then it is also okay and this is the result of the experimental evaluation you can see our server it is about 5000 servers and we take 25,000 containers to schedule and then we compare the two results if we cannot if we cannot schedule it any more that means there might be some faults and this is the current version the K8S and this is 1 yes it can be 20% and if we cannot be scheduled any more but we can still schedule about 20% so this is the result of the 25,000 containers about that means that I always work better and the second is success rate if the first is so our way is to schedule until it cannot be scheduled any more about the 25,000 containers some will be failures but some can get success so you can see the success rate on the right side so in practice I think this result is of great significance about 5% to 20% improvement so this is what we want to share my Kubernetes because it is a common system so you can adjust it according to your businesses and to optimize the system another host selection no matter if migrate or scheduling another metric or standard if it is Kubernetes it has the load balance and we will put these resources on the container to increase the users availability so at present it's algorithm is quite simple that is we should have the CPU fraction the weighted sum of CPU and the weighted sum of memory the memory fraction is to my previous idea why we need to have the weighted sum of CPU and memory if our application is CPU intensive so that we can give it a higher priority but if it is memory then we will give memory a higher priority so we will consider they should be set up at the same level about the workload and we will see the equations on the slide so after a long time scheduling because database is quite stable and we will know what kind of container does it have for example the 16G or it is a pound for 10% and according to this we can calculate out the CPU fraction and its proportion the weighted sum of CPU and the total weight sum of the memory and the memory fraction so we will take these factors into consideration and this is one way for us to improve another aspect is another way of optimization and you will consider during your optimization or scheduling of the container one fundamental thing is how to decide and the host with stable resource usage will be more usable so when you put it on the container you do not know which can be improved and it may be a flat line so it is predictable so that it can work better and this can also be taken as a metric that is mad, immediate absolute deviation so we can calculate the NAD so that it evaluates the resource usage of the dynamic system and if the resource usage changes a lot so that we can wish or not put more workloads on that container so this is another factor to consider another thing is about affinity analysis because database it is no rolling and it does not change very frequently so we can also take affinity into consideration because if you put this on the physical host and how they can reach the maximum value at the same time there is a simple way because if you already have a container to predict then it means it has high correlation and it can calculate it through this equation and just now I have already talked about the baseline that is the foundation so first how to size my circle and the idea is highly related to its state and second if you do not have enough space on the local host how to migrate or how to schedule it and how to optimize it we need to consider cost also which how to locate it the second is about the size and the scale because it should be conducted step by step and in present it is just a small scale and we optimize the techniques of optimizations of K8S as I have just mentioned also the memory including the disk but for some ones they are quite different now here it is I did the tutorial earlier I want to talk about the scale because as we have already done the scaling of this but for some times we need to check the calculation results and click the button to implement it or as I mentioned the scale is much more superior the performance is quite stable here come to the bottom part since Kevin has already been the industrial venture and my circle and other databases can also be contributionally and they can also offer physical stability and the other high-quality performance and if we can do some optimization we want to do the analysis and the choice of important idea and heuristic to guide the optimization and to guide the practice and as I mentioned the calculation is similar on this side but most of the time we want to guide the practice with the formulas most of the time we will guide the practice with some leftist font in the practice and once we evaluate the performance and we find some resources, the calculation will be increased and since we started quite early we focus much more on the cost and at the meantime if we can only do the cost effective and the default condition and we can reduce the labor cost so here is the conclusion part and if we have any questions we have three remaining if we have any questions we will please raise or you can talk with us later and answer the content and the concept the content and the cost and the extent of all the strength of the content and we have that my question is why do you think why do you think why do you want to come to the right direction and why do you want to do that because we want to have the deployment as a tool and we want to have the strength of the management resources and since we started quite early and the goal of that is actually the extent of the application of Kubernetes and we want to isolate that otherwise the cost will be much more high no matter for the VM or physical version the traditionalization is for the state I can also ask another question and we have a content and the full story aspect if you use a position volume I will briefly talk about that for all the containerization process my circle is only one aspect we have other containers and we have different solutions for recently we just have a database doc system well for the database management we use a disk well for the second question for the local volume process and when you do the migration there is a problem for the data dynamic mode so when you do the migration do you need to do you need to take the storage content from one node to migrate to another node yes we need to do that but for this system we can better support that but theoretically we also need to do that and we will have more discussion later before we have this afternoon I have another talk on that and I will talk about that in a broader sense to introduce a deeper progress in this aspect as you mentioned in the story actually we have a document support system it's called the Truba SOS it is the the best database conference that was adopted thank you thank you