 Hi, okay. Good afternoon everyone. I'm Yixin Lin from IBM and standing besides me are Dota Teng, Qiming Teng and Dota Hu, Yan Yan Hu. They were both from IBM Research Lab and in this talk, we are going to presenting the deep dive of selling project. And we are going to talk about why do we start selling project and what is this service is about and where we are today and where we are heading in the future. So the first part is why do we start selling project? Back in 2013 we have some requirements from our customers to provide some high availability solutions for their virtual machines. And then we start a research project to investigate how to do it in OpenStack. And besides the high availability solution, and we also looking into the auto scaling and the high availability and low-balancing technologies. And heat and salameter were both into our field of vision and they were both incubation project back then, but they all sound promising. And heat provides the auto scaling group resource and the scaling policy resource and the HA reserter resource to build up the high availability solutions, auto scaling solutions. And together with the salameter, and it might be the solution that we can provide to our customer. So we start to investigate the heat and the salameter. And we list some user cases that we need to cover. As you can see here that not all of them passed. Some of them are still not covered by heat and the salameter. And we also did some analysis between the gaps of the AWS way and the OpenStack way. How the low-balancing work and how to monitor the health of the virtual machine and how to recover the failure of virtual machines and how to do a cross availability zone policies. And by doing so, we found some defects in heat and we try to fix them. The first thing that we need to fix is the hierarchy problem in heat. We need to split the codes of AWS auto scaling group and the OpenStack auto scaling group. And when we try to fix them, we find that it is not an easy job. And we learn something that we didn't know before. During our job on the heat, and we learn a simple tool that is heat is just about orchestration. And heat team wants to focus on orchestration itself instead of something new or something else. So by orchestration, heat can be seen as a compiler. It compiles a structural definition of OpenStack resources into the real objects. And a stack update action can be seen as a recompile. So the auto scaling is a big problem in heat because it is based on AWS design. But auto scaling is no part of cloud formation on AWS. And it cannot fit the variable requirements from the private cloud. It is a big problem on its feet. And the heat team also aware of this problem. And they try to rework on the auto scaling implementation back in 2013. There are proposals about a new design of auto scaling. And you can find the links in here about the details. They register several blueprints for the auto scaling new design. And they were all approved back in 2013. Because it is an important feature that required by customers. Let's see what they want to achieve. All these things. If you go through all these work items, you will see it is a huge effort and a big change to heat. They want to implement a new API for the auto scaling. And a new DB schema and a client. And even a new engine for the auto scaling. What does it look like? It looks totally like a new service. It can be moved out of heat. And by moving out of heat, we have several benefits. First, the incoming codes cannot impact the stability of heat. And we can pull the whole thing stand up quickly. And because our customer cannot wait too long for the code review, you know. So during the Paris summit, we talked to the heat team about it. And then we start working on this. So when we build the selling project from scratch, we thought to ourselves, what do we really need? Do we need a resource pool? Yes, definitely. And we call it cluster in the selling project. And the resource pool, we call it a node. Now we have cluster in a node. We also need policies that can be attached to the cluster, like scaling policies. It tells the cluster how to do the scaling. So we build these functions step by step. First, we build the functions to create and manage a resource pool. And then we make the size of the cluster scalable. And then we automate the scaling operations. Now we have auto scaling group. Until now, it seems that we have advanced auto scaling group in our services. But when we built in the capability of managing a resource pool, we found that it is an important piece that is missing from the open stack. In other words, if we have a generic cluster in services, with auto scaling, auto healing, load balancing support, it will be a useful scenario of a resource pool to the open stack. And we found that all these advanced features are independent from each other, like a cluster can be for the HA purpose, with or without the auto scaling support. And it can also be load balancing, even without auto scaling. And by regarding the services that we can manage, we can in theory, manage any homogeneous services in the open stack in a resource pool. When we found that, we feel very excited because we know that we are doing something that is really, really important or useful to the open stack. And that's why we start a selling project. And with that, I'm handing over to Dr. Teng to give us an introduction of the selling design. Okay, thank you, Ethan, for the background introduction. So I'm giving you all a quick intro of the selling service design. As you can see from this picture, the whole selling project comprises several components. It's not very complicated. We have a selling client talking to the Sending API component using REST APIs. And the API then talks to the Sending Engine via RPC. In this design, we are enabling multiple engines to serve user requests so that the scalability in future won't be a big concern. We also invented some new concepts here to make Sending a more generic cluttering service. That's the profile concept and the policy concept. A profile is actually a template you can use to tell Sending how to create, delete, update an object. That object can be a nova server, for example. It can be a heat stack. So if it can be a heat stack, it means virtually we can manage any resource types. Heat is supported today. So that's a big advantage we are taking from heat. We are not supposed to reinvent every wheel. So that's one of the key design points. To make the service more useful, we also developed some policies that can be applied when you are managing your resource pool. We call them classes. This page, I'm showing you what kind of profiles and policies we are implementing today. And some are still to do items on our agenda in the Newton cycle, maybe for the future. For profiles today, we have already supported Nova Server and Heat Stack. That's the main profiles built in today. There are some people trying to help us provide a bio-metal profile so that Sending can be used to manage physical nodes. We have some collaborations with university interns. They helped us build a prototype using Sending to manage container clusters. Those are really some very smart students. They got this job done in three weeks. So we had a demo back in Tokyo Summit. We also made that container cluster auto-scalable as well. That was a very interesting work. But it's still yet to be integrated into the master branch of the Senior Service. Speaking of policies, today we are providing deletion policy. For example, placement policy. For example, you want to have your clusters deployed across multiple regions, multiple availability zones. You can do that. And you can set affinity properties using some of the policies. Certainly we will provide scaling policies allowing to specify many details if you are trying to scale your clusters in a very customized way. There are some other policies still under development. For example, the health management policy that will be one of the key features we want to deliver in new cycle. Load balance is almost ready. We don't have support to LBASv1. We started from v2 directly. That's the policy and profile abstraction. This page is very complicated maybe. But it's showing us some key design considerations we have in Senior Service. The core components as shown in the diagram are the green boxes, the sending API and the sending engine. Those are the core components of the Senior Service. All the other components are plugins. For example, if you want to extend Senior Service to manage things other than heat stack or nova server, you can just plug in your own profile. In that profile, you can just tell how to create or operate your resource as a node. To make the whole thing customizable, flexible, whatever, we really have those kind of requirements from our users. We have isolated the driver side. By that I mean today we are talking to OpenStack. We are OpenStack SDK. That is the only dependency we have on OpenStack. If you change that driver into something else, certainly you can manage your own cluster of integers, flows or whatever. Maybe pass components, web applications. It's all possible. That's another feature. The upper right corner, there is an abstraction called receiver. Sometimes we don't believe making a service too smart is a right choice. Our design policy is to make the service capable of reacting to whatever external event alarms. It could be, for example, a phenomenon alarm, monastic alarm, Zebix, Nagios, whatever. We know there are people who don't like a phenomenon for whatever reasons. Maybe they have their own data center monitoring systems. In that case, you can still trigger some cluster operations using the receivers we provide. With receivers, you can trigger whatever operations on your cluster. For example, scale in, scale out, even check, recover, those kind of things. We are making the whole thing very flexible. I think that's all for this page. Just to give everyone a look and feel what other operations we support today. For clusters, node policies and profiles we support create CRUD, all those operations. But for clusters, there are many more convenience methods such as scale in, scale out, and resize, all those kind of things. You can manage your cluster membership directly. We tried this previously using HEAT, but HEAT only provides one verb that's stack update. Sometimes we run into some problems. So that's some lessons we learned we will try to avoid today in sending. Here you can see from sending a client command line interface, you can do a lot of things. You can manage your profiles, your cluster node, you can manage your cluster membership all through the command line interface. We also have a sending dashboard project that Horizon plug-in is still under development. So interacting with sending service is really not that difficult. Here's something more, how to operate policies and cluster policy bindings. We do enable you to attach policies to clusters, detach them from your cluster. If a policy is attached to a cluster, you can dynamically enable and disable them. For example, HA policy is cool, but sometimes it's annoying. So those kind of features we are providing with cluster policy bindings. Other useful operations such as you can do cluster research directly using command line or through the Horizon plug-in. Other operations such as you can check the actions, background actions, asynchronous actions that were forked from your interaction with the API. You can check events, what happened to your cluster, your node. So you actually don't want to look at the service blog file. It's huge. Sometimes you cannot find what are the root causes of some failures. So that's ... I'm showing you one example, command. For example, the cluster resize operation. This command, for example, has been carefully designed to support many variants. You can change your cluster size by capacity, by percentage, or you can specify an absolute number you want the cluster to be. Sometimes maybe you want to scale in or scale out your cluster, but there are main size or mic size constraints on that cluster. So the service may get confused. Should I do it or should I reject it? So in that case you can specify, for example, the strict or best effort way. This is also from some customer requirement we are supporting. Sometimes along with the size adjustments, you want to tune the main size constraint or mic size constraint. For example, during weekend days early in the morning, you want to increase your cluster size. Then without scaling, no request coming in yet, the size drops down quickly. So in that case, you can raise the main size constraint and keep the water level at a certain degree. So with that, I'm passing to my colleague, Yan Yan, who can give us a quick view of what we have done during the Metac cycle and what's on our place in the coming cycle. Okay. Thank you, Jimmy. Okay. So this picture shows the journey of Senglin project in last 16 months. And actually the first line code was committed in December 2014. Actually, we initiated the Drupal of Senglin project in China Research Lab of IBM. And four months later, we were accepted by OpenStack community as a StackForge project with both Senglin client and Senglin service. And then in May 2015, we joined Vancouver OpenStack Summit and updated the project status to HIT code team. As introduced by Ethan before, that the first goal of Senglin project is uploading our scaling functionality from HIT. So it is very, very important for us to listen to the voice from HIT code team and we are very appreciated for those suggestions they gave us. And in August 2015, another Senglin sub project, Senglin dashboard, was initiated and it provides more intuitive and either way for any user to manage a cluster resources managed by Senglin. And then in November 2015, we joined Tokyo Summit. And in that summit, we gave a presentation about how to integrate Senglin and Magnum to support container scaling. And just after the Tokyo Summit, we proposed the application to join BigTent because we believe we are doing the right thing and we hope more people can join us to make Senglin better. And great news that we succeed. We got approval from TC and the TC think that cluster management is an important feature OpenStack needed and we are on the right track. So I think this is the most important milestone we have achieved over that time. And in January this year, we hold the first mid-cycle meeting in Beijing, China. The whole team sit together to discuss some important topics. And also invite some customers. For example, China Mobile, which is one of the largest telecom operators in the world. And also, for example, Jindong, which is one of the biggest online retail in China. And we believe that it's important to listen to the voice from the customer. We want to understand the problem they have met and the pain they have when using OpenStack environment, especially when managing cluster resources. And we got a lot of important suggestions from them. And now we are here in Austin Summit with our first formal release. And we also published the documentation for both user and developer. And we also provide some examples for some, you know, typical use cases to help any user to understand and to know how to use them to manage cluster resources in OpenStack environment. Okay, in this page, we list some important features we have finished in medical cycle. The first one is about API refactoring. Actually, we made some important changes about sending API interfaces in medical cycle to make the sending API interface strictly conformed to the guide from OpenStack API work rope. We believe this is very important to keep our API interface consists with all other OpenStack services. And the second work item we have done is about the support of OpenStack client. And actually now user can use both OpenStack command and also sending command to manage the cluster resources. And maybe in one or two cycles we will deprecate the command line in sending client. But of course that will depend on how OpenStack client projects going. And the third important job we have done is about sending resource types supporting HIT. Actually this job has been done for about two months. And now all sending resource types have been supported in HIT. User can define a deployment of sending resources using HIT template. And also some example templates were added to HIT template project and to provide some sample for any user to help them understand how to use the resource types. The fourth important thing we have done is about HIT management. And we actually implemented a primitive in which the workflow of cluster and node health data check and the recovery is supported. And of course this is just the first step we support HIT in sending and we will do more jobs in new time cycle to provide more powerful and flexible support here. And the next item is about documentation. We have published documentation for both end user and developer. And here I want to thank our PTR team because almost all documentation was wrote by himself. And the last one is about sending dashboard. Actually sending dashboard provided more intuitive and easier way for any user to manage cluster resources. This is very important I think. Okay in this page we show some work items we hope to finish in new time cycle. The first one is about API macro version. Because we believe without this support any change in sending API interface has a risk to break the existing usage of users. So we have to support it. The second one is about enhancing the health management in sending. Actually we need more details. We need more detail discussion here. We are still not very clear about how to implement it. But this is an important thing we have to finish in new time cycle. And the third one is about container cluster. Actually this is a very hot topic in OpenStack community now. And we try to use sending to manager container as the first class citizen. And we are trying to make support for container cluster. And actually there is another session on Wednesday. And we will make some detail discussion there. And everybody interested on this topic are welcome to join it. And the fourth work item is about testing. Actually we have achieved a good coverage for unit and functional test after medical cycle. And we hope to make our test cover more cases. So we plan to add the test case for both API scenario and performance, you know, benchmarking. This is very helpful to make sending more stable and more scalable. Some other work items include additional cluster operations which could provide more flexible way and useful command for any user. And also a metric queue type of receiver which we have introduced that we now support WebHook. And also notifications to other services which can help any user or other OpenStack services who want to talk with sending help them to track the detail workflow inside sending engine. And also batched operations and user-defined actions which is useful for application level deployment I think. And also access control, you know, to control the access to different resources belong to different user or tenants. Okay, after those introduction to sending project I think maybe the first question you guys want to ask is what is the relationship between sending and other OpenStack services? So as we introduced that sending is mainly about managing the collection of multiple homogeneous resources which created by other OpenStack services like a bunch of Nova servers, send volumes, or even neutral networks. So it stays upon other OpenStack services. Actually this is similar to HIT. The difference is HIT is more about the combination of different resource types. So the stack conception HIT is more like the struct data type in C programming language. And by comparison, sending is more like the list or rate data type in C programming language. So as we can create a struct of lists as well as a list of struct, user can also create a cluster which consists of multiple HIT stacks and also create HIT stack that contains one or more sending cluster resources. So this is the relationship between sending and other OpenStack projects. Okay, this is the last important information I want to tell you guys that we are actually here hiring, sorry, hiring. And actually sending now has an open ecosystem going forward and we have a computer from various companies and organizations and we talked with other OpenStack services, OpenStack projects and also other open source communities and standard groups. But we do have a lot of things to complete in your time cycle. So we need help for reviews, patches and blueprints and bug reports. So very appreciate any of you guys who have interested in this project can join us and we can work together to make sending better, to make OpenStack community better. So that's all. Thank you, you guys. Any questions? So we still have seven or eight minutes. Okay, it could be. It could be, yes. Health management, just my personal opinion, is a very important feature we should deliver in sending because without health management we cannot do auto-skating, those kind of things reliably. But detect note failure, that's a very, very complicated field. Failures can come from hosts, the physical note, the VM, network stories, whatever. So I believe health failure detection from the application level is out of sending scope. User long-term monitoring is out of sending scope but we do provide hooks. So if you can detect a failure, you tell us, we will try to recover for you. The question is about handling compute note failure. I think there are already some discussions from the OpenStack dev list if you watch that list. It is, yes, again, not very easy a job. What we can detect today is Nova Compute Service Don. If we can get that, basically your VMs running on that physical note are out of control. So we want to do fencing, stories, compute, and network fencing followed by some recovery actions you can specify. If it is not that critical failure, maybe just reboot the VMs, maybe rebuild it, maybe evacuate, migrate. We want to expose all those options to users. They know better how to handle those failure scenarios. Thanks. We have two mics, or you can speak louder. The question is about auto-scaling, but the use case is about auto-scaling. This page is VM failure. VM failed, and another one is about running out of resources. That's a scaling failure. Are there any requirements on the resources that Sendlin manages? Do they have to be idempotent, or I don't even know how that's pronounced, idempotent? Do they have to be stateless, or is there anything in Sendlin to help them coordinate as a group? We don't have any assumptions about the resources or the rest of the APIs. Just tell me how to create a resource, how to update it, how to delete it. That's the only assumption. I've got a couple of questions about the relationships with other projects, so you've already talked about heat, but what about heat once convergence comes along, which might then be able to tackle the HA case and maybe some of the deficiencies she worried about auto-scaling, and also the relationship with Tacker, which I know is coming from really the telecoms world, but is capable of, again, auto-scaling and auto-healing. The first question is about heat convergence. Although I myself is with a heat core team, my personal understanding about convergence is to make heat operations predictable. Heat really gets a good understanding of the situation about the resources, their status is orthogonal to HA. HA is a completely different scenario. That's my understanding. The second question is about Tacker interaction. We met the Tacker team back in San Francisco and also Tokyo, and the consensus we have with the Tacker team is they won't duplicate auto-scaling in Tacker project. We will spend some time sitting together to figure out how to make this interaction smooth instead of having Tacker again reinventing auto-scaling. Thanks. What about the Sendlin API and Engine? Are they highly available? Is there any special consideration in how to make them highly available so that if something happens to one of those when they're in the middle of doing some clustering operation that it would still recover? Okay, currently we have two basic primitives related to high availability. One is cluster check, the other one is cluster recover. All the other policy kind of decisions, we have not yet considered that. It's too complicated. So that's the only two APIs we are providing today. But if you want to do something automatically, we do have in our plan to implement kind of a health policy. In that policy, you can specify in this scenario you want to rebuild your instance. In that scenario, migration may be a better choice. Sorry, I was actually asking about Sendlin itself, making Sendlin highly available so that you can rely on Sendlin to do the workflows that you've configured it to do. I think you can apply whatever HAA practices you have for the control plan today on OpenStack Services. We do have some ideas, some proposals about distributed log management, all those kind of things, but not a very high priority today. Just returning to the earlier question. How can you do autoscaling and also punt on the monitoring of the applications? I don't see how those two decisions are compatible. Autoscaling... If you don't know the application is running, then you can't say for sure that you have enough copies of it running. Yes, today when a cluster is created by Sendlin, we inject some metadata into the cluster you created. If it's Nova Server, for example, we inject the cluster ID directed into that Nova Server metadata. When you are monitoring your VM instances using whatever monitoring software, you know what those nodes come from. You know the instances available is there and running, but you don't know the application inside it is running. Sorry, I don't think there will be a generic solution that allows you to monitor whatever application. It's very, very... So why does your project exist? Sorry? So why does your project exist if you're not trying to solve that problem? Knowing that the VM is up is not useful. That doesn't tell you anything about how many copies of the service you have. I totally agree with you, but it's out of Sendlin scope. You may want to try a thermometer or a monoskeleton because monitoring itself is a huge beast. You can use that based on whatever monitoring software, but we don't want to do monitoring by ourselves. But could Sendlin have an abstract interface to an application monitor? So for instance, if I can provide monitoring of my application in such a way that it provides a useful characterization for Sendlin, could Sendlin go ahead and use that? So you don't have to reinvent the monitor. I can give that... Yes, we tried that. I think maybe four or five months ago, if you were watching Sendlin service, we have an abstraction called alarm. By alarm, it can be a monoskeleton, a cylindrical alarm. We have some very preliminary implementations supporting those alarms. Then we realized, no, we cannot do that. Cylometer has maybe ten types of alarms today and it's going to A. And monoskeleton, I believe there are many more types. It makes no sense today to force people to use Sendlin Alarm to create that synometer type. Why are you doing that? You should use the synometer directly. But what we do provide is the receiver side. We provide a webhook URI. You can hook that URI, that webhook, to whatever monitoring software you prefer. That's the... Maybe not a good decision, but that's the current situation. Okay, last question. How do you scale? Because you're doing polling of the various instances. How many instances have you tried polling in that manner? We don't have a large machine in our development environment. So if you notice that we have fake drivers at the driver layer. We isolate Sendlin out of OpenStack talking to something dummy. That API, for example, if you want to create a nova server, that driver will just return, okay, I have created one. So using that kind of isolation, we are doing a functional test. We can scale to 10,000 nodes very quickly. If there are scalability issues, we believe it's not Sendlin's fault. Well, you are creating quite a fair bit of API traffic. Yes. That's one of the features we want to provide in Newton. It's highlighted there as batch operation. For example, you really don't want to issue 10,000 VM create request to Nova API. That's a DOS attack. That's something we want to control. Thanks. I don't think we have more time for more questions here, but you can come offline. Thank you. Thank you, everyone.