 Thank you for joining us So this is me. This is the lip and a beat. We are from Huawei Bangalore team Who are we bangalore platform as a service team? So we have Lots of areas we have platform as a service distributed across the globe. We have lab in US. We have lab in Bangalore We have a lab in Homeshoe, we have a lab in Shenzhen So we primarily work on Kubernetes. That is our base engine for provisioning our performance as a service We use we also try and do a lot of work on missiles and Couple of our Developers in our team. We evangelize missiles internally. Hopefully that this also be adopted We think that this also is very useful for Huawei It has got a good potential to solve a lot of problems that are currently there in the past domain so let me do you see I mean so he's going to give a Introduction about never talk. He's going to talk about the agenda is going to talk about similar projects He's going to talk about why this project is first of all necessary What we have over in agenda is My abstraction was there for a framework developer For in-app structure and what all benefits a framework developer is going to be get when we introduce abstraction with some tools or Some some library which we provide now Again, we will also scan through some different community which has similar kind of Abstraction provided to help you help a framework developer or a user to Deploy a workload now after that they will run through Through your missus voice stateful what all things are there but what all designs are there in this missus voice stateful design pattern so pattern is something about which is Frequently reacting problem inside your environment. So what we do is we design a solution for that and Well designing a solution Keep something in mind that the if the same problem In news near future, then the same solution we are going to put on and We are we are going to work out with that now Over so what we have a very is because the solution is already well tested and also it is a Industry proven thing so we can easily use it up Within the system seamlessly so for an example what we have is Whenever we have to talk about log management, so we simply go and we'll use the single term pattern if you see So what's there? Is there for a framework developer? Inside a design pattern so if we in a framework development board so For if you have to develop a different framework There are some tasks which each and every framework has to handle which is common across So what we are going what we are going to do is we are providing all those tasks and whatever So problem is there. We are going to Provide a better solution In the by providing a library on top of the language binding which we have So how we are going to do it is why I'm providing an abstraction on the language binding which we have so when we talk about abstraction so it is all about Solving a complex solution and providing a simple interface to a user so that he can easily use it to develop or develop a framework or to help him out to a Help him out to work it. Okay. Now for an example what we have is let's say When we are going to sell a car so in the user manual, we do not give How it's internal combustion is engine is going to work So what we give in the user manual is ABC of car that is your accelerator brake and clutch So what we really proposing over it is a design pattern for writing a framework of a stateful workload Along with the affection provided on the top of the missus When we started this project, so we did a little exercise So to check whether whatever Idea which we have in our mind is really a crazy idea Or it is going to put some work in the community. So we scan and out Scan around and we have seen a lot of communities So we what we found out is there are all other companies are also progress progressing towards similar approach where they provide as a layer of abstraction on The basic primitive which they have to help the user to deploy application or to develop Framework something like that. So we have Incubinators we have chance and hands. So So what it does it is a kind of package management So whenever someone has to deploy application. So what he do is he He just define his workload in a form of a specification, which is your charge and what we have in hands is Two component one is client and second one is Tyler. So Tyler is there in the Kubernetes deployed so with the help of client we are going to To the talent to go and deploy a particular Chart which has already defined with a proper specification. So what we see over here is Covenants has provided a layer of abstraction. So he's out the deployment process over here. So Next is our Covenants pet set again So again covenants pet set is this has been designed basically for a stateful applications so what it gives is a high-level of identity management, so which involves your Ordinary which again Provides the stable host DNS host name and storage linking. So what it has is whenever you deploy Application so if that applications Get down so the same same that application will get the same Identity when it comes up again. So that's what we have in the community. That's it So so this co-operator is something new which has been introduced in KTIS community So so what they did over here is they bring in the operator Functionality over here. So operator are the basic guy who has a good experience in a Application deployment. So the similar concept they are bringing and so They have come up without Some operator a subcontroller called operator controller. What it does is it? it go and get deployed inside a Covenanted system and it defines a task and that that particular task is What we're going to do is it will extend the existing api and Existing primitives and api to ease out the further deployment of a stateful applications. So This is all about go operator. Okay, so yeah, so now the for Developing a operator for a stateful application Okay, Oro has given some set of rules, which we have over here is is Beheading as a scheduler and we'll get deployed in the Covenanted system now again Operator will create a Time which is a third-party type with which is similar as a task in our MISO system so so that we're going to handle all the operation level primitives for a Workload to get deployed now. It also adds some some level of Version management information also and then it and what it does it decouples the operator life-cycle with the actual workload life-cycle So now yeah, we have DCS common So we have already everyone knows about this thing already. So we have a Java comfortable Common library already present in this year's common. So maybe okay now we have a Spring cloud. So again, this has been written in Java and miss mainly this This project but closely to with the cloud foundry and provided a lot of We have this some of over here, but it provides a lot of library So to for a user to deploy application even to another security Even to configure a workload inside the cluster also. So, okay, so so before Developing the project which we have is miss Missus voice state food. So what we did is a elementary Analysis which we have done. So if someone has to light a framework inside So which all common primitive and common problem which he is going to face For for developing a framework. So what we did is we have taken up the common stateful workload Which is a Kafka, etcd, Postgres, Redis, all these things and We have done some initial analysis and then we come up with the initial level of problem set Which we are planning to we have sort it over here. So now So to some of what Amit was saying is that These frameworks these type of abstractions or libraries that have been provided in the community Actually help speed up the company projects in the community. So it helps your project adoption So a new technology if it has got lots of libraries, then it's easy for me to Develop and use it in my company So, I mean if you see the available frameworks in the Missus ecosystem, you have like frameworks from Apple you have frameworks from Twitter so these frameworks are like Class apart. They are like written by best people best brain So it shouldn't be it shouldn't be a rule under the rule that if you have to write a successful framework You should be a unicorn know a common developer should be able to understand the concept and then pick up the Concepts and then you should start writing a framework. So it should be that easy. So that is our whole argument so When we started writing what we did a alpha level project for Redis. So we call it Project is called Mr. Redis So when we started out writing that framework so we had to deal with a lot of problems So I mean it's very intuitive when you see the example of Missus go So you understand how the callbacks work how the whole scheme of things work and you try to do some hands-on with it But when you actually try to write a framework, it needs a lot of things for instance It needs to be an STDP server So it needs to support the basic credit operation. Otherwise your framework is of no use And then you should actually deal with the offers effectively like it should be a good citizen with the Missus ecosystem You shouldn't be hoarding offers. I mean most often they're not if you have a complicated Stateful, I mean a stateful application is usually complicated But if you have a complicated workload, then you most often have to write a custom executor too and then the framework is going to deploy a lot of App instances so each app instance itself could have multiple tasks underneath So that could be you could think of installing a MySQL instance But MySQL instance could have like a lot of processors like masters slave etc. I think as a part of that instance So you need to remember Missus will not remember all these things But you need to remember as a framework remember that what group of tasks belongs to this instance So that's another overhead for a framework developer. And then you should also be You should also think of logics to distribute your work load Say if you are thinking about developing a framework for stateful applications Then there's no point in deploying all your tasks in the same Missus agent right because you have a master and slave on the same agent if the Missus agent goes down then Your failover or you know your failover has gone actually so you should not deploy certain tasks Certain tasks should not be co-located. So those type of rules should be allowed And then you may also need a little more degree of control in Docker So even though we have unified containers, Docker is still much mature technology So you may want to interact with Docker mode. So you may want your custom executed to directly start pulling images from our talking to the Docker team. So if you see that this is what we are proposing So this layer could be the layer of abstraction which would benefit End user to start writing frameworks much easily So you would have the core resource manager and you will have lots of language mindings And we have an abstraction layer in between and then we'll have service frameworks on top of that So this is the experimental project you've started like a month ago So you've taken a lot of what we've already did in the Missus framework and then we've added new stuff to generalize it So this is the overall design So the framework is going to have an HTTP server So which would be abstracted out using HTTP lip. So every single component is abstracted out So everything is HTTP. Tomorrow you can change that HTTP lip to a better HTTP package in Go That's absolutely possible. So there is a buffer management So it is very important for you to actually come out of the callback functions as soon as possible So if Missus gives you a task update like task running Then if you're going to do a lot of complicated work in the callback, then it is going to stop the other events coming into your framework So it's we think that it's not an ideal framework development method So you should basically push all the events to a separate queue and then you should differ processing those events separately So that we have some buffer management and what if you get a lot of spike in your framework like You get thousands of instance to be created at once again So I mean we think in terms of a public cloud provider. So you have a framework and then you get like You there could be a event or some festival and lots of people want to create Redis instance or MySQL instance at once So you also need to streamline or buffer the incoming queue So there is another buffer that requires that should select the offers And then we have a state management. So state management is the currently we differ the storage to some key value store We support ETCD and zookeeper currently so We have written an interface a very simple interface that interface functions are actually used throughout the code So that later if you want to add new storage mechanisms, you basically have to implement that interface So it's pretty straightforward. So we actually had somebody from outside For zookeeper, but we originally support only ETCD. So it's basically one go file You have to probably implement to start using a different storage mechanism And in the executor you get a basic executor and then you also get a Docker library Which can talk to the Docker demon and download the images control Run it delete it and then collect statistics on the Docker image without having to run Docker command Or you know execute it. So that's another simple wrapper over Docker client. We have written So for the HTTP library we use a ego framework. So tomorrow we can actually change it tomorrow to another framework If we find it much better. So without changing any of the your code will not be affected at all by changing any of these Meets. So the missiles go is also spending that I'm waiting for an upgrade To support this was 2.0 API's So once we get that whatever changes we make it will be a part of missiles live and Whatever frameworks you've written on top of the subtraction will not be affected. So your code change will not be happy I mean these coaches will not happen. That's the point. So in order to maintain the Flow so because we are not going to spend a lot of time on the callbacks itself We collect the events and differ the processing to a queue. So we have three different state engines So these are all basically go routines. So all the STDP request for creating a new workload will come to the creator So he will understand the terminology sent by the user He will know that what instance needs to be created and he will break down into different tasks So these tasks will be selected or matched with a particular offer from a source So it will go to the buffer management You probably push it in a queue and then the receive offer function call will be signaled automatically that there is a new New task is coming and then it will supply revive offers to the missiles master So if there is no Incoming workload coming then the the buffer man the offer manager will automatically Suspend getting offers to from the missiles master And then when the status updates, I mean you launch a task and then this was master says it is successfully running Then you get that instantly the task update pushes into the task queue and then the task queue is being watched by the Maintainer so any event that comes to the task you then made a signal by a channel Then he starts processing all the elements in the queue and then based on that he either Stores the data into the key value store or he informs the creator He sends another channel information to the creator to create more talks for instance if a if a slave dies in a stateful application There is a master and multiple slaves Scenario and a slave dies then it goes to the maintainer and the maintainer figures out that the slave is really started Then he simply sends a signal to the creator and similarly there is a destroyer destroyer too So destroyer would simply get all the stdb request and it'll send a framework message I'm sorry. It'll send a task skill message to the executor so that you can gracefully perform doctor kill or talk and destroy So if you have like lots of instances running on top of your framework and somebody starts Create trying to somebody tries to create a new instance Then you should you shouldn't be going and querying the database quite often So we have a small thing layer of cash We've also started writing it and then it's reasonably abstracted for now, but it's not like completely done So the case would I mean all the information will be there in the case and then all the dirty pages would be updated to the Key value store on a timely basis. So that's the whole idea. So Only one guy maintainer would be the person who is actually writing to the case I mean they're getting to the case and case is the only person who's writing to the database so that you know Streamline the way we access the database and then it's pretty faster And then we have also we are also thinking of implementing a reasonable protocol using Jason for Communication between the executor the custom executor and the framework. So this is simple design of the custom executor. So the customer I mean the custom executor will eventually store to the I mean come into the store, but not directly. It'll go via the abstraction and then that will be stored for example The custom executor can keep collecting statistics about your Docker container running Docker container So it will send it as a framework message to the my missiles go and then it'll automatically be updated to the store So there are two things we have we have tried. We are trying to generalize how you monitor a task from the custom executor So it should be pretty straightforward. How you start and how you monitor a task and then there was also a Docker live So these are the two main components of the custom executor So these are the few callbacks which we thought that you know Which we think that might be useful for a framework developer to start writing framework for several applications So what are the things you would need so before a task is being started? So he would probably want to generate a config file or probably wants to have a custom in argument For example in red is if you are starting a master, then you don't need much of command line arguments But if you're starting a slave then you should say which master the slave belongs to so you should probably add a slave of symbol And then the master's endpoint similarly some Stateful obligations require a configuration file to be generated on the flight. So all these things will be managed by the callback So basically if you're using the subtraction When the request comes from the STT bill and then before it is being matched with an offer this callback callback Of conflict will be invoked so that you can generate any conflict file or any such thing And then you can return it and then instantly it will be created as a task offer And then similarly you want to start a workload so it can be a general workload start and then you want you won't have some sort of Customization for starting a master some sort of customization for starting a slave Because when we looked at like few stateful applications or the variety we saw is that there is master and slave type of Stateful application. There is a peer-to-peer type of stateful applications basically like implementing graphed protocol So those peer-to-peer type of stateful applications also have a leader and follower type of concept So we thought let's stick with master and slave as a general theory which might be applicable for the peer leader and follower type of applications as well So in the master and slave In the master and slave Callbacks they have additional things like what what should you do when the master is running? So when you get a task update from the source then this callback will be moved So your master is now started. So what do you want to do now? You want to know now start all the slaves or you want to update a DNS all these things you can program it here And if your slave is running that is another thing you get another callback So we don't want to deal with different things like you know what happens? There is an executor lost what happened when there is a task error what happens when there is a task failure So we want to generalize everything because most of the time for all these events you are going to take same action probably restart The workload right so we want to put everything into the same thing so that you know you write less number of code So currently the development process because it's not like very old One month or you know forty-five days old So currently development process that we need to implement a lot of things in the Creator-Mainer and destroyer concept and then we need to implement we need to fix or you know They have some design thought process Happening in the custom executor area This one is pretty much done So what we need to do a little more work once the next mesos go library is launched it's released So I think we might get it end of this year And then we also need to do a lot of work on the state manager as I said We want only the caching system to actually update to the database on the deferred or dirty instance Instances we don't want to keep flushing the key store all the time So the cache should be the central place where all the Coroutines query for data and may there should be the only coroutine that writes to the cache so that we avoid waste conditions That's the whole idea Let me have a quick demo. Currently we have written a small utility called code gem. So that will help you bootstrap Creating a framework so it basically creates a skeleton of what is required for a framework So let's create a brand new framework and see what all the files are getting a framework called high And then it is created the scheduler.go executor.go and then it is automatically populated a config file So let's go and see how this project looks like as you can see Just being created Let's take a look at the config file So one other responsibility that I forgot to mention in the framework development is that you cannot handle everything in command line You definitely need to have a complicated config file and then process it and the requirement on the config file is going to keep growing So we have thought of generalizing it as much as possible. So because this abstraction relies on Custom Custom executor concept. So we already have an executor path. So the framework itself will host it And then that can be downloaded by the agents And then few other things like what's the database type? So tomorrow you add new key store type then like zookeeper You just have to mention it and then give the database end point and then if you want a lot of files And then the artifact IP where you want to host your artifacts like executor custom executor And then what is the host HTTP host? This framework should bind to in your local server and then the general description of this task So this specification workload specification So if you if you think that you can have a general workload specification for all the task running off for this framework You can mention it here or you can keep this as a default Specification of a particular task and then you can actually change it on the need to do basis when you do a create every single time So I can also show you the scheduler. I hope it's visible. So so it automatically generates the Structure Using your project name and then it generates all these stuff files for config all the callback stuff files And then it also creates a main file for you so that it's straight away compilable So you create a project and then you compile it. It is a basic skeleton framework that is running You don't even have to write one single line of code and it is going to have all these capabilities automatically like HTTP server buffer manager and a custom custom executor etc So there is also this Executor that's being created So it has got like basic tasks. So we have got only two callbacks in the executor one is you either config or you get a task started Task started callbacks, but you really want to improve this Capability so you could do more you could have more control over your custom executor So you could have a more sensible general callbacks for stateful applications So there is another project. I have already created. Let me show you The scheduler so this is this is the basic scheduler that got generated. So the name was the name of this framework is Test 2. So this is the default code generation generation that has happened. I've added only one line in the config So this is totally in a different path than the actual stateful library So every time when the task is going to be started, I'm going to simply Marshall the actual instance Or the structure I represent an instance inside and then I'm going to put it off as logs. That's all So let's try running this custom framework and Config file. I've just updated. So it's the basic configuration file that was generated. I just updated it with the master's end point data basic points, etc So I'm going to start the Scheduler so it connects to the master and then the moment it detects that there is nothing in the incoming job queue It is automatically going to process for declining offers for the next one hour So let me create a dummy task Yeah, so this is a simple curl command. I'm going to push to the framework So so far as a framework developer using this abstraction I've written only one line of code to unmartial my instance and then print it So Let me create this so it says request accepted And then as you can see the moment the task is being sent I think it's for you so automatically a revive offer is being called to the master because we Pushed something push some job in the queue and then we got an offer. We launched a task before we launched a task This call back conflict call back is being in mode So there we have added this one line to unmartial this instance and then print it out So it's been printed as a json and then we accepted this offer And then the task is running. So it's a very simple. It is without any command There is zero programming zero configuration we have done But we just auto generated this project and added one line. That's it So as you can see that no, you can see this the framework is registered and the task is running And then The framework id is automatically the framework name is automatically the project's id name that you gave And one more thing I want to point out is that So this actually helps helps us In a huge amount of code being written for a framework because these are all problems that is usually required to be thought It'll them you write a framework So this thought occurred to us when we were trying to brainstorm trying to run a different stateful application in our data center And then I was trying to argue that this also would be the best project or technology to run this Stateful application and then when I was thinking even deeper on that Then I thought that whatever sort of problems we had for building the red disc framework had to be redone for the stateful applications too Then I thought why not just take out all the common parts and then build a new project out of it So that's the whole history behind it So you can see that this one line Is actually doing a lot of work like you know registering to your master Passing your config file Creating an stdp server then hosting your executor So many things So as a developer I would be really it would be really convenient for me to start some start my project with something like this This is a very initial stage of the proposal a lot to go into this project to make it a little slightly more mature And like I mean I was saying This is not the first attempt to do actually generalize the existing concepts It is available in all the other projects. It's available in programming languages We have design patterns so many libraries implementing design patterns We have lots of efforts parallely happening in kubernetes community And we also have a new effort in mesos community, which is dcos commons So hopefully we might this might be useful for us too These are samples screenshot So the future work is that we want to also have a generic UI capabilities And then we want to first thing we want to do is we want to reimplement the redis framework on this abstraction and see how it goes So that we iron out a lot of issues And then we also want to build a regression shoot and as you saw that November 3rd or the first week of November we got the uh Kuro is operator launched for kubernetes So that has got lots of characteristics very similar to a framework in mesos community So they have mentioned that they need something like chaos monkey regression suit So we should also push for a probably general one And then we should test it with probably test it with different Stateful applications and have some basic examples so that user can start understanding it And more importantly support for persistence and volumes in the abstraction itself So as I said our other efforts in mesos is that we have built this redis framework And this is the new project and we are also taking part in the Federation work So we are attending the call so if you are interested in mesos federation, please join On the meeting that happens every fortnight Tuesday There is a federation channel like the keynote it was mentioned We also host we also host the mesos meetup in banglore We are planning to have a meetup 10th of December So I think that's all from my side. I'll be happy to have if you have happy to answer if you have any questions