 Hello everyone. I am Abhishek Kenan to give you a brief introduction about myself. I am deeply interested in last year. They are crossing the culture and mathematics. Currently I work as an engineer at the Air Force. So if you haven't heard of Air Force, then listen to Intelligent Customer Support Dashboard. We listen on all of social media and all of this data keep coming into our screen. And we automatically classify them in an actionable way. So like you see, it is the humongous amount of data that is coming in. So when you have a lot of data coming in, the amount of things that you do with the data also starts increasing. With such a system, you just know how and what size fits all frameworks. So most of the frameworks that you have are basically two geniuses. So you end up your infrastructure just keeps getting more complex and more complex. So there is no just getting around the way. You are building a studio system. So I can imagine today in this cloud computing age, when you start building an application, it can be as simple as a way to start, but when you scale it, it is out of nowhere. It is a distributed system. So this is the title of the paper by Mark Kovachov, a giant. I saw it in one of these parks and they just liked it so much so that I just threw it out there. So when you have so much, when you have such a complex infrastructure, there are just two paramount things that we care about. One is high availability and of course, all tolerance and then maximizing your cluster utilization. So what I am going to talk about is going to be centered around making sure how to build a cluster which is highly available, completely plastic and is for all of you. So, and yeah, of course, how we are going to do this is with, I may say, the dollar. So what is high availability? So you have all these, as neighbors, as self-explanatory as the commerce, high availability is just making sure all your services are running. Even if, for example, let's say you have 10 instances of your apps over running, even if one goes down, you may, it is sort of all the records that are coming in are slow-balanced across the remaining nine servers. So, I mean, it is like making sure that even if, of course, in a distributed system, you cannot always be sure that a server is never going to go down. So servers are going to go down, right? So even if this happens, we are going to be highly available. Next, next one is, I also, like I was saying, when there are just so many frameworks, you, for example, you have so many frameworks, right? So when you deploy so many frames, what you usually end up doing is, you have this cluster and you start dedicating machines to this framework separately. So what ends up, and also this is also the capacity line, right? You don't, it's really, really hard to figure out, even an N number of different services which are running, how do I figure out what is the most optimal configurations of the service that I need to deploy on my cluster. So, this is allocation is hard. And even a completely elastic environment, it's even harder. And all of these frameworks, for example, let's take Hadoop or Stompy, each one of these frameworks, they're super rich in terms of what they want. So for example, if you want to run Hadoop and MPI on the same cluster with whatever is happening, like, no, I don't know how you do that. Imagine, right, you have this same data set and you want to run two different jobs. One is like a simple analytics job which you can kick off as a map-to-use job and another one is some, I gave you machine learning thing that is happening, but they want to operate on the same, they're clear as that. So in such a scenario, what most people end up doing is you auction the cluster of goods and you allocate one part of the cluster to you and the other part of the cluster to MPI to, you know, here is your data, you just take care of whatever you want to do. But that's not really efficient because you're equating your data on your cluster on the cluster for all of these new frameworks that you need. So how about, I mean, doing this is hard. How about a resource allocator? Imagine your computer, you have like one. For example, this computer has a little bit of RAM and an iPhone processor. So you just, you don't, when you write an application, you don't really, you know, your operating system just takes care of you. It needs one of these jobs for you. So the whole idea is, Apache NISOs, like the name suggests, is one of the top Apache projects. It was, it came out of the search paper published by Benjamin at UC Berkeley. So later on when he joined Twitter, he got through the project. And I think Twitter and like most of their infrastructure is being powered by NISOs. There are a lot of other companies like Airbnb, Open Cable, Capacities, which are using, which use NISOs at the core of their infrastructure. So I mean, what's the deal with MacMeso? Obviously this is like the textbook definition of what NISOs are. Apache NISOs is a cluster manager that provides efficient resource isolation, sharing across the security applications. Yada, yada, yada. I think in the most simple stuff, in terms of NISOs, let's you treat a cluster or a data standard as one single computing entity. You guys, how many of you have heard of this term? Where those can compute it? Or probably good old days of Ondal. Ondal is just so old that it's specifically educating textbooks. So if your whole idea is, for example, let's say you have a cluster with 10 cores, each with 4 megabyte of RAM, and 4 cores of CPUs in each of them, what you actually look at it as a huge machine with 40 megabyte of RAM, and 40 cores, right? So that is the whole idea behind NISOs. NISOs lets you treat all these different, and of course all these are commodity-made servers, right? You don't say you're going equal to the architecture. So NISOs runs in a mass when it's there, when it's there configuration. NISOs masters answer in a higher way than we do. Like in all these systems, NISOs, it's NISOs master answer in a soft state. So at any given point in time, the state is maintained. The state of the master is maintained. That's an applicator. The law on this is to keep a forum. So what happens is, even if one of these masses go down, when the other master comes in, it can use those applicator logs and come back up as masters, right? And you have these NISOs layers which are installed across all the nodes in your data center. So yeah, of course like ZooKeeper is going to take care of your leader election, he's going to take care of your leader election, and it's going to take care of telling your slaves that hey, the master has gone down and this is a new master. So let's look at this, right? So we have these two famous Hadoop and MPI, which wants to make use of the resources available in a holistic manner. So what it's going to do is, the Hadoop scheduler with any of these, these are all famous. So in NISOs you have the NISOs which is the way, imagine it has a substrate layer and on top of that you have these frameworks. And these frameworks have two components, a scheduler and an executor. So we'll talk more, we'll get more different what a scheduler and an executor does. But here, so when the Hadoop scheduler is going to do, so these are two different frameworks, right? So they want out where the source is in a cluster. So what it is going to do is framework A is going to say, I want two CPUs with four gigabytes of RAM each and I probably want to run it on, for instances, as if these combinations might quite, quite two or three or whatever numbers you want. So what happens is, your framework is going to, so through this scheduler, the framework is going to ask the NISOs master, what is the amount? Hey, this is my job requirement. Is there anywhere on the cluster I can execute this job? So NISOs master is going to do its computation on, obviously the scheduling are based on some fat policies, you can also define your own scheduling policies. And also, for example, let's say this particular job, if you want that to be run on a GPU, you can also specify and write this, these are the NISOs servers I want to run this job on. Hey, don't give me it. So it is like these are about filters. So we get into filters. When NISOs master figures out that it has actually got something that it can actually set back to the framework, so the framework is going to receive the offer, it's called a resource offer. So with the resource offer, what it is going to do is, it is going to see, for example, let's say Hadoop works with the whole idea behind the Hadoop is acquiring data locality, right? So if, when I run this job, if the resource that is allocated to me is not going to let me, I think data locality, so the framework is going to just say, hey, I cannot accept this offer. Tell me when something I want to survive on. So the frameworks can reject offers. This is the beauty of this framework. So in most of the scheduling frameworks, what usually happens is, it's like most of the scheduling frameworks are once it's been solved, right? They try to start away this whole concept of, hey, these are what these days want. Let me see how much I can offer. If you want to see this, what a way he wants to do it. So this completely decentralizes the process by letting frameworks decide whether it actually wants to run this job on this, run this job on this particular mission or not. Is there any more, do you guys have any questions on this or can you go on with that? So building a framework on these those, so you, you have that right now, we saw that each framework that needs to be run on mesos has two components. One is a framework scheduler, one is a framework executor. So from the earliest slide we saw, we wanted to be able to, we wanted each scheduler or the framework to be able to talk to the master and in turn also talk to the executor, right? So we need to, these are components that needs to be, that needs to be implemented by the framework, whatever framework you're not, the framework needs to support it. So, there are two components, right? It's a scheduler, my master. So you'll probably have to implement something like this for everything that you want to run or better with the framework. By the way, is it visible? What it will show from this though is, if you have to implement anything, if you want to get anything to be running on mesos, you have to go through this whole ordeal of actually implementing these. For example, what to do when the scheduler has actually registered with the master and what to do when it is re-digest trained, what to do when the resource offer has been made and so on and so forth. As in, you have to go through this whole orduous process of mailing down what exactly, how exactly the framework should behave. So, right? This is, obviously, this is, this is really a pain. It's a pain in the neck. When you want to run, when you're having so much, so many things that you want to run, it just doesn't work. I mean, nobody has got time to implement these things. Exactly because for each one of these frameworks, right? So, this is where Matathon comes in. So, Matathon is developed by ESOS. You can use Matathon to run long running jobs. It's just like a meta framework. It abstracts the whole idea of creating executors and schedulers for each one of these frameworks. So, imagine if Mesos is like an operating system on top of your cluster, or Matathon is like upstart or an edit. So, in a Mesos cluster, the first job you run is Matathon. So, this is how Matathon looks like. We already have this people forum and then there is Mesos cluster and then there is Matathon job. Matathon can also run in a higher way. So, it basically uses people to do its leader election, stuff like that. So, for example, all these little boxes that you see now are jobs that are actually services which is started by Matathon. So, you have, like, what that is all about. And all of these things are installed. So, what Matathon lets you do is you just, all you have to do is you just upload a chart or a target of all your services encapsulated together, like all your application dependencies. So, all these things together into a target. Matathon basically lets you execute any bash. It can run anything that is executable by bash. So, you have elastic search. So, let's, this is probably how when you make a request, you submit a job to Mesos for your Matathon. So, you're probably going to see in the ID of the app, and give a command as the command that you need to execute on each one of these. And remember, you see if you recommend some, in a number of instances, you want that you are able to pull your tower ball from your end line with wavy moves. And the number of ports you want to use and the unique constraints. Or you can also set constraints in the sense, for example, I want the close names of each one of these to be unique. So, when I launch, so imagine like these are all, like we saw earlier, we can, each state can also have two services running on them. So, it might also happen that two services, when there is no control, it might also happen that there is an elastic search, that is an elastic search running there. If I set my host name as a unique constraint, it's going to make sure that elastic, at any Mesos there is going to have only one instance of elastic search running. And the best part about this is, for example, let's say, if I have said that I only want, like, I want three instances of elastic search running. If this instance goes down, Marathon can automatically figure out that this has gone down and automatically relocate the towers on some other node that's available. That's high availability, right? So, this label, let's imagine this label goes down. So, when this label goes down, Marathon is going to find out that this label has gone down on the number of, you have said that you want three instances of this particular services service to be running, and now the count has gone down. It is automatically, so it can figure out that it has gone down and it will automatically launch this service on any of the, if any of the other places in the customer, like this Mesos matches, like this Mesos matches. So, I mean, Marathon is completely restful. You can all, whatever services you want to run via Marathon, you just submit it via your SAPI, or that is also a web UI which you can use. For example, you can just, for example, you want to increase the number of instances of a particular service from 5 to 10, you can just go into the dashboard and change the number. I mean, there is just one huge drawback, right? What service do you want to run? You have to supply a power ball. I mean, seriously, who does that? Imagine deploying, okay, again, this is an example. Like, it just has so many dependencies, and since you want to make sure that it is completely, it's highly available, what you actually end up doing is making sure that all the system dependencies are installed across the nodes so that this job becomes easier. I mean, this is such a pain. Nobody wants to do this. So, I mean, how can we do this better? So imagine, so how many of you know what Docker is? Okay, anyway, Docker is a container management system. It lets you create file system images and launch, launch an LX container from them. Docker, at its best, is a container management system, and it's a better image management system. So, like, this is exactly similar to AMIs, right? I guess most of you would be using AMIs. For every service you have a container, you have your AMI container, and whenever you want to launch it, you just launch it from AMIs. Docker images are just like that, but there's literally a time spectrum. So, this, that comes to you at first. So, in AOS first, there's a copy-on-write file system. So, what AOS first lets you do is you start with a base 3.1D layer on top of that you can put as many layers of file system as you want. So, in reality, so each one of these files, each one of these files is a layer you have on top of the base layer that can have its own directory structure. And also, all these layers can be unified together and viewed as one single file system. So, the best part about AOS is, AOS enables, imagine, right, so when you're creating AMIs, what you usually end up doing is it's you have to again create an AMI for whatever virtual machine you are running it on. Like, it's not... So, with AOS, what Docker lets you do is you start with the base. You start with the base image on top of the base image. For example, whenever you add a service, you add a service, not a service, you add a dependency or whatever it means. So, on top of the financial system, you can make a commit. And only the before the previous state on the file system and the current state is taken and it's pushed out. So, each one of the changes that you make do your file system. When you deploy it, it's only the lift that travels. So, your containers makes this whole managing application and system dependencies a piece. So, now, let's see how Mesos and Docker... I mean, I spoke about just so many different technologies, right? Now, let's see how all these things fit together in the equation. So, what usually... what happens is you suck it and jump by the middle cut. So, I will talk about this. Mesos allows... when Mesos launches its tasks, Mesos launches its tasks in each one of the state by an executor. So, when the executor launches a task, it can provide... it also does containerization and isolation. It does resource isolation. So, how it does is it again uses the... it again uses NSEs and... and C-groups. That's how it provides isolation. So, in the recent version of Mesos, you can... what it lets you actually do is it lets you do external containerization. So, external containerization is... some... is like, for example, hey... or like how we were writing our famous... famous rectangular, right? You can just write a plug-in to take care of containerization for you. So, this time... so, and Marathon... Marathon has another project called Emos, which is... which is this external containerization plug-in. So, it sort of fits really, really well in the Marathon ecosystem. So, like, you just make a list when you're... when you're just... I'm kind of writing instead of sending the command. It's your percentage to send the image. The image you are... and you have your Docker registry and the image you are from the Docker registry and the options you run this command. So, you do a Docker run... and this is basically the... all positional and keyword-on in the second part of the time. So, the execution is... bad... bad execution... that's inside... inside your ASUS name is... property, say, somewhere and say... like, say, bad executor... property. And then you just... So, once you start the job, it's going to go through the same process of asking... talking to the ASUS and figuring out whether there is some place on the cluster where it can run this service. So, it's going to get back to Marathon and Marathon is going to say, hey, okay, yeah, it matches with what I want and it's going to launch... tell it to the slave. And the executor is going to... So, this executor is going to launch... tell the Docker... the container running inside the slave to launch this command. So, now that we have passed all of our app dependency as a Docker image, so this Docker name can automatically pull it from the Docker registry and it's just not running it. So, it's... it's... it could be a lot faster when... when this executor has already done this... when this slave has already done this service because the cap... the image cache would already be present. Otherwise, it will have to pull up the whole image but then again, the Docker image is the white. So, in any piece of master... you have like pieces of master running and that is Marathon or inside the piece of slave you have this and then you have the Docker... the Docker teamer is going to take care of launching this way just going to basically take care of launching and destroying the LXEs. So, when the pieces... when the pieces... you see that, hey, I want you to run this job it's going to have the executor just to run the Docker and command with the options options that we can run. So, putting everything together so, we... we help mesos to take care of this is allocation in higher availability actually it's both mesos and Marathon which takes care of higher availability but Marathon apps... but Marathon helps a lot more when you... when you actually want to... run a lot more services than what is actually possible possible with all these frameworks that you can write because the other one is the Docker you manage all your application dependencies so it doesn't sort of paint the picture on how you can get the... get a high-lasting cluster so, yeah, I'm hoping for questions you can do your questions so, do the questions and I'll do the whole point-bait with mesos so, with mesos you... you have these... all these mesos running on all across your cluster right here mesos... your mesos master is something that is external and it is going to take care of submitting the jobs to mesos like this absolutely awesome I mean, that's your whole point-bait in mesos Does it make sense for me to deploy using this infrastructure or would there be too much... If that is... if that is the only job you want to run then probably it does not make sense but you... but if for this job you're... like, to run this particular job you're setting up... you're allocating resources for it separately for example, you have a couple of machines just to do this and this job is probably not... like, it does not happen all the time then it makes sense to use mesos because when that mode is not working on this job it can... you can probably run one more APS over there but how much of an overhead does it add? time... time... I think the overhead is... the overhead you have is... maintaining the zuki perform right and maintaining the mesos master because the... this probably does not make sense when you have a really small deployment where you can like... things are not constantly changing or you don't want something like this truly elastic then that's absolutely overhead but when you cross probably a couple of hundreds of nodes and you want to squeeze... squeeze the juice out of all the... all the machines that you're having in your cluster then it makes a lot of sense and how does it monitor that you know this particular... the green resource has gone down it's not responsible I will figure it out if it's not working the job won't get down there I did not get it are you asking how does it... your question is how does it figure out the slave has gone down so you have zookeeper so it is going to keep talking to... zookeeper is basically a distributor coordination system it is going to keep coordinating between the master and the slave so when the slave goes down it's going to miss a lot of people it's... it's a homegrown wire protocol it's based on the actor model so behind the scene it uses some... limprost a scene like limprost it allows you to... program the actor model the messages that you send between the messages you send between the masters and slaves these are all complicated but... but it works it does not use rapid and pure it's a wire protocol the wire protocol is just... it's free but it... managed within an actor model