 All here. So this is a very introductory talk to Meso's frameworks and different kinds of frameworks. So I would like to show our fans who already knows what Meso's frameworks are, who's already using Meso's. Okay, so I hope I don't disappoint you. So I'm a software engineer working at Meso's here on Meso's. And before that I worked working on distributed databases. And I have a physics background. So that's me. You can reach me under that email address. I don't use Twitter or something like that. So let's first talk about what kind of players we have at work in a Meso's stack and at the different layers basically. Since this is one of the great features of Meso's how it separates things into layers. So at the very bottom of a Meso's stack we have Meso's sitting abstracting away resources like physical machines with CPUs, memory, GPUs. On top of that we have frameworks talking to Meso's which in turn the frameworks then run tasks which are what we are ultimately interested in. Since the tasks perform the interesting work we care about. And all that is nicely separated into layers. So the tasks up here talk to the frameworks below them. The tasks don't talk to Meso's. And the frameworks talk to Meso's and basically communicate with the tasks and Meso's. So just to reiterate Meso's is concerned with tracking and scheduling the results that it gives out. And the frameworks then are concerned with scheduling actual interesting computation on top of these Meso's resources. So this is the job of frameworks then. So one example here of that works in praxis. This is how resources are allocated. And there's some maybe some of you already heard on the two level scheduling approach and this is in action. So we have some Meso's masternode sitting here talking to agent nodes which are the physical machines performing the work sitting below it. And these agent nodes run some executors which in turn are running the tasks. So what happens here is that agent node shows up and says hey master, see I have four CPUs, some memory, some disk for you to learn tasks on. The master takes that and in some allocation module then some allocation module decides that it should offer that to framework one. Because framework one is not running all of tasks at the moment. So framework one gets offered these resources that came from that agent. And then framework one can decide what it wants to do with these resources. And this framework one here for example it decides it wants to run. So the framework has its own scheduler and the scheduler in the framework decides it wants to run two jobs on these tasks. So the framework talks back to the agent and says run task one on these resources and task two on these other resources. And these other resources that were offered here, so different resources. And then the meso's master in turn can tell the agent to effectively execute the task. So the framework talks through the master and through the agent. So this gives us a lot of flexibility. This approach because it isolates the framework from the actual agent running the task. And then it really separates the responsibilities between meso's and framework. The framework don't need to know about machines and meso's just need to know very little about tasks running on these and how they should be run. So just to summarize this introduction. So the tasks they are really solving domain specific problems and that's what they care about. The framework in turn provides some environment where these tasks can run without needing to know too much about that distributed environment. Task status updates, tasks going down, all that can be abstracted away in the framework and meso's provides all the low level abstractions for the physical realities. So that each of these different layers can work with what is most capable of basically. So each of these eccles here can be specialized. These are physical machines and how to work with these. The framework to set up some environment that fits well to the goal it has and the task is really solving the problems. Because there are other tools meso's provides to run distributed applications and I just listed some examples here. For example there are roles which are related to how resources are allocated or how resources can be in a certain role. They would be offered to frameworks and other roles under certain roles or we can set up fairness between roles. For example the big data team does not need more resources than it should be from the web team. So a web service don't go down. There's also quotas to impose that even more so we have some guarantees for how much will be allocated if possible. We can make reservations for example to set certain machines with certain attributes aside for certain tasks. And we can create volumes with a little longer than the task. So if the task goes away the data is not really destroyed. So we can protect like drag or expose basically how machines are available and go down from scheduled maintenance. And of course like isolation which is some related to all the modernization talks you will hear about. And there are more. But these all sit at a very low level so meso's provides them to frameworks to program against. So meso's is some sort of distributed systems kernel in the end. And frameworks they are like user interfaces. Like you would program some Lipsy against system calls and you would implement your GTK calculator against some like. So let's talk a little bit about the frameworks. So what kind of workloads do people typically run on some large installations. So I just have some examples here but I think they kind of represent the big groups of workloads. So like at the very top there's like interactive data analysis for example with Spark or ETL applications where we crunch some where we reduce some data from into some smaller form. Or we might be interesting in running applications which can scale to like large extent to have for example also high availability requirements satisfied. Or we might be interested in running large interdependent distributed applications where many services talk to each other. That single machine is just not enough to do that and that drives us into some large cluster environment. The grouping I have made here is roughly going from more general to more specialized. So the tasks on the top will be more specific to some certain problem domain. For example big data applications here on the top and then at the bottom more general applications of running a large cluster. And these different workloads in the end lead to some certain grouping of frameworks that try to satisfy these different workload requirements that users bring. So I will mention some examples here. This is taken from the Mezos documentation and this is not a clear cut separation but some sort of grouping. So if you feel this grouping is not fair don't be mad at me because it's just a little fuzzy. So there's a big group of frameworks that try to address big data processing problems. A big group of frameworks that perform scheduling of batch jobs like for example ETL jobs which I mentioned on the last slide. And a group of frameworks which deals with running long running services which need to be kept available and need to be scaled and that kind of thing. For example running distributed highly available websites for example that could be addressed in this group here. So just to mention some examples. So this is the canonical list from the Mezos documentation and what you might know is for example running Hadoop jobs. So this is kind of very specific to the problem domain of running analyzing data in certain framework or running API jobs. So they are also specialized Spark and Storm. So there are all these tools here basically or these frameworks here basically are problem domain specific adapters to Mezos. So we can run our MPI application not just on machine with multiple cores but on a large distributed cluster. For batch scheduling some examples here. So batch scheduling means that we want to run some job that is triggered by some external stimulus basically. And in a Linux system one example that would be Cron which is where the stimulus is just some point in time. Or would and would also be some data pipeline where we process some data and as soon as some data is ready we run the next object. For example if you run a make-to-make with running stages and produce some product product in the end. And this is then driven by being some dependency being ready. And like for example Cronos here is a framework which kind of just maps directly on what Cron provides. Or Jenkins is a tool to build software so that kind of maps on my make example. Cook for example is a batch scheduler to run Spark jobs and can also do Cook actually does much more. But it also fits in the batch scheduling category. And frameworks for long running services and like I said these are concerned with safely running some distributed applications so we can scale with demand. And also make sure that our service that our task the service our task providers always available. And the big frameworks I will talk more about here are Aurora Apache Aurora and Marathon. Since Apache Aurora and Marathon are really high-level frameworks which can be used to manage arbitrary workloads on a cluster including other frameworks. So you might use for example Marathon to launch a Kubernetes framework or talk you could do that. And so these meta frameworks like we call them are so general that I would say that they are in effect like some shell for a distributed system. Not a GUI shell, not a terminal shell but some other shell for a distributed system. So I don't know how useful that is, how much you can see. So this is just some example how you would configure a task in Aurora on the left-hand side always keep that on the left-hand side or Marathon on the right and they don't do the exact same thing. So this is just to demonstrate some concepts. So what you see let's start with Marathon. So Marathon you see this is some JSON configuration which is very descriptive in the end. And here we have some different applications that are grouped into other groups and we can for example define dependencies on the stream of applications. And make sure that for example if the dependency, so this group here has a dependency database, this one. So the database goes down, this service here will be, we will realize that it's not, cannot work anymore. And we can expose that dependencies to Marathon to Marathon. We can for example restart this database group here. So this would then restart the MongoDB and the MySQL framework. MySQL task. So this is very on a very descriptive level I would say and Aurora uses a different approach. So this is basically some domain-specific language you see here on the left-hand side which mimics Python with some templating mix-ins but not in this example. Here in this example we start some Python file that we run here directly but it has a dependency on that file being there. So there's another process which copies a file into the environment where this later process is running and we can formulate in this domain-specific language that is sequential. So we depend on the ordering of these tasks and there are other ways tasks can be combined and one can nest that to some depth. And so this is a more, I would say, more programmatic way to describe it. So this is more descriptive on this side and this is more programmatic. And then we can of course start some job and assign them to some environments or clusters, roles that I mentioned earlier. We can of course name them so we can identify them later on. And these meta-frameworks can also provide, since they are effectively shells, they can provide new additional extracts which are not available and not exposed by MesoCent. It might be making more sense on the framework level. For example, these frameworks can provide deployments, how I roll out my applications or how I update them in a way that they always are highly available and satisfied constraints I have. Or we can integrate tasks into service discovery systems. We can maybe launch stateful applications. So like this Python example I talked about before was the stateless application. A web server would be a stateless application where the database cares about its state. And these meta-frameworks can add tools for that. We can also, since we have a two-level scheduling in MesoCent, we can introduce some new scheduling ideas that we have on top of that, like constrain what kind of resources the framework uses. Or we can, for example, pre-empt tasks if we run low on resources. And possibly more, but I'm just going to go into a couple of these here. So high availability is a pretty simple concept, maybe, but very interesting. And if a framework can solve that for us so the task doesn't need to know too much about it, then the framework can be very useful. So both Marathon and Aurora, they basically hook into health checks and readiness checks, which can tell if a task can still function and still function. And with that we can make sure that our workloads are always running and become highly available that way. And that can be used when we deploy an application. For example, to check if our deployment was successful or we can restart tasks, which become unhealthy. We can even change the configuration for, let's say we run a web server and we would like to update a configuration. We can update instances one by one and make sure that the tasks we start with the new configuration are still healthy and satisfactory. Or we can scale up applications, for example. If we see that tasks become unresponsive, we can scale up more instances to make sure that we cope with the demand. So Aurora and Marathon do this in different ways. So Aurora basically launches the process and then monitors what the status of that process in the system is. So we have some executor which launches some process and we can look for the PID of that process and see if it's still there. So that's what Aurora will always do. But we can also hook in some other process in that DSL. I showed earlier we can hook that in and run it in parallel with our task. And that special health check process just needs to respond to two or three different requests. So like health, which would say, okay, this task is still healthy. Or we could instruct it to quit soon. So soon we will be quitting. So we tell that we can basically trigger some cleanup inside the task. Or we can just abort it. And this is the protocol Aurora used for that. So Marathon on the other side is more general. I would say that it directly exposes the... So Mesos has some abstractions to write down health checks and how they work. And Marathon directly exposes that in the app definition where we can. So this is an application definition here in JSON format. It's called Rails app and it executes some Rails server from some Docker container, for example. And we can just define an extra field here, which is called health. And in this example here we will talk via TCP to some port. So port zero here would be some port that is assigned to the task via some offer. So some port gets offered. So this is some dynamic port under here. And there's also other ways to run health checks, for example, HTTP. So this is more like some HTTP. We could map that into this definition also. Or we could also run some tasks to see if our application that we define in such a way is still healthy. And another interesting tool that meta frameworks can provide is to hook into service discovery. Since these meta frameworks they run, they launch a task. They can, they of course know where that task is running and what ports it, for example, uses. And that is crucial so that applications in some distributed environment can find each other. For example, where is my database? Where is my web server? And both Aurora and Marathon can announce where applications are running to some external service discovery system. And this can also be integrated into frameworks specifically ACLs. Access control is to control who can actually discover some other services and find them and eventually talk to them. So Aurora uses a very, uses one approach which is we define in our, so this is some other app I have here. So this runs some tasks, this task runs a process which is an app. So this runs the HTTP server. So this is running HTTP server listening on some dynamic port here. So this is the Aurora way to write down port which has the alias HTTP here in this case. And down here in the job definition we just say one way to hook into a service discovery is just to define some default announcer object which can also be customized. And that would just create zookeeper nodes. So Aurora arrives to use a zookeeper and which we can query then from some external systems or from some external tools. So in this case it would, for this kind of application it would create some status. So this task is alive and it uses some endpoints. So there's some Aurora endpoint defined here as some HTTP endpoint. And they are the same. So they are aliased. And then we can basically ask zookeeper by querying this node here. Where can I find that service? And then we can talk to that server on this node, on this port. Marathon can hook into, on the other hand hooks into service discovery, on discovery info fields that Mesos uses actually internally which can be queried via Mesos. And so here we have the same app in marathon speak. So here you see like this is more like some programmatic way to build up some job. And here the marathon way which is more descriptive. So we just have some app that runs some command using some port. So this is the zero port counting from zero that was given to that task running on some resources. And we have some definitions for ports here. So we have our port zero, which is this one. Some protocol, I guess this should be, no, that would work. And some name. So we also give some alias to that port. So we use that HTTP. We could call it anything we want. And then marathon will publish that to Mesos. And we can ask the Mesos some, for example, the Mesos tasks endpoint. Hey, tell me about the ports, this application exposes. And we would find something like that in that. So this will give us an adjacent response. And so this should actually really add here. So this is this here. And then we can find our port. And also our host, the property. So now we know how to run tasks which don't crash too early. And we know how to find them. And now we can look at something slightly more complicated. So since Mesos offers resources to frameworks, frameworks can then schedule on their own. We have, of course, a way to, we have some freedom how to run tasks then. We might have some constraints on how tasks are placed. For example, we might want to run some job in a machine multiple times, multiple times per rack or n times in the cluster. For example, the rack to make sure that the task is running at the power field on one rack. Or multiple times per machine so we can share some state work between different tasks. We might also want to say that we want to, we don't want to run a certain job on a certain machine, for example, not run jobs on a machine that doesn't announce that it has SSD disk, which is fast. And both these frameworks, Aurora and Marathon, they provide some ways to express these scheduling constraints. So this is in the end constraints, what kind of offers the framework would accept to run tasks on. So what, so here we run again, we run again some real stuff, okay. So for example this real set, we would like, so in the Aurora definition, the constraints go down here in the job. So this just runs a single process, but we say that this should be, for example, only running on nodes which we, which have some attribute public, maybe because we want to expose that real server directly to outside requests. Or we want to, we don't want to run more than three of these real servers per rep. Because for example, so actually this runs 80, or 10 instances here which you can hardly read. So this runs 10 instances at the same time. And we could say let's not run more than three per rep because it doesn't gain us much in high availability. And we might only want to run one per host. So this gives us some control over how available this task will be, in case of failure basically. And in marathon we would formulate that in a different way, but kind of similar. So this would, for example here, we would use, so these are some, like some operators we define here, and then we here have some attributes. So here we have some objects, and here we have some attributes. And these operators are then applied. For example here we want to run only on nodes with type public nodes. Or we want to group by as another operator, which means that we want to distribute tasks in a like round-robin fashion across Rack ID. So this would go and run tasks on each, or try to run tasks on each Rack. And we can also constrain that we don't want to run more than three per rep, for example, which mirrors this here. This limit per Rack ID and Aurora. And we want to run a single task for hosting. So this kind of mirrors this here. And there are also other operators like like and unlike where we can do some regular expression checks for one of these expressions. And with that we can build up some scheduling scheme that fits our application. So as a, I think this is my last point here. So as a last point, let's talk also about stateful applications. Since some big data applications they typically work with state and this is kind of, I think, one big advantage of using, using measures that can actually work well with stateful applications. So applications need to typically persist. But stateful applications need to persist disk to some physical disk volume. For example, your database needs to persist some information to disk. And since tasks can always fail and they probably will fail, it would be nice if we would be able to restart that task on the same machine where we would persist that data. And we might want to run, if we run some batch processing pipeline we might want to run tasks close to that data as well for example in the same rack of maybe in the same machine. And both Aurora and Marathon they can hook into Mesos and use for example persistent volumes scheduling constraints to give us some stateful applications. And so Aurora does this on a very, on a very general and rather, I would say it's slightly flexible but it works well for many people. So in Aurora we can, if we start some agent, major's agent with some attribute dedicated and then some identifier here. This is treated especially by Aurora. Then we can add some constraint which I talked about earlier. So we can say this task, this job here, this Aurora job should run on nodes which have the attribute dedicated set to dvData. This should be lowercase. So this is lowercase, this should also be lowercase. Otherwise this wouldn't run. And by that we can make sure that we run that task on that machine again and for example we could then mount some disk into that machine or we could have just some directory where we store state. In Marathon this hooks more generally into persistent volumes which are a way to express that we would like to work with some disk resource which should not go away immediately with the task going away. So here we have some command running. So this is called who and runs this command here. So some database working on some directory. This is how I wrote this down here. And this application can declare that it wants to work with some volume which we would like to see under the path data in the root disk of the task with some circumcise. So this application declares that it wants to use 128 megabyte of that volume and it wants to read and it wants to write. And we need to set use the mesos container as I think so this field is also required. And with this app definition this task will then be able to find this volume again should it fail. Okay. So that's my summary. So I'm done. So I showed you that frameworks are user interfaces and distributed applications. And there are many kinds of frameworks and they can interface to different kinds of tasks like big data processing toolkits like Spark or Aduk or MPI can be used to implement batch schedulers to implement data processing pipelines or distributed products or there's even frameworks which can manage other frameworks and be more general. And these meta frameworks that I showed at examples they provide new tools on top of mesos to build distributed applications with high level of control. And they are a good starting point to go from there. That's it.