 But before I get too far into that, I just want to give a bit of clarification. You'll hear a lot about Mesa's and you'll hear a lot about DCOS at this conference. And I just kind of want to clarify what they are. Mesa's is the kernel of DCOS. So just like you wouldn't use Linux directly. You would use a Linux distribution like Ubuntu or Red Hat, Enterprise Linux. So when you necessarily use Mesa's directly, by itself it doesn't really do anything. You need to pair Mesa's with something like Marathon or Chronos or Jenkins in order to actually get it to do anything. Where DCOS comes in is kind of this ecosystem of all of these tools and projects around running applications and running distributed systems. So some of the things like security and monitoring, the health of the cluster can all be bundled together with DCOS. So like I said, Mesa's is kind of this general-purpose cluster resource manager. It takes many machines in your data center and allows you to program them against them as if they were a single entity. Resources are advertised directly to these frameworks, or Mesa's term for frameworks is these distributed applications. Just like the Linux kernel would advertise resources directly to user space. And some of the companies that are using this successfully include Apple, Twitter, Airbnb, Netflix, some of the largest companies on the internet. So Mesa's does this by having a two-tier scheduling. What that means is it's taking resources from individual hosts, CPU, memory, disk, network ports, and it's advertising those to these distributed systems. The frameworks, the scheduler, can then decide whether it has work to do or not work to do, and accept or decline the resources that it's offered. The Mesa's masters are highly available. They use Zookeeper for leader election and to keep a small amount of state about the cluster. The agents are also fault-tolerant. So using a feature called Checkpointing, they can actually have the processes on each individual agent stay running if a Mesa's agent restarts. So this is really powerful because you can essentially upgrade Mesa's across your data center without having to reschedule your workloads or incur any downtime. If you think of other distributed systems, for example, if you try to upgrade a Jenkins master, your entire Jenkins cluster is offline while that master is being upgraded. So this is a very key point that the original Mesa's authors wanted this to be highly available. Using, like I mentioned, using C Groups or using Docker, there's resource isolation between processes and there's enforcement as well. So if a single process or a single container goes over its amount of memory that it was allocated, the Linux auto memory killer will come in and kill that process. It might be restarted elsewhere, but it's up to the scheduler to decide what's going to happen with that container. Another really key selling point to Mesa is the language bindings that are available for you to write distributed systems on top of it. There's C++, there's Java, there's Scala, Python. Some of the folks at Mesa's here maintain the Go bindings. I've seen Ruby bindings. I think there are early bindings too. Just a lot of different languages for you to write in against Mesa's. With the new HTTP API, that's not even easy. So this is kind of what the Mesa's architecture looks like. Here you have the three masters and they're using ZooKeeper for a bigger relation and for a small amount of state of public cluster. One master leads at a time while one or more other masters can be stand by waiting to take over. So usually masters are deployed in odd numbers. Three works well if you want to allow one master to fail. Five or seven works a little bit better. That way you can take one out for maintenance and still be able to tolerate one failure. The schedulers for distributed systems connect with the Mesa's master core and the schedulers launch tasks directly on two agents based on resources that Mesa has given that scheduler. Each of the agents advertises its available resources when it starts and the Mesa's masters persists that and offer those to the frameworks using an algorithm called DRF or Dominant Resource Experiments. That allows one framework that may be CPU heavy to use the cluster the same fairness as another framework that may be memory heavy. To kind of illustrate that resource offer process, we can visualize it like this, this cyclet diagram. Each of these agents has resources to offer 16 CPUs, 128 gigabytes of memory, a terabyte list. Those resources are advertised to the Mesa's masters. The master advertises it directly to the scheduler. If it's accepted, a task can be launched and if it's declined, those resources can be offered to another framework. It allows for the fair sharing of a single computing cluster as opposed to giving one group 10 servers and another group 20 servers and partitioning the data center that way. It's a very intelligent system. I just want to go over some use cases that I've seen as far as when I was getting started with Mesa's what was a really popular and compelling way to kind of demonstrate this power. First and foremost is Spark. Apache Spark actually started off as a project to test the hypothesis that specialized frameworks are more valuable than generalized ones. If you read the original Mesa's research paper, Spark was actually created at the same time as Mesa's to be able to test the theory that the Mesa's creators were over trying to prove in the paper. Since then, Spark has gone on to become a very successful project in their companies using Spark in production within and without Mesa's. Another popular use case is Jenkins. Jenkins can be a traditionally deployed standalone where you would dedicate a certain number of virtual machines or servers in your data center to run a Jenkins cluster. But for anybody that is running Jenkins, Jenkins isn't always busy or running builds. Sometimes it's just sitting idle. So if it's sitting idle, those resources can be put to better use. So there is a Mesa's plugin that was originally developed on Twitter for Jenkins that allows Jenkins to use Mesa's as its underlying cluster manager. So that Jenkins isn't really doing the management of all those machines. So that allows containers to be spun up on demand, run the build, and then after a certain idle period, the containers are then torn down. Marathon is probably one of the most popular use cases. Like I had mentioned earlier, Mesa's only cares about running processes that are running containers. If that process dies, it's up to the scheduler to determine what to do with it. Marathon allows you to say, I always want this number of processes running specifically for an application. And if one of them dies, when Marathon is notified of that, it will go ahead and spin up a new container based on the next resource offer it sees. So it allows you to declaratively state, I want to run this many containers. It doesn't matter where, make sure that this number of instances of my application is always available. Chronos is the one that I actually got started with. It's a distributed cron. So it allows you to run commands on a schedule. So we had a use case at a previous job where we had some cron jobs running on individual machines. And one day one of the machines failed and it was replaced and nobody thought to recreate the cron jobs that were running and nobody had documented it. Chronos is really powerful in that it is also highly available. And all of its cron jobs are stored, I believe, in a zookeeper. So when the schedule comes up to run a process, it gets a resource offer from Mesa's and launches that as a task somewhere on the cluster. So the individual machine doesn't matter as much as I have this process I wanted to run at this time, just do it. You can also create your own schedule where, like I had said before, there are C++ and Go and Python bindings for Mesa's. This is an example of using the Python bindings that are available. I believe the only method you have to implement is resource offers in your scheduler when you're subclassing. So I apologize for any non-software engineers in the room if this is more of an operations audience. But all this is really saying is for each of the offers we receive, take a look at the resource names, aggregate them, and then it's just going to print this message that we've received an offer with a certain number of CPUs and a certain amount of memory. And from there we can do whatever we want. You know, it just ends here, but we could launch tasks, we could take a look at that offer and see which attributes it came with, determine which data center it came from, determine which rack it came from. You could do some really cool scheduling things just based on the information coming out of the cluster. Usually when it comes to deploying Docker or deploying applications, something like Marathon would fit 90% of use cases. There are other distributed systems that might also fit your use case. I would say that most times you don't need to develop your own scheduler or your own framework, but if you have custom business logic that warrants it or if you have a specific scheduling scenario that warrants it, that might be an option. I've been talking about this idea of clusters and like I alluded to earlier, we're now taking the resources from many individual systems and treating them as if they were one computer. And that presents some of its own challenges and a lot of these sound like old problems and they kind of are except now they're distributed. So for example, in the past we could look at service discovery and load balancing by having names in DNS or you could have a spreadsheet or a puppet or some other tool to track which services are running on which machines. Now that that service could be running on any machine, we need a way to dynamically do that. So you could use something like Mesa's DNS console, Mesa's or Marathon LB and it queries the state of the cluster and builds DNS records or builds HAProxy configurations based on that information. So where you might have had to go in and manually configure load balancer before, you can use the system to do that for you. When it comes to monitoring a metrics collection, we still have the same issues. You still need to run an agent on each of your machines in order to collect that information. Mesa's provides some information via its APIs. I'm giving a talk tomorrow on metrics collection with Mesa's. So if you're more interested in that, I'll do a deep dive there. But traditional tools like collecting, Nagios or Prometheus work just fine. We also have an issue with persistent storage. Mesa's is largely a stateless system. We've made a lot of progress on running stateful services, especially distributed file systems and databases, by writing custom schedulers that can tolerate failure and can tolerate upgrades. For example, I believe the HDFS scheduler, the Cassandra scheduler, ensures that when you're doing upgrades, you only take out one node at a time, so that you don't have any data loss. But distributed systems that would normally run on a fleet of servers can easily run on Mesa's. And using Mesa's is a scheduling system. You can tolerate failures that way. So if an agent goes down, the scheduler gets notified and that failed task can be restarted on another node. Or it can wait until that node comes back. Maybe it was only going down for schedule changes. There's also, if you're running in the cloud, there's Amazon, EBS, S3. We also have an issue of administration. We're running multiple masters. We need to know which one is leading, which one to query for information. Some of the tools that we can use for that are, like I said, Mesa's DNS, agent proxy. We have a load balancer called In-A-Man built into PCOS. I gave some examples here of different tools for solving some of these problems, but I'd like to wrap all of this up and just kind of point out PCOS here. And the reason I'm pointing out PCOS is because it is an open source project that has all of these tools built around Mesa's. So, kind of like you heard in the keynote earlier, you can build your own Mesa stack. And that might be something that you want to do or that your business wants to do, but not everyone needs to do that. And if you build your own stack, you have to also maintain it. So, if you look at an open source project like PCOS that has all of these tools built in and great all of them that ensure they work for every release, you can really save yourself a lot of time. So, like I was saying, Mesa's and DCOS are really better together. DCOS is more of an operating system distribution built around Mesa as its kernel, just like you would expect Ubuntu or Red Hat to be the distribution around the Linux kernel, except this is across many machines. Some of the things that I talked about earlier, like security, networking and operations, those are what you start to get in DCOS. And a lot of this is open source. There's the web interface, the command line interface, the marathon, the universe, out store. And then everything just runs on top of that. So, it allows you to bundle all of the components that make up a modern Mesa's infrastructure in one easy-to-use way. And we can consider that for deploying applications, we have these five basic requirements. At some level, there's scheduling. There's actually running a process on a machine. There's deploying your application, whether there's a systems administrator doing that by hand, or Jenkins is doing it, or Puppet is doing it. Somehow that application means to actually get on to that server. Next, you need health checks. Things like making sure the service is alive, responding to you. Service discovery is another big one. A lot of times, we would be able to deploy an application and just tell it to configure or connect to this database running on this hostname and this port. We can't really do that if database could be running on any node in the cluster. And finally, persistence is running these stateful services in containers like SEP and HDFS and Cassandra and being able to tolerate failures and not incur any data loss. So before Mesa's and DCOS, a systems administrator would have to provision one or more physical or virtual servers in order to host the app. And in some cases, these systems administrator may deploy that application or hand it off to a software developer to deploy the application. With DCOS, it kind of draws a line almost between the developers using the cluster and the operators maintaining the cluster. So essentially, you can have operators providing this platform as a service, this infrastructure as a service that is Mesa's or DCOS. And the resource authors are advertised to the distributed applications running on top of the cluster like Marathon or Jenkins. And your applications or your workloads run that way. As far as deploying applications, I had mentioned this before, Jenkins might be SSHing into the machine to deploy the app. Puppet or Chef or Ants Bowl could also orchestrate that for you. But note that all of the dependencies have to be present on that machine. Now, using something like Docker, you can package your application and its dependencies and run it on any machine in the cluster. In Marathon, we deploy those containers, hopefully using a CIRC tool so that you get the same deployment every time. I really want repeatable, robust appointments. Before health checks, not to notice my email or send an alert to a systems administrator in the middle of the night or at any given time, letting them know the service is down and they need to go to work. Marathon performs these health checks for you. You can have it look at specific URLs of your application or make sure a specific report is listening and returning all the expected information or valid error code. And if any of those go unhealthy, Marathon restarts those instances automatically. So if you bottom it away, they get a page for a single instance scaling. For service discovery, like I was saying, when you deploy applications, you would have to tell it which database to connect to or some other dependent service to connect to. I've worked in an environment where all of the host names and IP addresses and reports for these applications were stored in a spreadsheet and you just had to keep it up to date. And if it wasn't up to date, then with Mesa's DNS providing DNS information about the running applications in that cluster, we're worried for a known service name. So let's say like redis.marathon.mesos. And it doesn't matter where in the cluster it's running, Mesa's DNS will have a record of it and will resolve to that IP address for that host name. Alternatively, systems administrator would need to configure a load balancer either manually or with Partitive or Chef in order to handle incoming user traffic to the various instances of an application. And now with Mesa's and DCOS, these load balancer configurations can be built dynamically using the state of the cluster available and the running applications on that cluster. DCOS and cluster computing, we had individual servers that had to be fault tolerant all of our data to live there. Their storage had to be grade one, five, six, or ten, depending on performance and the level of security that we wanted to have for data. Or you could have statically partitioning, Chef or cluster data stores running in your data center and have your applications connect to those. But then again, you have that static partitioning problem and you have to maintain your set or your cluster as if it was a separate thing. You could turn around and run separate cluster right on top of Mesa's and it will heal itself if one of those versions fails. You can also use these external or persistent volumes. EMC has this awesome tool called RecSpray that allows you to dynamically provision storage volumes based on your application. Your application is restarted, it automatically reconnects to that volume. So this is really powerful if you're running out of Amazon, but you can automatically provision these EBS back volumes if that instance fails and all of your data is still safe. So I'd just like to end this talk with given that we're all engineers here, whether we're on the operations side or the development side, which tools would we like to work with? Do we want to be paged at 2 o'clock in the morning when an application goes down? Or do we want the cluster to be able to handle the failure for us? Do we want to maintain host names and IP addresses and reports in a spreadsheet or in a text file somewhere? Or do we want to hand that information off to the cluster and let the machines make the scheduled decisions for us? I think the automation that Mesa's and ECOS provides is very powerful and it's a different way of thinking, I honestly believe that this is the future of computing and it will allow us all to do our jobs much more efficiently. With that, I really thank you for attending this talk and I hope you enjoy the rest of Mesa's talk. If you have any questions or would like to follow up with me, you can send me an email at roger.mesaspear.com. Thank you.