 Hi, Nihao. I'm very excited to be here in China and to give a talk to the at the first mesoscone in China. A little bit about myself. My name is Alex. I'm a passionate mesoscameter and I'm working in mesosphere. I would say that my primary activity is to make sure that mesos is great and becomes better over time. That's good. What I'm going to talk about is what I observed recently when people come to me and say, for example, if they use DCOS, that's a product that we build on top of mesos. They say it's great. I can just click, click, install Kafka and it runs on my data center in my cluster. I can do click, click, install whatever spark and it runs in the same data center. They say, well, it's magic. And even though it is a great talking of appreciation of my job or our job as a company, I think that it's not good for people to think about this as the magic. I would like that people understand there is no magic at all behind it. And that's the purpose of my talk. I would like to reflect on current technology and about why do we observe what we observe now? Why do we see this technology and how this technology is going to evolve? So the short contents of my talk, so first we do some. We talk about philosophical things. It's Saturday, so it's technically not a working day. I think it's good to talk in the about philosophy a little bit. Then we talk about modern technologies, namely containers, microservices and cluster managers. And then I will try to give some processes about the nearest future. You may trust me, you may know it's up to you. The first, some background information. I'm a software engineer. I'm pretty sure that you are software engineers as well. Most of you are software engineers. The question is what is what is the primary activity of a software engineer? What is our job? I tend to see my job. What I'm getting paid for is finding the right abstractions, not writing code, not testing it, but finding the right abstractions. Because I think that the right abstractions is something that allows others to become more efficient. I looked at Wikipedia, what Wikipedia thinks about what abstraction means. And an important bit here is that it allows to defer complex details, or to forget for a moment about some complex details of any system. And because of that, to comprehend the system easier or faster. Another way to think about abstractions is that we reduce the amount of information that one person needs to interact or again understand the system. And well, of course the question is what the right abstraction is. And I think that's why I'm paid for. Because I'm probably not that bad in distinguishing right abstractions for bad abstractions. But yeah, we should always use our own judgment, our experience, our education to understand what the right abstraction is. But I would like you to keep this in mind, the idea of finding abstractions while we will be going through this talk. How it works, how I think it works. If you have an abstraction, if you abstract away something difficult, and you provide a user of your new abstraction as succinct DSL, a way of expressing himself or herself in this new abstraction that allows, that creates a new level of interaction and allows to abstract away everything what's behind or what's below this level of interaction. And by abstracting everything what's below, you can achieve better efficiency. For example, if we talk about cost manager operations, instead of explicitly saying that you need 10 instances of a particular task, say your web service or your website, that should be run on these and these and these particular nodes, what you do instead, you say, I just need 10 copies of this task in this data center with these constraints. But constraints can be availability zones. So this is a totally different mindset. It's a different level. Instead of providing a set of steps, what you do, you express your intent. Yes. And that's what I said. That's how I see my daily job. That's my daily job, finding the right abstraction. So we are talking with colleagues about our projects or about new features that we add to methods or whatever system, always try to filter them through. Do they provide an end user, an operated cluster operator in my case? Do they provide users with a useful abstraction that will make his or her life easier and will allow for better automation? Some examples from real world, you probably all know what this is. So it's a telephone number and I bet almost, I bet no one knows what code, country code 375 stands for. The thing is, you don't need to know this. If I give you this number, it's abstracted away. You don't, you don't know, you don't need to know how to call to someone who is, this is a country code for Belarus, my home country. You don't need to know how to call to Belarus. You don't need to know what physical mean the signal will use. Will it be a wire or a satellite? You don't know all this. You don't really care. All you need to know is the number. And this is the, well, this DSL is very succinct, just a number and some international format. You may want to know more if you would like to, for example, you may know that first part is the country code and even know some several country code by heart. But again, a lot of things are abstracted away. Another thing is that this abstraction is scalable. For example, when GSM came in, what you had to do had to introduce more network codes, which is the second part of the telephone number. So these kind of abstractions available in our everyday life. Another thing that is probably you most familiar with, when you create a socket, a unique socket, you don't really care what, what underlying network technology is used. You also don't really care what operating system is being used on the target machine you open socket to or you try to connect to. These things are abstracted away. You don't really need to know how this works, whether you have token Reno, you have other net. So we're done with the philosophical part. Keep these abstractions in mind. Let's talk about container. So again, I'm a method developer. And I think you're familiar with containers and why containers, why container technology is good or useful. And I'm pretty sure almost everyone has heard about containers. From my point of view, from my agrees to the point of point of view, containers are good for two reasons. First is isolation. If you can't do isolation, you can't do efficient computation of resources, you can't calculate where to put this particular task or container that you can isolate in the cluster. You can't do that. In this sense, containers, basically is a prerequisite for a cluster manager like mesos or Kubernetes or DCS to exist. Because if you have, for example, a node with four CPUs, and you have two tasks, it should use two CPUs, and you put them into the same node, and then one starts to use four CPUs, then, well, another task would start. What shall we do in this case? If you cannot ensure isolation, your smart placement algorithms of tasks into your nodes in the cluster, is a random void. It doesn't make any sense. Another reason why containers are good in terms of cluster managers is packaging. So you're most probably familiar with packaging for us. This is a bundle from Mac OS when you just have a single instance, which has everything inside. You can actually even look what's inside. You're probably familiar with Android operating systems and APK format they use, and you're also probably familiar with App Gats or Hume. This is, again, something that enables me, as a mesos engineer, to be efficient at my job. I don't have to think about dependencies, about all these thousands of configuration files when I just want to run a task somewhere in the node. I can just say, well, run this task on this node. I just take one entity, one instance, which is contained in this case, and everything is already inside. I don't have to care about this. So this is the right abstraction for me because it makes my life easier by abstracting away isolation and packaging. That's how containers basically influence the rise of cluster managers. A little bit of history. When I say container, I don't necessarily mean docker container. I mean just something that provides you with packaging and with isolation. For example, mesos, mesos are a little bit older than docker, and in mesos we have how we call them our own mesos containers. And before the docker was a big thing and everyone was talking about docker, we were already doing containers using C-groups directly. So, but without that, mesos wouldn't have been possible. Same Google guys who appear near cluster managers with their Borg, they basically invented containers and they pushed for C-groups to be committed into Linux kernel. Microservices, another abstraction or idea or architectural approach, how to write applications, which is very often mentioned in relation to distributed systems. I love music a lot, so I will be using one of the analogy, I will be using throw the talk is related to music. So, if you look at this, you have each module, each entity has its own job and usually it does just one thing. There is an amplifier, there is a speaker, there is a CD player, probably a tape player, probably an equalizer, and each particular component does just one thing and it does this thing well enough. And here you have just four, five speakers, four speakers and one extra bus speaker. And suppose you have a bigger room and you would like to scale your sound system, what you do, you just add more speakers, probably an amplifier, an extra amplifier. And the same idea that sound engineers are using for four decades and also other engineers, software engineers are using what they call the Unix philosophy, where you try to have your your program split in simpler, modular parts that just do one thing short and that are great in doing these things. And the same idea now we try to apply to the world of distributed systems. Yes, microservices are hard, yes when you, they introduce a lot of dependencies into different components, yes it's very hard to work with them, especially manually with these hands, if you have to use your own hands to make sure how to put all these charts and instances of microservices in your cluster, how to connect them, yes it's hard but unfortunately, at least to my knowledge, is the only way how we can scale efficiently. If you, if someone comes up, comes up for a better abstraction which is easy to use, that will be great. So if you have any ideas, that will be great. And in the world of a patch analysis, I hope now you get a better idea how, why we need and why, how did we come to the world of cluster managers, how containers enable that, and how we use of microservices, which is probably the only known way to efficiently scale the implications, how all these prerequisites motivated the development of the rise of cluster managers. Let's talk about abstractions that we use in Mezos. This is how most of the companies run their data centers today. So this is a traditional way of running a cluster or data center. I hope you can see this slide. It's pretty ridiculous. If you start an application on your browser, you don't really say, well run on the core number one, or core number three. You don't have to do this mapping in your head. You don't do this. Why do we do this in data centers? It doesn't look efficient. So the first idea, the first abstraction, is that we think that treating your entire cluster as one single computer is a good idea. Right? Second. Okay. Now you have containers. Now you have a task that you can isolate and package. You don't care what's inside this container. You don't care what is some explosives or toys. You don't really care. And you're pretty sure that what's inside can really affect other containers. It's also very easy to transport. So you just put this container on a boat and ship it somewhere. Now I have multiple containers. And you know that with microservices, the number of containers rises pretty fast because you try to have smaller, you have to smaller pieces for your applications. That's one way how to transport. And some people do exactly this. They take virtual machines. They put the applications already wrapped in a container to this virtual machine. And then they have a fleet of virtual machines that they have to manage. Well, that's one way of doing things. Probably not the most efficient way. And this looks like a more efficient way of doing things if we're now usually containing analogy. So you just put everything into one big boat and just ship it. So the second idea or abstraction that we think that is very useful is that you should run different types of workloads in the same cluster to allow for efficiency. And by doing this automatically, you actually free your time and your free resources to focus on different things like how are you going to scale your applications and not about how actually scaling them. Okay. Again, coming back to sound analogy. So here you see we have an amplifier. We have a tape player. We have a FM radio and a CD player, right? So imagine you don't have a tape player. So you're like music. You say, well, I'm a young guy. I'm still under 30. I don't have tapes. And then you're dead or your grandpa comes with a tape and says, well, look, I have a great record in here. Let's listen. I'm pretty sure you were liking you like. Well, we can't. We can't do that because you can't really play tapes. So the idea here is that each component knows how to work with a particular form and it uses the common API or common technology, specific cables to connect to the amplifier that just takes a signal and produces the signal. And how it transfers to methods within a single DSL is not enough, at least not enough yet, which means spark workloads and Kafka workloads or Cassandra workloads are very different. Do you want, do you want the cluster manager to to be so general that it doesn't really count for those differences? It's like having the all in one boombox from, I don't know, from nineties. Well, it's doable, but probably you won't be able to really have a modular architecture that to be already decided is good. Probably the sound quality won't be that great. Okay, we can embed everything into cluster manager. So we'll make cluster manager know about every different type of workload. Probably it's again, not efficient. And it will be very hard to support code. So what we opted in instead is to level architecture messes. I'm pretty sure you've heard about that. That's why we have in messes core, which is messes and frameworks that do, they do, they do know a lot about the particular domain they are working in. There can be general types of frameworks. I'm pretty sure you've been to, to have talked about my colleague Benjamin, who was talking about different frameworks. But that's the essence of having two level, two level of scheduling and messes. Yes, sometimes it's harder. Yes, you have more frameworks, but this gives you flexibility without polluting messes name space. So what we do instead, between messes and frameworks, we provide primitives, like launch it does, kill it does, send a status update, acknowledge a status update, reserve some resources, and then frameworks, each particular framework builds a DSL on top of that. And this DSL, DSLs can be very different. For example, if you look at DSL and Marathon, Jason, task specification or Aurora, they are very different. And they use different mindsets. This flexibility allows for two things. First, people can come up with their own solutions for their own problems that we don't even know about. We don't even know about the existence. So we couldn't even possibly create something in the messes to account for that problems. And second, unfortunately, in the real world, we have to support legacy applications. And it can be, well, it's not always easy. And by having this extra layer, and by giving people the ability to do whatever they want on that second layer, we facilitate port and legacy applications. So bringing all pieces together and coming back to the container shipping analogy. So you have a container, you can put everything you have in the container. And then now we have to ship this container. The traditional approach, how a lot of people around the clusters today, what they do, they try to say you want to ship container from China to Germany. What they do, they open the internet, they search all the companies that ship containers from Germany to from China to Germany, they find a particular vessel that suits their own expectations, I don't know, price, color of the vessel that they like, time that it will spend in the spend on the way. And then they bring they contain it to that vessel at a specific time when the vessel is loaded and they make sure that the container is loaded somewhere, somewhere, I don't know, in the right place on the vessel depends on the route of the vessel. So if Germany is the first destination that the container should go on top, if Germany is the last destination that the container should be, should be loaded first. And then when the container is brought to Germany they have to come to the harbor, unload that, so all these steps, what they can do instead, they can just go to a company and say, I need to ship one container from China to Germany with these constraints, namely the price should be say less than 1000 euro per container and the container should spend, should be delivered not later than one month after that. And that's it. And this is the abstraction and this is, this is the abstraction that allows you as the shipper of container to be very efficient. However, oh, it's got a little bit cut. However, in the real world, we have legacy applications. So suppose there is a small dot, a small shape somewhere in that new modern harbor which is completely automated, a steam shape that doesn't know about containers, that barrels of rum are shaped somewhere on that boat. And you want to support, you want to be able to handle those ships as well in your, in your modern harbor. And this is also a task that we try to solve and that's, and this is why the two level or two levels, two levels scheduling can be useful. So that we can create a separate terminal, which will be a separate framework, probably that knows how to handle barrels with rum in our modern harbor, which is mezzos. Let's talk about the future a little bit, the future of mezzos, the future of cluster managers. I think it's pretty, the future will be pretty common. So of course, there will be new features in mezzos. So what we're working currently now are priorities for tasks, hierarchical roles, multiple roles. I don't, I don't want to dig into details. There will be improved multi-tenancy. For example, what Google uses and calls CPI squared when they analyze the performance of different tasks, well, they don't actually analyze, they just track and use the statistics of performance of different tasks running on the same node and they try to find workloads that play well together and they don't. We introduce different type of resources, for example, the scarce resources that will be revocable resources. I also think that once people become more and more familiar with mezzos and the allocation techniques, we will see more allocators and more allocation algorithms. I can't imagine that our built in hierarchical allocator is best for all needs. I can't believe that. I just think that people still trying to figure out the first steps, how to run workloads, how to run their clusters in the more than paradigm. But I do think there will be new algorithms and some companies, for example, Two Sigma, they have their own general proper scheduler called Cooke, and they implemented a scheduled algorithm inside that framework. So it's not generally usable because it's just for their framework, but I expect that such algorithm will be backported. Another thing, of course, the reason there will be competition. But like on that slide, these are all operating systems. They compete from one perspective, but they don't really compete from the other perspective. Like, designers, computer designers, they tend to use Mac computers no matter what. Or people who play games, they tend to use Windows computers no matter what. And if you have a health device, like a mobile phone, you probably will be using Android or Apple, Apple iOS. And I think the same thing is happening and will be happening in the world of distributed systems because there are some cluster managers, but they don't really overlap. They do overlap in some cases, but they don't completely overlap. And the ideas will be borrowed from one distributed system to another. And that's what we already see. And when I was recently designing the health checks and measures, of course, I looked how Kubernetes implemented health checks. I didn't copy everything because I thought we can do better than that. And I hope we did a little bit better than that. But of course, I looked what they did. And also, they look what we do. And it is great. This is a great thing that different companies and engineers exchange ideas. But I don't want to dig into benefits of all of our technologies now. Another thing that we will be observing is unified and stable APIs. As systems evolve and become more mature and stable, the APIs will also become more stable. Like with this T-Pod, I think the API didn't change for hundreds of years. There is a lid, there is a handle, and there is a nose. And that's it. And once we figure out and get more and more experienced how to run clusters, then we will also figure out what APIs are necessary. Now we're still in the process of figuring out if we talk about measures, what primitives are necessary, what primitives are useful. Do we really need to attach, say, ideas to persistent volumes? It doesn't matter if you don't know what persistent volumes are. Just imagine we had these conversations and I'm pretty sure that in three or five years, everyone who is in the world of distributed systems, they will know, oh, OK, yes. You need to create a persistent volume of course. Each persistent volume should have an idea because of this and this and that. We just need to get that experience. The architecture of our data centers will be simplified. This is, for example, if you run Mesos cluster on top of GCE, that's how you're running. So we have hardware. We have a system similar to Mesos and similar to Borg. It's not completely Borg. It's something that reminds of Borg. Then there are VMs running in containers. And that's what you get as a GCE offering. Then you run Mesos on top and then run your tasks in containers, in VMs that run in containers on hardware. So this is pretty ridiculous and it's not efficient at all. But that's how a lot of people still run their systems and they don't really think about that. So I think once again, people figure out how to code distributed system, how to run their clusters, they will turn to these questions. Like, do I really want to sacrifice cycles of white CPUs for these extra unnecessary layers? Probably not. And I hope the ultimate goal will be when people ask these questions, when people come up with better algorithms, with better allocators, when people write their own frameworks, we will achieve high utilization, which means we can save more energy, we can save on hardware and make this planet a better world, a better place to live. If you have any questions, you can ask them now or you can just come to the Mesosphere booth afterwards. I will be there open for questions as usually. So now it's time for questions. Thank you. Okay, so the question was that I mentioned about possibility of multiple allocator implementations and why it didn't happen today. I think writing your own allocator is hard. It is very hard. That's the first reason. And I don't think that there are a lot of people who do understand how current allocator works and it's a pretty simple allocator. Another reason is that we're still struggling with the API for allocator. We change API a lot. And if someone wants to write an allocator, technically you can do it. So allocator is modularized. You can do it. Everyone can do it. And I think I also have somewhere in the GitHub dirty version of a little bit slightly modified allocator, so you can do it. But we're still changing the API pretty fast. You remember Unified Stable APIs? That's in the future section, not in the current world section. So that's the second problem. And third, we haven't completely implemented our allocator. So the reference implementation, which is our allocator, is not completely yet. So we haven't figured out how do you want our roles to be structured? Hierarchical roles, multi-rolls. For example, there is a call request resources. It's not implemented because we think that we probably don't need it for now. So when someone wants to implement a new allocator, they look at this and they're like, well, but the reference implementation, I still have questions. However, however, I know that there exists at least one alternative implementation of the allocator. And I think I can talk about this. There was an IBM research, there is an IBM research center in Israel in Haifa. And these guys, they were important, one of their system to Mesos. And for that, they needed their own allocator. So they implemented it. I know that because I was talking to, I was helping those guys to implement their own allocator and all I know that they implemented it and since then I don't get questions. So either they don't maintain this system or it works perfectly. Any other questions? Okay, then we are overachievers and finished ahead of time. Thank you.