 So, morning, everyone. I'm Yagnik. I work at Shopify. So before I go forward, I had a couple of people yesterday who confused me working at Spotify. So before going forward, I'll just let you know Shopify is not Spotify. We have nothing to do with music streaming and we're not the same company. So Shopify is based in Ottawa, Canada. And what we are, our e-commerce platform, we allow our users to sell both online and offline and essentially make it really easy to build their own store. We currently have approximately 100,000 store owners and sold approximately $1.5 billion in products last year. So it sounds a little bit like I'm preaching Shopify, but to be really honest, my point is to tell you that Shopify is growing. And the main part of that is, well, my animation got screwed there. And I told you everything before. So whatever. We have 100,000 stores, which actually means to us is 500,000 requests per minute, which spans across 450 physical servers, which have 1,700 processing cores and six terabyte of RAM. That's what Shopify looks like right now in its production cluster. Sorry, lost my flow there. So yeah, that's what Shopify looks like in its production cluster. And it's not really, you can't really see how hard it is to maintain such a cluster. You can't see how much work a developer puts in, how much work an operations person puts in. And essentially, one does not simply ship code to a server. So my point here and to talk about today is to tell you what's going on right now, what we're trying to fix and how we plan to solve it in the future. Let me tell you a short story, which you may or may not relate to on your daily life as a developer or an operations person. So I started my own self. I'm a developer. I go around and go like, I write code, I build applications and essentially build new apps that would help scale Shopify. When that happens, before the whole putting into production phase, I go and beg an ops person and go like, can you please give me a server which will allow me to put my application on it and help Shopify do its stuff. And as all ops people, they're awesome. They actually listen to me. Then they go like, there you go. We got your new server. We did the engine X configuration. Right now we use Chef. So we got cookbooks happening, engine up, routing, security, and all that is managed by them. And then they go like, go for it. And I'm like, yes. Go forward. And then I get into post provisioning hell, which is get my code on to production server, but it doesn't work because my server is not exactly the same as my Mac machine, which is a big pain in the ass. And I'm pretty sure most people are familiar with that where your dev environment is not really the same as your production environment. So you go through this whole issue of dev and production and juggling between what's going on, what are libraries it needs, try to fix it all. And then once that post production provisioning hell and setting up the server happens and you're done with the deploy scripts, codes on there, you finally have code in production and you're like, amazing. This is heaven. This is what life is all about. But that kind of comes to an end when your service is actually useful and people start using it and then you cannot scale. You can't go forward. It's just stuck because you can't scale it that much. Then you go through the whole process all over again, go back ops again, I was going to give you some more servers, give you a little more manpower and resources, get it on and you're able to scale it. There's a lot of friction from going code to production to scaling and a lot of work by a lot of people to make it happen. What really happens in the end, me as a developer, I'm super unhappy, really, really tired because it takes forever to get my code from my machine onto production in front of a user. What about ops? I'm pretty sure they're grumpy because I wake them in the middle of the night and go like, we can't handle this load. You need to give me something and we're going all over night to make up new servers and provision them and then get them running. Finally, Chicago. I call my data center Chicago A because it's in Chicago and B because I like to give it a name. Chicago is also feeling underutilized. Something like, you know how you feel where you're like, really, I'm not doing anything with my life and my server feels the same way. It's like, I'm not really doing anything with myself. We're using 15% of my processing power and just using two gigs of the 32 gigs you're giving me, do something with me. I'm like, all right, take it easy. I'll give you something more work to do. And then finally, Shopify. I think Shopify as a whole is disappointed because we can't really keep up with the growth. We can't keep up with how fast we're growing, how many users are coming to our site and really helping out scale. In the end, our merchants are really unhappy with that. What we really want to do is find some goals and try to achieve those. And these goals are being consistent in the environment. That's for me. That's for, as a developer, I want to make my life easy and always be smiling. I go to Shopify and work there so that I could be happy. So making myself happy is primary goal. That's goal number one for me, consistent environment. Fast to deploy and maintain. Obviously, if you have code that you have consistent environment, you want your code to be able to deploy it faster. You want it to be able to maintain that code and essentially go to the whole life cycle a lot faster. High utilization of server. This one's especially for Chicago. Chicago, the data center. Yes. You want my data center to have high utilization. So that's the other goal. Next two are related to scalability and high availability. By scalable, I mean if you've got more users coming in, I should be able to scale on demand. I shouldn't really be worried about what's going on. I shouldn't be worried about waking up ops. It should just scale. And then high availability is making sure that at least minimum number of servers are always online. They're not falling down. They're not falling behind. We're managing their requests. And with these goals in mind, we went down and looked down to various technologies to see what all can we do. And ended up picking three different technologies. Messos, Docker, and Marathon. You can't see the name there, but I decided to use logos because I'm going to be talking about each one of them individually. So if you hate logos and you want the name, you're going to find out soon enough. So let's go, Daya, and deeper into Docker. I'm not going to talk too much about Docker, mostly because I had slides and I deleted them because I saw a lot of people talking about Docker and the workshop on Docker was actually really good. So if you're interested, that would be a good place to look at. But for those people who weren't there and those people who didn't attend the workshop, those people who are seeing it on a videocast, those people who like me want to keep the presentation complete, I'm going to talk a little bit about Docker. So what is Docker? It's a container solution. It allows you to take your code and take your dependencies and contain it in a single container or what Docker calls a container. So it puts it in a container. Essentially, a container is similar to a VM, except that it doesn't have another host. So it really runs on your main machine with isolation of resources and isolation of file systems, pretty much isolating everything in there into the container. So you don't have to worry about cross-managing them. Why Docker? Essentially, you have environment isolation and repeatability. As I mentioned, isolated from host. But the bigger win is it's really fast and simple to implement and use in your daily life. For instance, right now I use Docker to do a lot of my work in terms of building my environment and using that. And I have different Docker containers for different applications inside Shopify. But from that, we also use the AUFS, which I forget what it stands for, like All United File System, something like that. So yeah, talking about Docker. And then, finally, the Delta Update to Container. That's again a part of AUFS, which I'm going to talk about soon enough. So how does Docker work? Essentially, you have your own host machine, which is host one. Sorry I took this picture from Docker's website. It's not the clearest, but gives a good understanding of how it works. So your host machine has a Docker engine on it. You take your code and you have a Docker file, which you run Docker Container, and then you build a container on your local machine, and then you push it to a Docker registry, which is a place where you put all your containers, essentially like... I don't have a good example of what would relate to a Docker registry. It's a list of all the containers that you have or you've pushed, and anyone else can look up that registry and know, get the containers. So say, for instance, we have an internal Docker registry. I will push my container, which has the latest Shopify code, and then another developer from Shopify can pull it down and get the latest code and all the dependencies and work right through it. And then, finally, that second developer is also the host, second host, which is also our production server. It can pull down Docker images and then run those containers and have Shopify code running on there. This is the part that I wanted to bring up mostly because we use it heavily in our new cluster. So Docker allows you to have layered containerization. So you could have the base layer, and on top of that you could have another layer, which is just system dependencies. On top of that, you could have another layer, which is just your code, and then another layer, which talks about just code dependencies, and so on and so forth. So the best part about that is you're not really copying the whole container, you're just adding on delta updates on top of that, something like how Git works in a way. You're adding the delta updates, and in the end you get an overall snapshot of all the updates together. So Docker allows us to meet the first two goals of having a consistent environment because Docker is building a container which can be used on any machine, and it is same across all platforms. So you have a consistent environment and it's fast to deploy because you're having layered systems you would, because of those layering systems, you are building your code really fast and then you can just drop the container on the machine and make it work. So the next part would be on, which is essentially the main part of my talk, Messos. Messos is an open source framework by Twitter, technically by guide, you see Berkeley, but he now works at Twitter. I don't know if he still works at it, but he was working into it when he worked on Messos. So this is how your data center currently looks like, like a bunch of servers on your data center, and you got a bunch of apps, databases, services running on there, possibly some data crunching happening too. And then some people who are actually a step ahead in that deck, they have a lot of services running, so what they do is they end up statically partitioning their data center, which looks something like this where all the red ones are the app servers, the blue ones are the database servers, and then white one is the Hadoop cluster or the Hadoop data nodes. It's really common to see that because then you can, you have a group in Chef and then you could do stuff for specific kind of servers, scale across those kind of servers, and then depending on what kind of stuff you're doing with them, you build those servers accordingly. So say for instance you're building an app server, you would end up giving it good RAM and good amount of good amount of processing power, but you'll give it a really small hard drive, whereas in Hadoop's case you're running an HDFS, you give it a huge hard drive, make the processing power really slow, but give it a bigger RAM, really depends on what you're trying to do with it. So you have a statically partition data center at that time and what ends up happening is when you're doing something like this, you have really low utilization of resources. What I mean by that is a single app does not really use the processing and memory completely. It does not make it 100% usage of the power and resources you're giving it. Secondly, it's really hard to scale elastically. So you can't just make it go press a button and have it scale across multiple servers. You have to do provisioning, you have to go to the same hope that we at Shopify went and some of you probably facing your production environment right now. Unable to self feel, if someone attended the talk yesterday about self feeling infrastructure, it talked about how for the first level support gets removed and then essentially your system is able to automatically get up again on its feed if at any point fails. So for instance, you have 10 app servers, one of them goes down. The system is automatically able to see that one of them went down and able to come up again on its own. Those kind of things are hard to do in these kind of situations. You have to write scripts, have to manage all that stuff and usually have an ops person involved in this process. And finally, it's really hard to deal with failure. So for instance, your whole stack of partition goes down. The idea of getting that whole stack up means you have to repartition or make a new server or fix whatever the problem was. Either way, that part of the time is gone or you're down for say a good hour, which is not really a good case in terms of Shopify or anyone else. So how Mesos actually sees this data center is one single compute resource. Imagine stacking all your data servers into one single box with 1,700 cores and six terabyte of RAM, which is what Shopify production is right now. You see that as a single, you don't see anything else. You don't see different machines. You don't see different servers. All you see is a single resource box. So having said that, I'm going to go through the process of how Mesos works in terms and in a data center, how I'll actually walk you through the process. So let's go ahead. This is my data center. Doesn't look like one giant box, but in reality it's not. It's multiple servers. So you have multiple servers. The blue ones are actually my Mesos master. So on those nodes, I go in using Chef or Puppet or whatever the case is. Go in, I install my Mesos and I go, this is my Mesos master. I run the Mesos master command and I tell it where ZooKeeper is. So ZooKeeper is yet another service which is used for discovery and quorum selection. So selecting the leader essentially and saving metadata. So you select, you have your masters spawned up and then after that, when you spawn them up, all the masters are going to go report to ZooKeeper and they'll be like, hey ZooKeeper, I'm a master. I'm ready to go. What do you want me to do? So ZooKeeper at this point of time will go back and actually go, I'm going to elect one single master. So this is how Mesos fault tolerance work. ZooKeeper knows that the tree master is available, but it's going to use one. And it's going to tell that master as the current master. Whenever that master fails, ZooKeeper will know that it's failing and it will reward to a second master or secondary master. So once you have that, now you go through the process of booting slaves, which is all the red machines. I don't know if it's red color for you guys, is it? Red color. For me, it's red. I might be a little colorblind. Red color, things are slaves and you go through. So you have the Mesos slave running on those. Once again, all of these will go and talk to ZooKeeper and go, hey ZooKeeper, I'm a Mesoslave, tell me who's the master. So it's essentially finding which one of those three machines. No idea about the slave doesn't know. It just knows that one of them has to be the master. So it goes and asks ZooKeeper and ZooKeeper returns back saying, hey, the one in the middle is the master. At that point of time, all the slaves report to the master, telling it, at this point, the slave will tell the master, hey master, I have four gigs of ram in my computing resource. I have a processing power of whatever, six cores or something. I also have network available. I also have IO bandwidth available. I have this much hard drive and all those details are sent to the master. So this is how you boot up Mesos. Mesos on its own is kind of useless if you see it just Mesos. So if you go, if I run that framework on its own, it will not really do anything for me because Mesos needs this framework on top to actually work with it. For example, let's say Hadoop and Rails. Disclosure, Rails cannot work directly on Mesos. It's not a Mesos-enabled framework, but for the sake of the argument, I'm going to say that Rails is Mesos. So you understand how a service would work. So you have two frameworks, Hadoop and Rails. And at this time, both of them will go and ask Zookeeper who's the master, the same theory that applies to all the slaves. It's going to go ask who's the master and then go talk to the master. So both Hadoop and Rails are going to go talk to the master. I'm going to get into the detail of what they do, what this talking is. But once it goes through that process, it'll actually run all the frameworks and their tasks on all the nodes of the cluster. So all the yellow ones are Hadoop tasks and all the purple ones are blue ones. I don't know how you're seeing it. I can make out between the blue and the purple now. But the masters are still there. They won't run any tasks. The slave nodes will. And if you notice, the second from the bottom is actually half purple, half yellow. What it really means is on a slave, using mesos, you can run both Hadoop and Rails simultaneously as long as they're not clashing with each other, which they never do thanks to Linux containers or Docker. So now going into the details of what the talking part is, this is another picture I took from mesos. I think it really explains how the resource allocation works. So mesos at its core has a resource offer model. What really happens is a slave, which is at the bottom. A slave tells in the beginning how I explained about its computing power. It tells it that it has four CPU cores and four gigs of RAM. And at that time, master knows that this slave has these resources available and it's ready to give it to framework one, which is again four CPUs. It tells it that these many resources are available on slave one, so four CPUs and four gigs of RAM. At that point of time, the scheduler on the framework one starts on to see that, hey, I got two tasks which both need some amount of resources, not all of them, but some amount of them. And I would like to use that. So it tells it that it has two tasks, one with two CPU cores and one gig of RAM and then plus two of one CPU core and two gigs of RAM. And it tells it back to Meso's master and then Meso's master tells the slave to go execute those specific tasks which are already present on slave. So you load your code everywhere and it just goes and executes that command knowing that it will run. At the end of the execution cycle, so when the slave is done running its job, the whole process is going to get repeated again where the slave will go like, I again have new resources available, the master is going to find out and then the master is going to go through the whole process again and tell all the frameworks what all you need to run. So at its heart, Meso's controls a resource allocation to schedulers which are part of the framework and the schedulers make the decision as to what to run on those slaves. Does that sound good? Yeah? No? Sure. So going back to this picture, right? So you have a master running. Yep. There is a slave. The slave is kind of running on the same server that also runs your Hadoop or Rails. Sure. Sure. The framework sits where? The framework is actually, how do I explain it? The framework usually runs on Meso's master. Master. So it's, there are two processes running simultaneously but the framework doesn't really executes the task. Yeah. All it executes is the scheduler and it's usually a lightweight program which just knows what all tasks are there and how much resources they need. So pretty much the apps go through this master because I'm not seeing the layer with the app there, right? You have tasks in here? The tasks are... Who's feeding in the tasks to the framework? I'm sorry. Can you repeat that part? What's feeding the tasks to the framework? So at the time of loading all this thing, the task or the code for the task is loaded on each of the slaves. Yeah. So you tell the task name or the task command what needs to be run from your scheduler down to the slave. Okay. You send it across saying run this task or run that task, whatever the task is. So the framework knows... When you're deploying, you have two parts of the framework. One goes onto the master which is a scheduler and the other is the executer that goes on the slave. Okay. And that way the framework knows which task is what. So it's sending it, the master is just forwarding it as it is. Yeah. Make sense? Yeah. Thank you. All right. So how does scaling work? In Meso's case, you could essentially just tell it to shut down Hadoop and run scale up rails and that's what it's going to look like. I should have used the different color apart from purple. I'm really sorry about that. But blue is the master which you cannot see and purple is all the rails that have been scaled out. So if I have to give a final slide of how Meso's look, you have all the nodes at the bottom. You have the Meso's cluster which is the kernel of your data center. All the framework sitting on top of that and then your code runs through those frameworks. So with that, we got another goal cleared out. We have the high utilization of server because Meso's make sure that you will reutilize all the resources again and again. The final part in this is a marathon. Marathon is a framework on top of Meso's as I mentioned. It's a meta framework. It's used mostly for high long running services. So something like a Rails server or usually any of your servers which you spawn once and you want it to run pretty much forever. That's what you use marathon for. In this case, I'll take up an example. Say for instance, you have this cluster and you have three apps running. All of these three apps are running on marathon with the help of marathon. So they are constantly running. If you want to scale up app three using marathon, you can just tell it, I'm not joking. You literally click a button saying scale and you give it a number and it scales it up to that many nodes. So all the yellow ones are again scaling up to those numbers and those are how you would scale Shopify app servers. Or in your case, any of your app servers are going to scale that way. But say for instance, power goes out on one of your servers and it goes out of provision. Marathon actually knows when that happens when a server went down and it's going to automatically rebalance and make sure that those minimum number of apps are always present. So if I have 10 apps, 10 app three servers running, it will make sure when a server physically goes down that the app three is executed on all the new and all the other servers that are available for execution. So that's the job of marathon. Messos doesn't do that internally. This is job of marathon to make sure that the app is always highly available and it's super easy to scale. So I think we got all our goals achieved with Docker being the first two marathon being the third one and the last two being... Sorry, messos being the third one and marathon being the last two. So if you can see the logos, that's how it is. But it's not as easy. In a production system, it's not really as easy as just putting it out there. You have to really make it work together. And at Shopify, we make it work together by using a system called Borg. Borg is just joining messos, Docker and marathon together. And we call it Borg. A, because probably Google called it Borg. B, because Borg is a Star Trek reference. C, because Borg sounds awesome. So I'm going to give a little bit of detail on things that we learned over time of how to implement it together. First part was the Borg API. As a developer, I need to be able to use any of my apps with Borg. I need to be able to just give it my code and it should run it. So in order for Borg to know how to run that code, we decided to have three commands that tell it how to do things and what to do things. Borg prepare was the pre-compile compile. So essentially, it will do the system dependencies. Anything that your system requires, say for instance, libgda or libbython, all that stuff on a bunch that you have to install before you can actually run the code. Borg prepare will do that. Borg compile is what you would do for an app-specific dependency. So in case of Rails, it's usually gem files. All the gems get downloaded and installed at Borg compile. And finally, Borg roles are roles in which the app can actually run. So once again, in terms of Rails, it's Rails server, Rails console, rake, whatever the case is. So all those three are... Each of our applications actually have these three files defined. So we have the prepare, the compile, and role. So that application will respond to these three commands running. And that's simply executed by putting it in a specific folder. Borg slash prepare is the Borg prepare command. Borg slash compile is the file that does the Borg compile. And then Borg roles is the roles file. So the next part would be containers. As I mentioned, we actually use AUFS pretty heavily to make sure our container building process is super fast. We tend to deploy a lot of times in a single day, guessing around 100 times. Never counted it, but assuming. So in order to do that, we need to make sure that it's super fast and it's really not doing everything each and every time. What we, in order to do that, we actually have levels of containers that we build. So latest base images, what all of Shopify uses. On top of that, we run Borg prepare, as mentioned before, which will install the system dependencies. It's cached. All of this is cached at every single level. So till the time that part changes, it will always remain cached. And that's part of Docker itself. Then we copy over the first ref of the month. By that, I mean the first ref in the GitHub commit history. We usually do that again and again. So if you see first ref of the day and then the current ref, mostly so that the dependencies are always nearly up to date. So till the time you update a dependency, only then will it actually take a little more time. Otherwise, the compilation time is two seconds or three seconds because it's just putting on code there. All the dependencies are already present in the container. And finally, we have the lowest at the lowest level, the container that gets dropped onto the production server. One of the bigger issues that we saw going through that process was secret management. Obviously, all apps, pretty much every single app, has some kind of secrets that they're storing in there. You usually don't put it on GitHub. So in case of Heroku, you put it as an environment variable and then your application reads it from the environment. We did that for Heroku. We did that for Shopify 2.0 at places. But it doesn't work when you have something that's scalable system like Mesos. So you want something that knows about secrets in its container and it's not related to the environment at all. So we don't have to worry about states or making sure that secrets get onto the environment before they actually start running. What we ended up doing was building an open source tool called EJSON. You can look it up on Shopify's repository or Shopify organization on GitHub. What it does is it stores all the secrets as JSON files and that file is actually called .eJSON. And then before I commit, it encrypts all those keys and then pushes it to GitHub. So in your GitHub repository, you not only have the code, but you also store encrypted keys with it. And before launching, all the containers have access to the private key. Before launching, you would decrypt those keys and make sure that applications are able to use it. There's a middle step in the middle where we decouple private keys and private and public keys that you use to encrypt versus the private and public keys that are used to decrypt. So there's a middleman happening. I'm not going to talk in detail about that, but if you're interested in knowing, you could meet me after this. Yeah. Lastly, routing. Obviously, if you're running so many Docker containers and you're running all of them with different apps, Shopify, ThemeStore, you want to be able to route. So currently, this is how our overall infrastructure works, where you've got HAProxy and then at the nginx level, we update the routing information. How it gets updated is every time Docker is a new app coming in, you would get updated to Routify. Like you would notify Routify, which is an inbuilt app or a home-built app. And Routify will change the nginx configuration files based on whatever new information comes in. So if an app dies or it's no longer valid, it's going to get removed from the nginx configuration. And similarly, if a new one gets added, it's going to get added to the nginx configuration. Monitoring. So Mesos has a monitoring toolkit available to it using its API. It tells you details about what's running, where it's running at a higher level. Same goes for Marathon. So Marathon has a REST API, which will again allow you to get details about which apps it's running, what are its scaling, what's its current number. Docker, again, gives the same details. So we put all of that together in Borg UI, which is our internal UI toolkit, which tells you what containers are running on, what applications, what all is going to happen. And finally, logging. At an application level, this is an issue that we face that if a container gets deleted, the logs get deleted with it. So in order to solve this problem, we actually have a logging service or agent running outside of containers, and it can access all container files, so it will constantly index all those logs outside of the containers. If you're using Docker, something like this, you'll probably end up doing something very similar, either wherein you will log outside of Docker or you will be sending logging information to a third party in some form. What does it end up allowing us to do? We now create containers every time we merge to master. Same strategy. You change a code, it gets created to a new container. Once it deploys, we literally just click a button and it gets deployed. Routing information automatically gets routed or updated for IngenX, so you don't have to worry about what kind of app it gets. You will find out where it's running, and the Borg UI will tell you exactly the URL that you can go and access it. And finally, keep X containers alive, so your app automatically is highly available. Is this production ready? I'd say so. We don't really put it out there completely, but what we do is, while currently we babysit it, we put Borg out on production during the day when we are available and make sure it's running nicely. So far, it looks exactly the same as our production cluster that we use to actually serve it throughout the time. I don't see any difference in terms of what happens. And what it ended up leading to was this. We ended up borgifying, oh, we don't end up doing it yet, but that's the plan. We will end up borgifying all the apps and it will make all the developers happy. Super easy to do that. Ops happy because they're no more worried about production or provisioning servers and waking up in the middle of the night just because there's an app going down. And Shopify is happy, I believe, because we can grow really fast. It's always awesome. So what does the future look like? I think it's pretty awesome. I see CoreOS coming, so if any one of you is interested in the other options on Mesos, CoreOS and Fleet is the other option and they're really good at what they do so far. We're looking into testing it out on our infrastructure and seeing if it might give us some better features. Though CoreOS cannot handle Hadoop and stuff like that right now. The other option is actually get our... We also run a Hadoop cluster separately to join our Hadoop and our production cluster into one single cluster and have this massive, I don't even know, 3,000 core cluster which would allow us to just run our data across all of those nodes. And that's what the future looks like. Any questions so far? Yeah, I have a question here. So you spoke mostly... Right here. Sorry. You spoke mostly Mesos in context to Docker, but how does it work with other technologies like virtual machines, OpenStack or AWS? Can I install virtual machines instead of containers? Yep. So Mesos is pretty generic in term... Like the latest version of Mesos actually uses containers in itself instead of us employing containers, but you could theoretically use virtual machines or actually build a whole cluster on virtual machines. You could also do it on AWS. Really depends. There are scripts available which will help you deploy too. So it will help you set it up on AWS. I was going to do any demo on setting it up on Mesosphere which is an open source... Oh, sorry. Which is Mesos has a service platform. So it's pretty open. You could do it on whichever framework and in whichever way you want. It's not limited to just Docker or just the way we use it. But I would say that this is coming out to be a very standard way of using Mesos. And a follow-up question is, it seems like in order to use Mesos, the frameworks has to adapt to the way Mesos work. And Hadoop seems like a good fit for this. So people who worked on Mesos actually already have Hadoop, MPI, a couple of other bigger frameworks already ported over to... Not ported over, but have the Mesos part and put in there. One thing to note is they also built from ground up framework called Spark which is what we use at Shopify which is built on top of Mesos and use all of Mesos technology. But most of the things that are production ready right now are already transferred over to Mesos. Thank you. Okay, I have a question. So if I have a new type of job which I would like to run on the cluster, I would first of all deploy it on the sleeves and then the sleeve will communicate with the master and tell that, okay, I'm capable of running this type of job. Is that how it works? The Mesos master and the framework scheduler both usually run on the same box. But I would say the Mesos master is not aware of any task per se. So the scheduler is aware of the task and so is the executor. So when you deploy, you will be deploying to a slave which is the executor and you'll deploy to the master as the new updated code which is the new scheduler and the new scheduler will know of the new task. But the master will know which slaves are capable of running that task. In terms of capability, we're only talking resources. So processing IO, network, RAM, not the code itself. So the assumption is that every slave is capable of running every kind of job. So Mesos has the ability to filter out. So you could tell it, say for instance in Hadoop's case, you have some terabyte of data on some specific node of your server, right? And you obviously can't run it on all the nodes. So in Mesos, you can tell it that I have access to this data and the scheduler is able to then use that knowledge to realize whether it wants to accept that offer or not. Got it. Makes sense. So like the offer gives the detail of what all it's capable or what all information it has and the scheduler decides how it would use that information to make a decision. All right. Good. Thanks. Couple of questions. First, the resource allocation. Is it dynamic or you have to tell that system A, like program A will take two GB of RAM, program B will take 500 of them. That you have to declare beforehand or it can be dynamic that... It really depends on the framework. So it really depends on the framework. You could make it both dynamic or static on what you want. Like in terms of, as an example of HTFS, like the data that's available on a specific node is static. It's not going to change because that's terabyte of data and you wouldn't really be moving it over. So that data can go into Messos too. But you could also tell it that it can now use more RAM or more processing power and get more like run at a higher speed. So you could have a recommendation and like a minimum use case. So like it... Yeah. You could have... It could be both dynamic and static. And this... Like you have to configure this or Messos will... Messos will do it for you. So you don't have to worry about how it's going to get run. Okay. Secondly, you mentioned that Rails currently can't run directly on Messos. So is there an intermediate layer? Marathon. That's why I mentioned. Marathon is what we use. So Shopify is a huge Rails app. And we use Marathon because Marathon is the framework which allows you to execute any random command as a service. So in terms of Rails or any other web service that you want to run, you'll probably be using Marathon or Aurora, which is another framework on top of Messos to run those things. Okay. Thanks. So the Messos... You mentioned about three Masters. Yeah. And at any point of time there's one that's... Probably. Supermaster, let's say. Yeah, sure. Right. Is Messos smart enough within the Masters to identify a Supermaster that is stuck for some time? The Master keeps getting pulled for its availability by Zookeeper. So it's only the Zookeeper that can do that. Zookeeper is the reason... The reason Zookeeper exists in Messos is for that purpose to make sure that the fault... To provide the fault algorithms. So Master itself doesn't know about the other Masters. So they don't... How is the... I'm assuming there's some metadata around this, right? Yeah, so the metadata... How is that made? All three Masters give their information and their metadata to Zookeeper. Everything the Zookeeper here. And then Zookeeper decides randomly on one of the Masters and then it keeps a check on whether one of them is available or not. Okay. And whenever it goes down it will automatically switch. One thing to note though is if Master goes down it doesn't affect your slaves. So your app still keeps running and your Master gets re-elected and once it gets re-elected all your app, again, like all your slaves go back and notify the Masters again. Go like, I'm running this, I'm doing this job. So you kind of like never lose your state. Okay. Theoretically you could lose state like I can think of other ways where you can have situations where two things simultaneously go down and then you're screwed. But in most cases your whole state gets rebuilt and you will never lose data. Yeah. Makes sense. Thanks. Okay. I think we're good on time. Thanks, Yagnik, for a wonderful presentation. Round of applause. Thank you.