 So let's start. I'm Tomak, and I'm working at Allegro, which is one of the biggest e-commerce solution in Central Europe. We have been using Mesos and Marathon since 2015 on a production. And today, I want to share our insights of why you should consider writing your custom Mesos executor to obtain cloud-native application ecosystem. So if you are not familiar with a Mesos executor, I strongly encourage you to see the Mesoscon Asia talk by Vinod and Greg about deep dive for the Mesos executor. They're talking about more technical details of the Mesos executor and how they work and the authentication and stuff like this. Today, I will focus only on functionality without code. Just we will focus only what Mesos executor could give us in terms of building a better ecosystem for our applications. So just a quick recap to share the same standpoint. Executor is a process that is launched on agent in order to launch a task. It works for frameworks. It just get a task info or executor info, in fact, from Protobuf and start the task that is defined there. So I hope you are familiar with this image. This is the architecture of Mesos. And today, we will be talking about these guys. They are the custom executor for different frameworks like Hadoop, MPI, and what they are working. They are getting information from a framework that are passed by master and works on agents and starts the tasks. And executor is responsible for maintaining the whole task lifecycle. So there are four types of executor. There is a command executor that is probably the most known and used executor in the market. It's using the V0 API of Mesos. It's bind to libmesos native library. It has the capability of running only one task. And there is a Docker executor that is really similar to a command executor, but it's running Docker. There is a default executor that was introduced in Mesos 1.0. It's using V1 HTTP API, so there is no native binding. It has capability of running pods and multiple tasks. And what we are talking about today is a custom executor. So it can be written whenever you like. You can use deprecated API if you like, or I strongly encourage you to use an HTTP V1 API for writing your custom executor. To build a cloud native applications, we need some guides. At Allegro, we follow 12-factor apps. If you are not familiar, it's something like a manifest of how to write a cloud native application designed by Heroku. It consists of the 12 factors that need to be fulfilled in order to say that application is cloud native. And what does it mean to be cloud native in a Mesos ecosystem? It means it's not only bind to a Mesos. When the application is cloud native, it has a really small contract that need to be fulfilled in order to run that application in different schedulers. So if you have a cloud native application, there is no worry to switch between Marathon, Mesos, Kubernetes, whatever scheduler you like. There are no dependency in app, so there is no problem with starting it up or upgrading your ecosystem because application has a small contract that need to be fulfilled in order to application to run. So let's walk through these 12 factors. The code base application should be built from one single repository. So executor will not help us here. Dependencies, this means that applications should have all its dependencies in the application. So it should not rely on a system. So if your application has a dependency to Java, it probably should have embedded Java inside the application package. If it needs a Tomcat to run, the Tomcat should be there or any other HTTP server runtime, like Nginx or stuff like this. Everything should be in a single package that could be run just with a single command or in a container if you can't statically link your application. Configuration. Configuration should be stored in an environment variables. And this is the place when executor can help us. I will talk about it later. Bucking services, this means that if you need to some other services, for example, application require Tomcat to be run in, it should be embedded within an application. Built, released, and run, it's pretty simple. So the steps should be separated. Executor will not help us here. Processes, application should use one or more processes. And it depends on your workflow. There are some applications that are single-threaded. And in theory, executor could help us scale them by running another processes. But currently, marathon does not support changing the resources on the fly. Portminding, pretty easy. Applications should expose services. Some services bind it to a port. So it is easy to achieve in the method ecosystem. Concurrency, we are scaling up the process by adding more applications, more instances of the same application. Disposability, this is interesting. Our application should be available to start and kill at any time. And we should provide a graceful way of starting the application up and a graceful shutdown. Devpro priority, so your DevOps team should have the same environment or as close as possible in terms of configuration and use the tools. Logs, application should stream the logs to standard output. And it should not have dependency on some Kibana on EL cast stack to get to know where to send the logs just standard output or standard error. And ecosystem should take care of where the logs are putted. Admin processes, run management tasks is a one-off task, so executor is not helpful there. So to sum up, there are three factors that we can fix with a custom executor. Confix, disposability, and logs. Let's talk about configs. So the 12-up source configuration in environment variables. What does it mean? There shouldn't be any config files that are stored in a repository. There shouldn't be some magical endpoint to download configuration from. Application should start up with the environment filled with the proper configuration. How it's working at Allegro. We have a certificate authority, which is signing a certificate obtained by Messos agent. We have written a Messos hook that is generating certificates for an application before it starts. Then Messos agent starts an executor. Executor gets the certificate in the environment and call the configuration store to get some configuration. Configuration store keeps the encrypted version of the configuration. And it need to be authenticated and decrypt with a certificate obtained by certificate authority. So the Messos agent is a single point of trust because it has some token to talk with the certificate authority. And then executor take this configuration from the config store and fill up the environment variables for the task and task runs. And it just read the environment and set up accordingly to the configuration. Yesterday was a talk about secrets. And you may wonder why we haven't used the secrets. Because when we started about more than two years ago, there was not that feature in Messos. Secrets are available in, I think, 1.4. So it is pretty new feature. That's why we built something like this. And why we are not filling up the configuration in a hook, like a Messos hook? Because the certificate obtained from a certificate authority is used also to communicate between the applications. So if the application needs some stronger security due to our compliance, they authenticate using this certificate. OK, second thing, disposability. So the application can be started at any time. What does it mean? It should try to minimize startup time just in order to be able to scale fast or if there is an outage to quickly minimize the outage consequences. And it should support shutdown. When there is a sick term, applications should gracefully shut down. Graceful shutdown means that the application will try to finish all the transactions and operations that were started before, but do not accept any incoming traffic. So take a look at an application lifecycle. When the application is started, it's fetching its dependencies, like packages. It's done by Messos Fetcher. Then the health checks start checking application. And application is starting, booting up, like loading dependencies, and stuff like this. Then it has some time to warm up caches. And then it answered to a health check that, OK, I'm healthy. I can get incoming traffic. So what happens here? When application is healthy, we should plug it into our discovery service solution, into load balancers, external caches, like varnish, into monitoring, and stuff like this. So this is hard to obtain with events sent from a framework. Hm. At the end, when the application should be killed, it should get a sick term, finish all its jobs, like finish a database transaction, and pending processes. So when the application gets a sick term, it should be removed from the discovery service and every place that it was plugged during the startup. And then it moved to a killing state. And when everything goes down and the graceful timeout in a couple of seconds or minutes depends on application ends, it's killed with a kill minus 9. So sick kill if it doesn't stop properly. So why we can't rely on events delivered by frameworks? If you remember the message architecture, the communication between tasks and framework comes by the number of hoops. So when the instance is started and it's healthy, message executor is sending the event of task is running and it's healthy to a message agent. Messos agents send the information to Messos master. And Messos master informs the framework, hey, that instance is healthy. So there's a huge delay between when the application is healthy and when the framework gets that information. What's more, the framework will send an event when it gets that information. And it gives us our events delay. So the time difference between when frameworks send an event or created an event and when our config service discovery solution got it. So as you can see in the peak, it's about 30 milliseconds. And in our scale, that means that some people will get a blank page when it comes to Allegro. And it's not acceptable for us to have errors just by design. So that's why we moved to monitor application lifecycle in Messos executor. So here is a diagram of a task lifecycle in Messos. It's simplified. There are not all connections that should be here. And there are a couple of tasks of statuses missing. So I will describe what is here. The gray circles are managed by Messos. So you have no control in an executor over that statuses. The double circle are the terminal states. The task could not be recovered from that state and probably died. And the dashed circle are optional. For example, command executor does not send starting information, prior to 1.5, I believe. And so let's take a look how the lifecycle looks. So the first task is staging. At that moment, Messos is trying to find a place to set up the task. So there's no instance running. In fact, nothing happened. It could finish in a terminal state if that task has the wrong configuration. It could finish with error or it could be dropped. And then there is a starting phase. As you can see, in the starting phase, Messos executor is started. It registered to a Messos agent. And then it should start the task. So when task is in the starting phase, it's all dependencies. In our case, it's just starting. And then we start health checking that service. So when the service is started and the health checks are running, it moves to a running state. The running state could have set a healthy state. That's why the task could have multiple running statuses, one after another, because it could be healthy, unhealthy, and it works in a loop. And from the running state, when task is in the running state and for the first time is healthy, we register the task in Discovery Service. In our case, it's a console and also with some load balancer if it's exposed to the rest of the world. Then when task is going to kill, it should be moved to a killing state. So before sending a SIG term to an application, we remove the instance from Config Service Discovery and from load balancers. So in this state, application has some time to finish its job. And then it gets moved to be killed. It happens when it's like a regular upgrade. Failed if it was killed by a messes health check, or finished if it's just a simple finish. So why do we need a messes executor right here? Because you can say, OK, but default executor also has a property called graceful startup shutdown and grace period. And in fact, it's not working with the command executor. Why? Because the command executor started a command with a shell. So it started with SH minus C and then some command to run. So when you want to kill a task, messes executor will just kill that you have a handle to the shell. And when it sends a SIG term to a shell, shell immediately goes down. Then the messes executor captured the event, the SIG child signal and see, oh, that application is no longer running. But in fact, the application is running underneath, but the only shell goes down. So there is no real grace period in a command executor. Second thing that we find when working with messes health checks, in a messes UI, messes startup time was wrong. And we accidentally fixed it by introducing the starting phase. Default executor does not use a starting task status. So there is only a task running. And because of how messes is handling the same task statuses, sometimes application keeps the last health check timestamp as a task starts it. It was not the big issue, but it could be problematic when you go to UI and see that your task started like a couple of minutes ago while they're working for a week or something like this. OK, so here is an example of graceful shutdown of one of our biggest application that is handling the traffic from our customers. What you can see here, at 3.15, we enable executor. So this is the regular deployment. There was no graceful startup nor graceful shutdown at that time. As you can see, we have over thousands of errors. Then we do a couple of restarts. And you can see that with a graceful shutdown, the errors has reduced to less than 500 and, in fact, about 200. And in the last at 3.15, we disable executor. This means we disable the graceful shutdown of the application and do, again, the couple of restarts. And when you sum up that bars, it's returned to the previous state when there are thousands of errors during the deployment. Yeah, so that's how it helps us. Why there are errors, even when the application has a graceful shutdown, we suspect that the application needs some time to warm up and that was not implemented at that time. So I haven't got a fresh metrics for the applications because it's pretty expensive to perform that type of test to just restart applications. And some users may be complaining that they've got 500 on telegrow.pl. OK, logs. Application should write its logs to a standard output. So there should mean no log files, no need to manage log rotating and stuff like this. When we started, most of our application had a dependency to a Kibana and all of our application were sending logs directly to a Kibana. This means when we want to change the URL or anything related to a logs, every application in our ecosystem needs to be updated. For example, when we want to add some new field in metadata, each application in our ecosystem needs to be updated. And this is hard to obtain when you've got more than 100 services because you need to ask the personal people that are responsible for a given services, hey, could you update that dependency? They probably said, oh, no. We have some business to do. We don't have time to do it. So remove any dependency that is not needed from the application perspective. So how it's working? Executor launches tasks. And because it launches the task, it has control over the STD out and STDR of that task. So it can manage to send the logs to some central log store and also to write them to a file. Executor has all metadata about the task. It's running. So the whole executor info that contains command info, container info about the task. So there is no problem to add information for services so the Kibana can know in which bucket put the log. You may wonder why we haven't used container logger for this because, in fact, the container logger is an interface that has the same capability. You just implement one abstract class from the message. And it gets the same information as executor. So there is executor info. And you can write your logic what should happen with the logs. The problem with the container logger is that you need to write it in C++. And it's strongly related to a message. So whenever you want to update a message, you need to recompile your container logger and see and check if your dependencies has the same version as the message. And if everything was compiled and linked properly and everything is working. So that's why we prefer to use an executor because it has an HTTP API. And there is a guarantee that whenever we update the message, the executor will still work. So finally, who is using this approach? Because I've seen that the Marathon community is not using custom executors. There are some code related to a custom executor in Marathon. But it looks like it's not maintained from the couple of years. Like code is untouched from two or more years. And we have some troubles when it started using custom executor starting from the UI that was completely unaware of the executor thing and some configuration options in Marathon. So Aurora is the best example of a custom executor. I think it's used from the beginning, but I'm not sure. In Aurora, executor is called thermos. And the similar thing that we are doing, as Aurora does, is registering in our case, it's a console. So executor takes care of registering in the service discovery, performs self-checking, the same as our executor. And we do not have some custom DSL like Aurora. So it does not have this feature. And the second popular executor is a singularity executor. It supports custom fetcher. I don't know why they need it, but it uses some fancy S3 downloader. Probably something works better or faster. It supports log rotation. So it helps you manage the logs and stuff like this and support the graceful task killing. So two of the three factors from 12 Factors App are covered by a singularity executor. And the third, I have heard about this executor yesterday. So there are other companies that are investigating. In fact, I think this is OVH that are considering using a custom executor. This GoMessage executor is similar to our other executor built on Go, built on top of the MessageGo library. Internally, we are not using Docker so much, but this has support to the Docker and has a similar approach. So I think you should start using executor just. And here's a quick recap. Why we need it? Control over application lifecycle and environment without the need to query scheduler, muscles, waiting for events, and stuff like this. You are a master of your application, and you can control how it behaves. And when it's healthy or not, and when it's starting, killing, or anything. So it's a replacements for framework events. So this means your framework can do less because it will not have to send any events. And there is some space for work runs and hacks, as I showed you before, that Messos could have some bugs like this graceful shutdown, or not necessary bugs, but could not work in a way that you expect, and you can fix it with a Messos executor. So that's all. And I will be happy to take some questions. OK, no question. So thank you.