 So yeah, so my name's Kevin Cluz. My background, when I work at Mesosphere, so most of the stuff we're gonna be showing in this tutorial is around DCOS, not necessarily just Mesos and Meso specific. We're all employees of Mesosphere. When I first joined the company, I was working on the Mesos core subsystem, the Mesos containerizer. I added some GPU support to Mesos and things along those lines. What I'm gonna be talking about today though is the task exec support that I added. So the ability to sort of mimic the same functionality you get from something like Docker Attach or Docker Exec, but adding that capability to Mesos so you can go ahead and debug your applications once you launch them. So why don't I let these guys introduce themselves as well and then we'll jump into the presentation. Thank you. My name is Johannes. I'm also working for Mesosphere. I'm working in the marathon project. Have this be good. Yeah. Yeah. I'm really happy to show you today all about marathon and networking options you have when you use marathon and especially when you use DCOS on top. Hello, my name is Gaston. When I joined Mesosphere, I worked first on marathon and a bit of Kronos and nowadays I'm mostly focused on, I'm in the core team, so mostly on Mesos and before the executor has checks and stuff like that. And that's what I'm going to talk about. Hi everyone. My name's Phillip Norman. I work on the metrics project in DCOS and I also run the day two ops working group. I don't know if anyone's here from there, but if you are, hi. Okay, back to you, Kevin. I think the feedback was coming from that other microphone, so I'm going to see how I do with this one. So just a quick sort of intro before we jump into the main topics. As most of you probably know, deploying to production can be pretty hard when you're trying to run something on as sophisticated of a system as Mesos and all the other moving parts that come along with that, but it really doesn't have to be. And when you do finally get to the point of deploying something to production, if things go wrong, it's not always obvious to figure out how to deal with those problems, right? So our goal of this tutorial is really twofold. We want to teach you how to get your applications ready for production, how do you configure them so that you can actually deploy them on a Mesos DCOS cluster so that they can run in a production environment and then show you some of the best practices for debugging your applications when things inevitably go wrong during that process, right? So we want to move you from this point of things being hard to move you to the point of things being at least a little bit easier. Just a quick rundown of the different topics that we're gonna go through in this. So Johannes is gonna start off talking about container networking. How do you configure the different settings in your containers so that you can launch them and have the right networking between them for service discovery and things like this. Gaston's then gonna jump into some of the health check work that we've put together recently and including readiness checks to check the health of your applications to figure out if there's something wrong so that you can eventually then go in and try and debug that. Johannes is then gonna talk again about debugging deployments. So if you have problems when you actually go through the deployment phase of launching it by a marathon, what goes wrong, how can you figure out which wrong and then fix it. Then Philip's gonna jump in and talk about some of the best practices around using our metrics and logging services that we have in DCOS and how to actually get the most useful information from those logs and metrics in order to debug things that are going wrong and then I'll conclude with an overview of how to use this new attach exec support that we built in so that you can launch a remote shell inside a container, figure out things that are going wrong, debug that and then get the thing running again after you've diagnosed what was wrong in the first place. And then after that we'll jump into an interactive session with you guys. So we've set up a DCOS cluster. We've created a bunch of user accounts on that and all of the scripts and all of the configuration files that we put together that we'll be showing during the different parts of this talk will make available via gist so that you can try and deploy them and play around with them and use some of the debugging tools that we demonstrate to you. You can play around with that yourself, right? Okay, so here's Johannes. Yeah, so before we're talking about production we need to make sure that our services like knows each other. So when you're having a service which don't export a port or don't consume a port it's basically something that runs but don't have interaction with others. But usually you're building like a service-oriented architecture with some microservices and data services and you need to make sure that you have producers offering services and they need to define a name or whatever to expose those services. And on the other hand you have consumers who wants to talk to those services. And the first thing or what I saw when I was a user of these kind of services was that you have like hard coded dependencies or names where you want to connect to but when you have a highly distributed and highly dynamic system like DCS and all the services on top, you need to make sure that you are able to dynamically change those configurations and make it configurable from the outside. So again I'm from the marathon team so everything I show you today is basically this marathon JSON configuration. And a marathon for an application this will look like something like this. You have an environment property and you up definition and you say my database should be something like this. And you see that here's a symbolic name configured like database on some random port for the MySQL database. But how do we get there that we have those kind of symbolic names where it can connect to? So here using Meso's marathon and especially DCS you have a couple of options to solve this problem. So the first and probably all those in established solution was Meso's DNS. So when you launch a task in marathon you get a proper DNS name for the task and all running services for this. But this is done via surf records, so DNS surf records. So this is really easy for the up reduce of a service because you don't care about configuring something in your marathon applications. It's just there when Meso's DNS is running. But the downside is that only a few applications can handle those surf records and it's not really low balance. So you need to connect all those together and it's done by your own. And if we go to what is exposed, it's probably like this. So our application is service one. It's the app ID in marathon. And if you're using this you get the DNS name service one dot marathon dot mesos. And if you do a simple dig on this you get those one IP address back. But if you have two services on the same host name running you get only one entry. So you can do a dig for surf records and then you get additionally the host port of this. But again, so you don't have load balancing so no automatic load balancing on this and your most system don't understand surf records. So we can use a second version of this. So we can use virtual IPs and the underlying technology we use in DCS is Minuteman. So what do you do with DCS? You say on the producer side that you want to export the service like a host name and port combination. And Minuteman make sure that there's a little balancer in place and you can connect to those symbolic name and port combination. And you're not only restricted to symbolic names you could also use IP patterns for this. But the downside is you need to install Minuteman or like use DCS, so it's pre-packaged in DCS. If you're using DCS you don't worry about anything and you have some DNS suffix. So I show you what I mean with this. So on the producer side you go to your marathon app definition and configure a port mapping. You say you want to export, you contain a port 306 and you can define labels. So the first label VIP zero is a symbolic name. You can say I want to export database on the port and every traffic in this DCS cluster on this host name port combination is mapped to a load balanced version of those marathon applications. And the same is true for these IP patterns. So you can define a virtual IP but when you're using virtual IPs you need to talk to other people in your team making sure that this IP is not used by anyone else and stuff like this. So I would use like this symbolic version. And then as consumer you can use something like this. You can, if you're having like want to connect your database just configure your JDBC string as 3306 on port 3306 and you're done. If you have a database who supports load balancing. If not, you're only able to start one application basically. And on the top you see what I saw, what I named the DNS suffix. So if you're exposing the database symbolic name you get the suffix marathon.fob.dcds.directory. This is something you need to know but when you're used to it you can define between this virtual IP our services recovery option and the others I show you now. So the last one is virtual networks and in DCS we use nav stuff for this. And virtual networks basically is you configure that your container should join an overlay network and this container will join this network and will have every, I will have IP per container so a dedicated IP for this container and every port is exposed. So you don't need to do some magic port mappings or stuff like this. Everything is open for this. So if you have technologies who rely on starting a service on a certain port let's say 8080 for some traditional Java applications and you are not able to change this port for this application because the application is crashes otherwise. This is a good way to have an own IP for your servers exposing 8080 and you're done. And again, you need to have nav store or DCS and also have a different DNS suffix for this kind of things. Okay. This would be looked like in Merlform. So we'd configure an application with the ID database and say this container should have an IP address and it's coming from the overlay network DCS. And I want to have two instances now because I want to show you how this would look like when I do a dig on this. So if I now do a dig so now I have database.merlform.containerap.dcs. This DCS directory. So it gets a little bit longer now. You would get the following answer. You would get two answers for an A record, DNS A record again with the whole host name over there. It was too long for the slide so I added three dots. You would get database blah, blah, blah directory and then get two IPs for your two containers for this. Okay. So this is what we offer as like basic set for networking and services carrying options in DCS but you can bring your own technologies. Like our console is a great services carry technology and it's used by many companies and really popular. So there's a installation where you can install a console in your DCS cluster and use this for services carry. And if you don't like Nefster, you could also use we for works container networking solutions. So this is really open to like to go for your own networks and own services carry tools if you want to. Okay. And I think when we're talking about house checks and networking, I think the next Augusta I think you can just, I want to hand over to you now when we're talking about networks and productionally, you probably want to configure house checks to your applications and Gaston will tell you about. Yeah. So before I start talking about health checks, I would want to know a bit more about you. So can you please raise your hands if you have ever used mesos or marathon before? Okay. And how many of you have deployed anything in production or isn't this in production? Okay. It's about one third of the room or something like that. So whenever you deploy an application, it's pretty easy for mesos and for you to find out if the application is dead and if it crashes because then marathon will simply restarted or whatever skill are you using will restarted. But what happens if your application gets stuck in a loop or kind of reach a database or something like that and it doesn't exit, it will still be up and running. So mesos might think it's okay and it will not be restarted. And it will still not serve what you want. So you have to somehow detect that condition and react and probably restart the tasks. So how can you do that? How can you detect that? Well, that's where health checks come into play. If you use health checks, then either marathon or mesos depending on the kind of health checks that you're using will probe your application periodically your tasks. Could be either via HTTP or TCP or running commands in the same namespaces as your application or as your tasks. And then mesos or marathon will keep track of how many failures there were and if your task fails, the probes, a certain amount of times that you can define, then it will be killed and marathon will replace them or whatever other framework will replace them. So if your application crashes by itself whenever there's a failure and you are 100% sure that that's going to happen, then you need health checks. But normally that's not the case. Normally it can hang or maybe there's some connectivity or something like that and then you will want to use health checks. There's, when you define health check, you can define the protocol that's again HTTP, TCP or command. And depending on the protocol that you're going to use, then you can define the command to run or the port to connect to you or the path to use when doing an HTTP request. Something important to say that you have to decide when you're writing your app definition is how often your application will be probed. And you also define a timeout because it might be that your application hangs forever answering a request, so you only want to give it some time and then if that passes, you will consider that check failed. Another important thing is the grace period. Marathon and Mesos will not consider failing checks as failures during that grace period. That gives your application some time to initialize and listen on the port. It has to listen to stuff like that. So the first failures will be ignored until the application either succeeds a check or the grace period passes. And another very important one is the maximum consecutive failures that you want to allow. That's Marathon will keep track of how many times your application failed health checks and if it reaches that number, then that task will be killed and now we will replace it. So if you configure this on your application, then you're certain that it will not hang in a loop or something like that, it will be replaced and also the state of the health checks exposed through different APIs and then your load balancing solution can look into that and decide where it wants to start serving traffic or not. Besides health checks, there's something else that Marathon offers called readiness checks. It's easy to confuse what the differences are between those two, so I will explain again why we have that and how they work. Yeah, let's say that your task is up and running, but at the beginning it needs to warm up, it needs to load stuff, populate the cache or something like that. And your load balancer needs to know when it's ready or Marathon also needs to know when it's ready. Let's say you're upgrading an application, it needs to know when that task is up and running so you can kill one of the old ones. And how can you detect that? So you can use something very similar to what we did before, so use readiness checks. Again, your application should expose through an endpoint or you can run a command or something like that and Marathon will probe that endpoint. So in this case, I'm only talking about Marathon not Marathon and Mesos because Marathon right now doesn't support Mesos checks. That's something that was added recently. So yeah, when you're using readiness checks, it's all done just by Marathon, and Marathon will try to connect to your tasks and will evaluate the result, but if those checks fail, it won't kill the task. That's the main difference between a health check and a readiness check. If a health check fails a number of times, then Marathon will kill the task and replace it with something else. But if readiness checks are failing, Marathon will just not continue deploying your application. It will wait for the task to be ready and then continue. So if readiness checks are failing, you will observe your deployment that it will get stuck, but your task will not be killed. So what you can configure here is similar to what you can do with health checks. The same product was supported, HTTP, TCP and command. You can also define how often and set a timeout. The main difference is that there's no grace period because there's no... Like if it's failing at the beginning, Marathon will just wait longer. So there's no need to give it a grace period. And again, because it will not kill anything, we'll just only wait for the far successful probe. There's no need to set any maximum consecutive failures. Something that I forgot to say before is that Marathon will stop doing readiness checks as soon as the first one passes. So it's just there to have Marathon wait a bit longer and make sure that it doesn't kill your existing tasks before the new ones that it just launched are ready. I mentioned that these things can be done either by Mezos or by Marathon. So I want to go into some implementation details that might help you define what's better for you. So yeah, the task can be brought by a scaler that could be Marathon or Aurora or something else. So that means remotely. Or the Mezos building executors also support health checks and they can do the health checks locally and then send a frame on the results. And when you write your application, you can choose which approach you want to use. If you want to use, for example, HTTP checks, health checks, then you can use either HTTP as a protocol that means that Marathon will do the HTTP checks or you can use Mezos underscore HTTP. They work in the same way, but it's Mezos now doing the health checks locally on the agent and that has implications that we will see in the next slides. So there are some issues with checks originating from the scaler from Marathon. One of them is that, well, the API for health checks varies from one frame to the next, but that doesn't matter here because we're focusing mostly on Marathon so we can ignore that part. One of the main problems is that in Marathon, health checks are always performed by the leader, so you always have one node who's trying to connect to all your tasks. So if you have a big number of tasks, it will generate a lot of extra network traffic and that can be a bottleneck and also a reason of failure. Again, because Marathon will try to connect to all the tasks across your cluster, if there's any network failure between separating Marathon from your tasks, the health checks will fail, but that doesn't mean that your tasks are not healthy. It could be that there's a network partition or something like that that's harmless for users and only affects the connectivity within Marathon and the nodes. So it can give you false positives or what, negatives, unhealthy things. And we did some scale tests after I had support for mesos health checks and they scale much better than Marathon health checks because again, you can distribute the load horizontally across your agents. Each agent checks its own tasks and that scales better. So yeah, to solve those issues, we introduced mesos native health checks. If you use them, then the health checking is not centralized. It's done by every agent and that runs your tasks. It scales better. You can use command checks. Command checks always run on the agent anyways. There's no way of making Marathon run a command on the tasks, so that has always been delegated to the nodes. But in starting with mesos 1.3, also HTTP and TCP health checks run on the agents. And all built-in executors support all of these checks. The Docker executor, the regular mesos executor, or command executors is a name and also the new default executor is mostly used for bots. But most frameworks and workflows are migrating to it. There are some limitations with mesos health checks anyways. More than limitations like trade-offs. Since they run now on the agents, they consume more resources on the agents. So you need to prepare for that and reserve extra resources for them. Normally they should be lightweight, but in the case of command health checks, you can run any command. So if you run something very expensive, it will run in the same namespace as your tasks so it could potentially starve it. So you should be careful and either allocate more resources or use lightweight checks and lightweight commands. They always connect to local hosts in the case of HTTP and TCP checks. So your application must always listen to local hosts. The advantage is that they are not vulnerable to network glitches or stuff like that, but you have to make sure your application listens on that IP. Because mesos will fork processes to run those checks, it includes some overhead. So yeah, again, you have to plan for that and make sure that your agents have enough capacity for that. I want to make a short demo of how they work. I will just scream because I need both hands to type. They cannot use this microphone. Let me go to the last slide. So we have a cluster set up that you can use at the end to try these things out. I'm going to use the same cluster to do this demo. It's a DCOS cluster. I'm going to connect to it first. It's a live demo so things can go wrong, hopefully not. Connectivity seems to be quite slow. Okay, so I'm logged in as admin. What I'm going to do is I have a very small demo application that you can deploy on any cluster or even run it locally. It has a web server and exposes two endpoints, one for Hashtags, one for AINES checks, and we can see details about the requests to those things and we can also make them fail and then we can see how marathon and mesos react to those failures. I'm going to first deploy a basic version of that, achieves with the JSON ready. You will be able to use this again at the end and play with it a bit. So let me quickly deploy the basic application first. So it's a Docker container that listens on a fixed port and then that's exposed in a dynamic port in marathon. That's what this host port zero does. And we have marathon LB running on this cluster and marathon LB will let you get to this application through the public IP. I have to set here the port that it will listen on so that we can easily find it. So we're saying that it should listen on port 80 and I'm missing a closing, thank you, Johannes. So we're going to run this. It should deploy pretty quickly. It doesn't say it's healthy because we didn't add any Hashtags there. It only says that it's running. So now I'm going to connect to it. Let me get the IP of the publication. Okay, so this is the application. It's very simple. You can go to Hashtags and say that the next end should fail or not. And if you go to healthy endpoint, then you can see that someone did a request, which IP it came from and which result we returned. Yeah, so I'm going to use this and set it to fail the first end checks and then we can see how that affects marathon. Let's do this quickly. I'm going to use another definition that I have ready. Obviously, we can use the same one and just add Hashtags basically. So we're adding here some arguments that will make it fail the first three checks. We're going to use Mesos checks. So we will see that the checks are coming from local hosts from the agent. Here we set the path. It's healthy, such healthy. We're not going to give it any grace period. So if it fails from the beginning, then it will be killed. And we're going to make the checks every five seconds. So I expect this to, I expect marathon to kill the application after 15 seconds because we are telling it to fail the first three checks and we're not giving it any grace period. And let's see if it works. That's something you don't want in production. You don't want this to happen to you because this means that the deployment will never succeed. After 15 seconds, your task will be killed. So we can discuss, or I can show you how to fix that. So you can see that the task is running and now it's considered unhealthy and it will be killed soon and replaced by another one. We have to refresh it. So yeah, so this is not healthy because there's another application now. If you see all of them, one was killed, that was the failing one and it was replaced by another one. And this will happen all the time because we're not giving your application enough time or yeah, we're not waiting long enough for it to become healthy. So if we want to fix this situation, what we have to do is just increase the grace period. So in this case, we're making it fail only three times. So if we wait, and we're checking every five seconds, so if we wait at least 15 seconds, then your application will be healthy and then your deployment should succeed. So just to be sure, I will change this to 20 seconds, not 15 seconds, but 15 should be enough. Let's deploy this and then wait for the application to become healthy. Afterwards, you can play with the clusters and we can also play with right next checks and see how instead of killing the task, Marathon will just wait for it. That's something you can do by yourself. You can play with that later. So we can see it fail checks, but if we wait long enough, they will pass. So we can go to the application again. Let's close it. So first we have to wait until the application is deployed and the container is fetched and that stuff. And we can see it is now passing checks. So the first three failed, but it's now passing checks. And if we look now in the DCOS UI, we can see it's healthy and then the deployment has succeeded, the application is up and running. I could make it, well, yeah, we can wait 20 seconds. So I can make it fail the next five checks. And then in 15 seconds, this task should be gone, should be killed by Marathon. We can see how it went unhealthy. In the meantime, this could mean that your application in production, for example, is not responding to requests. So it will take a few seconds depending on your settings to detect that, but now you can see Marathon detected that and now it's running on our instance and waiting for it to get healthy. And that's pretty much, yeah, the demo. Oh, let's stay to DCOS UI maybe. Okay. So if you're at that point, you're really happy. So most of us in Marathon was able to start your applications, but sometimes you have a problem that you have a starving deployment and your applications never get launched. And in Marathon 1.4, we introduced a nice feature to make it a little bit more easy to you to debug these kind of situations. When you start a service and start a new one and say, we call it service one and we say a simple sleepy application and no Docker container, something involved, but you accidentally said that you want to have 10 CPUs for this application. So what happens? Marathon will wait until the Marathon 1.4 will receive an offer containing 10 CPUs, but it's really unlikely in this cluster that this will ever happen. So we see that we're waiting and we're waiting and we're like to Christmas for this. And what we introduced in Marathon, we introduced a REST API where you can inspect the current deployments. And when you have a DCS cluster, you can go to Marathon and go to service, service Marathon V2Q. Oh, thank you very much. Oh, it's missing an H, okay. So what you will see is that you have a pretty nice formative JSON file. And you see the processed offers Marathon has seen. So you see a summary of the last offers. So you see that you received six offers in the last time and one was declined because of an unmatching role. So in this cluster, we do have seven agents, six private ones and one public ones probably. No, five private ones and one public, so six in total. And you're constantly declining one because it's the public agent, the wrong role. And no offer was declined because of an unfulfilled constraint and so on and so forth. And you have constantly declined offers because the CPU count is not matching. So this is a summary of the last offer cycle of your cluster. And on the other side, you get an aggregated view on your launch attempts on your offers you saw. And like when I would refresh it now, the number of processed offers would be increased and you see that you in total saw 18 offers and you declined three because of the role and so on and you saw 15 offers and declined all 15 of them because of the unmatching CPU, configured CPUs. So if you see this, it's probably really easy for US developer or an administrator to like go into and see, oh, I need to fix the CPU count to get this application running. And the great thing is, oops, you don't need to, if you're using DCS, you don't need to go to the REST API to see this. There's a proper UI for this and also a good CLI interpretation of this. So when we go to the UI, you need to go to the debug tab and then you see these summary offers. You see the declination because of the role and really present the decline because of the CPUs and like this table showing all your nodes and you see that you have all the red crosses on the CPU column. So it's really obvious that the fix the CPU on this side and we go back to the slides. This also in the slides. Let's check. Yeah, this one. And if you would use the CLI, thank you very much. You could do DCS marathon debug list and then you get an overview of all your applications, Meritons currently trying to start an example that it's a CPU task going to launch one instance and you're waiting and you process six offers and didn't use anything of that. And if you go to details and then enter the app ID, you get the same table in CLI as you would see in the UI. Okay, so this is like, if you're having a stuck deployment and don't know why. Oh, we have a question. Okay, so the question was if we would add a CPU count to this table as an overview. GPU, yeah, GPUs. So, there was a GPU feature in the past. So if you're using the REST endpoint, there are GPUs listed as well, as well as if an offer is declined because you don't have the needed resource reservations for this if you're using local persistent volumes and stuff like this. I think it's, there's a JR ticket open to GPUs as well. So GPUs is fully supported. The calculation is everything in place. I think it's just not displayed currently in the CLI. Okay, thank you. Great, logging in metrics. All right, if you have any questions, just feel free to raise your hand and shoot. Okay, thank you. Cool, hey everybody. So yeah, metrics and logging. So the first kind of question is like, what are they for or what are they good for and what's the difference? Because they're both kind of like outputs from your application and they're both time stamps and they both provide some insight into how your application's performing. So I just wanted to call out a couple of differences before diving into how they might work. Logs are specific and metrics are generic. So a log might say something like a user logged in and this is their ID. Whereas a metric might be, for the same event might be this many users logged in in the last minute. Logs are generally text, metrics are generally numeric. So it's reasonable for me to go and read a log file but it's kind of unreasonable for me to go and read a whole heap of metrics. So I'm gonna want them pre-processed somehow first. And there is one particular thing that they have in common that's that you can monitor off them. You can monitor your application by looking at their outputs. So a metric might help you detect an outlier when your application's performing weirdly. You kind of hope that your metrics are always gonna be pretty dull. If you're doing a lot of batch processing you might see peaks, sustained peaks and then back down again and if you're running a social network you're gonna see like a spike at 11 AM and a spike at 4 PM because people don't like working. And it's when you see a deviation from that pattern that you know that you've got an exception emerging. And if you get smart with your error bounds you can have a situation where something develops and before anything catches fire at all you know that something's gonna go wrong and you can get in there and be fixing it before that happens. And so you can catch things like your disks are filling up with logs before your disks completely fill up and freeze everything up. And logs help you debug exceptions. So after something's occurred you can go in and look at a particular event. So in this case somebody from Chrome OS logged in and looked at groups in marathon and I know their IP address and I know exactly what they did to cause whatever happened next. But you have a problem. And that's that there are a lot of logs. This is the output. This is regular output from a node that's not doing all that much. Which I took this gift this morning. And so you're just looking at a lot of data when you're looking for your logs and picking exceptions out can be pretty hard. So we give you some tools to help a little bit in Maysource and DCOS. On the left we have the output from the CLI when you're calling out one particular instance saying okay what's happened with that. And on the right we have the same logs spooling through the UI. But so far what I've told you isn't particularly new. It's like probably familiar to everybody. And if you're familiar with the Maysource logs interface this is pretty much the same thing. So I wanted to speak to what DCOS can do a bit more. Maysource has no built in log DB. It just writes to file and pages through those like a regular application. And DCOS also has no log DB out of the box. But what it does have is the ability to install services like elastic cabana and log stash, which is the Elk stack. And easy integration with things like Splunk. And so that's kind of cool because once you've got your application piped out to Elk or Splunk or whatever you can start to write queries like this. So what we've done here that the highlighted bit is a framework ID. And we've searched for every log related to that framework. Now that's cool because that framework might have processes running on multiple nodes. And by searching the specific ID we can get those without having to pull a load of log files and do that thing manually. So much for logs. Metrics are important. They're the first thing you see when you log into DCOS. The dashboard shows you these graphs. But those graphs are allocation. That's what Maysource knows about what it's trying to do at this moment. So if we're going to be responsible cluster operators we need to look a little bit deeper than that. But there's loads of different types of metric. This is an attempt to explain the architecture of the metrics project itself and calling out the three different types. Which are metrics from the node, metrics from the containers running in Maysource and metrics from the apps running inside those. So this is Prometheus. We've just queried node CPU. It's right at the top of that slide, you can see. And this is a smallish cluster and these are the CPU utilization per node. At over the last, I think this is a two hour query. So again, so far so normal. This is something that you can probably get out of AWS or GCE or whatever you're using to deploy your machines. But we have more than that. We have container level metrics. These are kind of similar. They look at first glance kind of similar to node metrics because you're looking at the same kind of fields like CPU, like memory, like disk consumption, network. But they are slightly different because they are operating within constraints. So when you're looking at your node metrics you might say, okay, if my CPU rises above 80% of available resource then I probably got a problem or if my disk gets completely full then I've definitely got a problem. So you're gonna set some overhead in your constraints. Whereas when you're working with a container you've constrained it already. So if it's using 100% of CPU you're not necessarily locking anything up. Although you do still want to monitor this stuff because it's probably not great and you probably need to increase your allocation of resource there. And the last one is app level metrics. Now these are the metrics that your app emits from within its container. And what you're looking at here is many processes running HTTP server and this is the latency on the requests to those servers. So you can see all of them stacked up there and you can see a point at which latency suddenly rose and a point where it suddenly dropped. So alerting is the last thing I wanted to talk to because you're going to want to alert from your logs, from your metrics. They're both important and you can't really just pick one. If you start seeing many exceptions in your logs then you're gonna want to get paged. If you start seeing very high traffic from your metrics you wanna get paged. If you start seeing very low traffic you really, really, really wanna get paged because something's gone down. So I just wanted to say briefly there's this concept of white box metrics and black box metrics. A white box metric is when you have full visibility into your app and something like Cassandra for example will give you, will start to emit metrics about the state of the JVM it's running on. So that's kind of cool. I know exactly how much cheap space I'm currently consuming and that sort of thing. But I also want to know whether I can still write to it. So a black box metric is can I actually access the site? It's from the user's perspective. Kind of similar to what Gaston and Johannes are just talking about with health checks there. So yeah, do both. And that's sort of, oops, that was the wrong screen. That was, that's pretty much what I've got for you. That was a quick charge through. And that was a very screenshot heavy few slides but everything I've shown you there was deployed on a DCS cluster. And that's all like very, very achievable today. So yeah, I'll hand over to Kevin again. Oh, sorry, we have a question. Go ahead. The question was, is the Prometheus plugin baked into 110? No, it's not actually baked in. You have to deploy it on top. Go ahead. So the question was, if you're using the Docker agent and you deploy the Fluent D logger, does that prevent logs showing up in the UI? It shouldn't, no. Because as far as I know that does not divert them and they should still be, they should still be written out to, yeah, they should still be available. I believe. Anybody else? Or can I, shall I hand over? Okay, over to Kevin. All right, thanks Philip. So what I'm gonna talk about is the attach and exec support that we added into DCS. So it's basically mimicking the same kind of exec, attach and exec that you would get if you were trying to use the Docker CLI but now added into, for support with Mesos. So a quick overview of what we did to add the support. Like I said, it's added functionality to Mesos to enable building tools that mimic the functionality of Docker attach and Docker exec. So you can jump inside your containers, debug what's going wrong, and then come back out. Everything runs locally on your client. So with this support, there's no need to SSH onto the node where your task is running. You can just jump right into the container itself from your local laptop. We have a reference implementation of wrapping the APIs that we had to add to Mesos to build the support in the DCS CLI. So if you wanted to invoke this to actually take advantage of this new support, you would run something like DCS task exec, say whether you wanted it to be interactive, attach a TTY or not, pass the task ID, the command you wanna run, any of the arguments and so on. So we really tried to mimic the exact same command line that you would have from a Docker exec, but wrapping it for DCS. And then similarly for attach, which is implemented on the API side inside Mesos, but we haven't pushed that through to have it be supported in the CLI yet, but it is coming soon. And also native Meso CLI support for this is coming soon as well. We built the support into Mesos. It's obviously usable by a Meso CLI. We just haven't prioritized building that piece of it yet from the company's perspective. So what do the exec commands really do? What do they allow you to do? They basically, under the hood, if you look at the implementation details of this, we leveraged the nested container support that we added to Mesos. Nested containers is also the same technology that we built to allow us to support something like pods, similar to what you get from Kubernetes. We call them task groups. Kubernetes calls them pods, but under the hood in Mesos, they're all nested containers. The one difference or the one piece that we had to add to our existing nested container support was that we had to be able to tie the life cycle of the nested container to the life cycle of a connection. So whenever we do one of these task execs into the container, we launch a nested container inside of it with a brand new process. It runs until that connection is dropped and then that container gets killed and cleaned up and kind of disappears, right? When this process is launched inside there, we isolate the nested container inside the same set of C groups and namespaces as the parent container. So logically, all you're doing is launching a new process inside the container, but what you really need to do to make this robust and usable is launch a separate container that just happens to have all the same C groups and namespaces as the original container, right? And then once you have that, we just stream the input and output of the command back to the local terminal and we use HTTP. So it's all streaming HTTP. There's not a TCP connection that we set up, which makes it a lot more flexible to be able to use from a UI, from your CLI, from everything, because there's nice tools you can do just around the HTTP. So similarly for attach command. So if you think of exec as launching a new process inside your container, attach is, and then exec is launching a new process and then attaching to the standard end and standard out to stream that back. Attach is just getting to the standard end and standard out of your primary process inside your container, right? So once you launch that initial container, you just wanna be able to get at its output or feed something into its input, right? That's the purpose of something like attach. And currently, even with the support that we have for Mesos, we, and going forward, we're only gonna allow people to attach to containers that have been pre-launched with the TTY. So when you set up your container and you're about to launch it, one of the fields in your container config can be, oh, I want a TTY attached to this. And then under the hood, the containerizer will allocate a TTY for it, stream all of the input and output through that TTY so that you can come along later and attach to it. And the main reason we did this is that when I get into the implementation details of it in a second, we kind of how this is architected, but we really want these things to be able to, we want the containers themselves to survive agent crashes. We want everything to be really robust. So in order to implement the support for being able to attach to its input and output, we have to run a separate process alongside any of our tasks that we launch. And so we didn't wanna force every user that ever launches a task to have this separate sidecar process associated with it. So we decided to limit the scope of this to only if you want a TTY attached to it, where you get this extra process kind of launched alongside it so you can come and attach to it later. So this is kind of the architecture of what we did. So the boxes in blue are pieces that existed prior to this work and the orange ones are the things that we added. So if I start on the left here, you can see that the main thing we did is we added some new agent APIs. And I'll get into what the details of what those APIs are in a second. But the idea is that there are just some APIs that you can attach directly to an agent to stream input and output out of a container, right? We had to add a component for the handlers for those API calls, which now call into something that we created called the container IO switchboard, which it has a component that lives inside the agent, but its entire job is basically to both launch a new Mesos IO switchboard process and glue the standard in and standard out between the IO switchboard process and the task that's actually being launched because the IO switchboard is the thing that you're actually gonna connect to to make that connection with the input and output. And so there's basically just a bunch of file descriptor and pipes that are set up to make sure that the IO switchboard gets your input and output, make sure that you can feed that from the connection and then also write that to the application properly. Does that make sense? And what the API calls actually look like are this. So there's three calls. One of them is called launch nested container session. And the purpose of that is to, you pass it your task ID and you tell it what new command you wanna run inside that. It then launches the new nested container inside your container on the agent and it sits there and it's waiting for the next command. So the next command that you wanna be able to send it is container inputs. You can start streaming your input into that container. Now the third one we have is called attach container output. Nested launch nested container session actually overloads the attached container outputs. You only really need the attached container output call if you're doing an attach rather than an exec because what ends up happening for launch nested container session is you open a new connection into the agent. That connection stays alive forever and then the response that comes back is just a streaming output from the container that you've now launched. Whatever command you've launched, once that connection happens, you start getting an infinite stream of the output coming back. If you come along later and do an attached container input, now you have a persistent connection on the request where you're constantly streaming your input into the application. And then similarly with attached container output, it's the exact same thing going on in terms of streaming output that you get from launch nested container session except you don't actually launch a nested container. You just attach to the output of whatever the container primary process already was. So going along with all this, we have full integration with mesos built in, auth in, and auth c, so you can set up your ACLs to make sure that only certain users are able to access this functionality. It's also integrated with DCS's fine grained ACL support as well, so you can make sure that only limited users have the ability to exec into the processes that they own and the tasks that they launch themselves. And yeah, so that authorization is all set up and available if you decide to use it. So what I wanna do now is sort of demo how you can use this. And like we mentioned before, we have an interactive session coming at the end of this where you can kind of rerun through these exact set of steps yourself to play around with some of the support. But what I basically wanna show is how you can bring up a vanilla nginx Docker container, run it through marathon, make sure that nothing's serving on the port that's been allocated to nginx. So by default nginx is launched with port 80 setup. So I don't change anything about nginx, I just launch it, it tries to connect on port 80, but that's not the port that Mesos is allocated to it, right? So what I then do is say, okay, well I wanna try and figure out what's wrong. I start an interactive session with nginx's container and once I'm inside that container, I update the port that nginx listens on, I restart nginx, and then I should be able to go to the webpage and see that it's launched, right? Because the port is now actually, or nginx is now running with the port that Mesos is allocated to it. I pre-recorded this demo. I usually give a longer version of this talk. This is a very stripped down version of it and so this is kind of a clipped version of the video from the talk that I've given in the past. So there might be a few commands at the beginning that look confusing, because it's kind of going from the flow of the other steps that I usually give when I go through this. But yeah, I'll go ahead and start this. And the first thing that I'm gonna do is I'm gonna actually set up my cluster to point to a DCOS cluster and I'm gonna log into it. So by default, whenever you launch a DCOS cluster, the first user, the default user you get is called BootstrapUser. You type in the password for that and now you're logged in to that cluster so you can start deploying things on it and playing around with it. So once I'm logged in, next thing that I'm gonna do is I'm going to take an nginx container that I have. Well, I guess first I'm gonna show that there's no services running on this thing right now, right? This is a vanilla cluster, nothing's running on it. There's no services at all and now I'm gonna deploy this nginx container. So I have this nginx.json file that I've sort of pre-created for this. If you looked through the contents of it, let's see here, yeah, so I'm opening it up. You can see that I've got the name of it is gonna be called nginx demo. So when we go to the UI and look at this, you'll see that reflected in there. The container type is mesos, so I'm gonna use the mesos or Unified Containerizer to launch this. It's just the vanilla nginx Docker image that you can get from Docker Hub. The command I'm gonna launch, again just for demonstration purposes, is some long sleep command so that this thing will come up and stay running forever. And then I'm gonna give it 0.2 CPUs and I'm gonna run a single instance of this and then I want it to run on a public node so that I can actually access the IP address and get to the port from that on a public node rather than something private inside the cluster. I could have also set up Marathon LB, the same way Gaston showed earlier, but for the purposes of this demo, that wasn't really necessary. So yeah, so the next step is to basically take that JSON and deploy it on the cluster. So I've got this command here that you can execute from the DCOS CLI to go ahead and deploy that. So you see that the deployment's been created and I can go back to the UI and I see that it's running now, right? So in your next demo, you can click through it and you see that it's running and Gaston talked about the health checks. You can see that it's already healthy. Now what I can do, what I do now is I come in and I look at what the endpoint that it thinks it has for that cluster is, right? Or sorry, not for that cluster for this application. This is the endpoint on a private IP address on the port that's been allocated to it. So there's obviously two things wrong with this. One, it's a private IP. There's no way for me to be able to get to that from my local laptop. And the second one is that that's not port 80 and Ginex thinks that it needs to be running, or thinks that it's running on port 80 and that's not what's been allocated to it. So how can I deal with this? So first thing I wanna do is I wanna get the public IP of this node so that I can actually navigate to it, right? So we have a, because we're running on AWS, I'm gonna be able to hit the metadata server to figure out what my public IP is, right? This is a standard URL you can hit inside AWS on AWS machines and so you can get the public IP. So what happened here actually was I tried to run this, I tried to use this DCS task exec command that I've now added, but what ended up happening was curl wasn't installed on that box. So what do I do? How do I deal with this? So instead of trying to run that command directly, I'll now jump into a bash shell using task exec. I'll run apt-get update and I'll install curl, right? So you can, you know, kind of, you can just kind of show the power of being able to jump in to the containers the same way you would have with something like Docker exec. So I install curl. Once curl gets installed, I can now from this interactive session basically rerun the same command I tried to before to get the public IP address of this node. Now, you know, this is kind of contrived. You probably know the public IP address of your node out of band of this, but this is just kind of showing, you know, how you can use these tools to go and figure this out if you need to. So I do that. I get my public IP. Everything's great. I try and go back to my UI to replace the private IP with the public one. And obviously it still doesn't work because the port's wrong, right? So how do I do with that? How do I go in and actually get this application up and running so that it's actually, you know, running on that port? So if I go back to my terminal and I go back to the list of instructions, you can see that, so, you know, I'm still inside the bash session, so I don't need to restart it. But so basically what I'm gonna do is I'm gonna try to open up the nginx's config file. Again, Vim doesn't happen to be installed, so I need to install Vim so I can open the file. Once Vim's installed, I'm gonna open up nginx's default.conf file that should be coming in a second here once the Vim install is done. So I open up default.conf. I see that it's listening on port 80, as expected, and I'm gonna change that now to the IP, or sorry, to the port that I've actually allocated to this application, 83.16. There it is. Now I close that out. I come back, I restart nginx, and I would say, let's hope the demo gods let's this work, but because I recorded it, I know it works. And when you refresh this page, we should see the welcome to nginx and come up. So there it is, and that's it for the demo. And yeah, so let's see what else I have in the slides here. So yeah, real quick, I just wanted to give special thanks to a lot of the people that helped us work on these various features that we've kind of demonstrated today. There's pieces that are involved with security to make sure that all the authorization stuff happens properly. There's people that put work in so getting the different HTTP APIs for Task Exec put in there, I'm sure. On the marathon side, there was a lot of people involved, more than Johannes, for getting some of that stuff working and so on, and also on the CLI bits that we had to put together. So yeah, with that, does anyone have any questions about this or anything else? And right after this, we'll jump into the live interactive session with you guys so you can play around with some of these things. Yep, so the question was, can you prohibit someone from exacting into the container that they themselves have deployed? You could, if you want to. Yeah, it's very fine grained ACLs. You have to give them explicit permissions for hitting those APIs that we've added. So if they don't have permission to hit the Launch Sensor Container Session API, they won't be able to do it, even though they have to create permissions for the container to begin with. Any other questions? Okay, so we'll leave this slide up, but basically we have a DCS cluster up and running that you guys can navigate to and play around with. The public agent that we have, so I think Astan mentioned the layout of this cluster. We've got five agents that are private and one public agent, and we're running Marathon LB on that public agent. If you wanted to deploy that IngenX container that I showed you before, you can deploy that on this public agent. It'll happen automatically by putting the slave public constraint, but that's where it'll land in case you need the IP address. We have a set of user accounts that we've created underneath the students group. The username for that is user one through 40. I don't know the best way to allocate that out to you guys, but you can try and log in with one and see if you're unique on there, and if you don't care about other people messing with your environment, then you can switch to another one. And then we have a gist with all of the scripts and JSON for all the different things that we showed during this presentation so you can deploy those yourself and see how all this works. So yeah, that's it. There's no other questions. Me, Philip, Gaston and Johannes will be around for any questions that you have as you're going through this stuff yourself. So thanks again.