 Alright, so welcome everyone. My name is David Duncan. I'm the open source Linux lead for Solutions Architects at Amazon in the Amazon Web Services group. I want to talk to you today a little bit about running the Amazon EC2 container service and specifically running that with Fedora Atomic. So we'll talk a little bit about just cluster management in general, scheduling a little bit, running the services. Then I'll walk through adding the ECS agent to Fedora Atomic and that should get you up to the point that you can actually start working with a cluster. So I'm not going to go into a lot of the detail around containers and the container building itself because I'm assuming that almost everyone in here is pretty comfortable with that side of the business will stick to what's happening on the AWS side and lower that barrier to entry. So why containers? It's easy for microservices. This is sort of a natural platform for microservices. One of the things that happens in the public cloud obviously is that a lot of the standard applications and some of the simpler processes are abstracted away and it makes it very easy to work with for a lot of people who are looking for a public cloud process. So just building the containers obviously is fairly straightforward and that part we don't like I said I'm not going to emphasize today but what is hard is getting them scheduled and this is a challenge to a lot of businesses that come to us and they want to know how they're going to handle that. We want to make sure that you have sort of an intelligent way to manage those containers, figure out what instances you have available, etc. So we start with the EC2 instance. The EC2 instance itself is going to be the base for what you're running Docker on. Inside of the Docker we're going to separate out the specific containers and groups of containers and groups of storage into what we call a task definition. So when you run a task definition that is basically what's running as a solid group for container management. So we're using this to basically track just the CPU memory and networking resources that are available for containers. So the scheduler is responsible for the tasks and their execution. Once we have the task defined we'll use a scheduler, we'll use a couple schedulers to manage those and place them in exactly where you want. So we have a cluster management engine that underlines the service. So there's no requirement for you to actually touch that. You're just going to leverage it for utilization. So what we're going to talk about ultimately is the ECS agent. And the ECS agent is what's running on each one of the individual container instances. So what you would refer to as a standard instance, in this case running the ECS agent becomes the container instance. There is a agent communication service and an API that's available here that's going to talk directly to the ECS agent. So the ECS agent actually wraps a lot of the Docker commands so that they have almost native access to the EC2 container service API. So basically what we do in order to coordinate this is we provide a key value store underneath. This is kind of the heart of the process and keeps the cluster state available across all of the container instances. So maintaining that at scale is a really big deal. So we're going to take a look at exactly how that we maintain that. So this is sort of a description of a fairly simple transactional algorithm. But what happens in the key value store is we're storing only writes that are handled after the last read. So if your system, well, if we read a key value, right, a key value pair, then that key value pair becomes the snapshot that we base our next write on. So you never get out of order. So if you have multiple clients and they're all doing writes and they're all doing reads, you actually end up in positions where if I have a system that's read at, say, N plus two tries to write at N plus three, but I have another scheduler writes a transactional event that occurs at N plus five, then the only place that that will actually be available to write is at the N plus six step. So this allows us to have sort of a combination of events occurring from different locations, all talking to the same API. That API then goes back to the cluster manager and ensures that everything stays same. So looking at it next in action here, we have the API, the agent communication service, all talking to the cluster management engine, also all sanity checking done by key value store. And this is done across a range of availability zones. And so you can actually you can maintain an entire cluster across span of high latency communication data centers, single region, single region. Yeah, it's always single region. There are ways of spanning it. But when you define a cluster, you're always defining it in a specific region. And that's by design. So one of the principles of the AWS configuration is that every every region is independently active. So if you want a service that is multi region. So, so your if you want a service that's multi region, we tend to suggest that you actually create a system that is singularly regional, right? So it's able to across one single region, but then make those latency based, make latency based communications across across the for clients, so that if you have clients who are who are hitting systems, they're actually hitting what's what's most local to them. You can also create restrictions based on what what geographic locations you have. If that is critical to your, to your business case. Okay, so with the API, you can choose whatever scheduler it is that you want, want to use. If you have your own scheduler, you can actually leverage this API. And this is all open source. But I'll show you where it is on GitHub. To actually make make requests for CP for resources, and then run those tasks or run tasks, accordingly tasks, of course, being combinations of containers and storage. So here I'm just showing multiple schedulers, right? So you have multiple schedulers, multiple resources. You can schedule a task by any one of these. You know, this could be this could be a long running service application task scheduler. This one could be for batch operations. Each one of them could be working at vastly different rates. And the cluster management and scheduler is going to keep that in check, right? So your cluster management is always going to look back to the key value store, verify that these resources have not already been allocated or they've been freed prior to the actual deployment. And then your scheduler is free to work as it chooses. So this is just demonstrating that same thing we had in the algorithm review there that if if our if scheduler yellow is trying to access resources that are already being provided from scheduler blue, then there's there's always going to be a prevention for that occurring. So basically, we get a full scale shared state system that you can provide your own scheduler for. You can allocate resources as you see necessary. You have your instances, the container instances that you are actually providing all of these resources can can then be auto scaled. So you don't necessarily have a single group of container instances that are associated with your cluster resources that can literally scale up and scale down. All of your central control and monitoring is happening through the cluster scheduler or through the cluster management system. And so the scheduler can function independently of that and just provide information back and forth. So these are some of the scale out numbers that we that we've we've actually pushed from our own cluster. So looking at this, you can see that we can scale up fairly quickly. A number of nodes where, you know, sort of a singularly allocated system would provide you sort of a flat number. Just reiterate that, you know, with the schedulers, we have two to two specific schedulers that are there by default. One is for long running services. So you're going to actually have a service definition that's associated with that. And then the other is just for batch jobs. So if you want to run a single job, that's just going to take resources up for for some period of time, like something that's related to clean up for after a backup process. Then there's a scheduler that actually handles that those resources as well. Obligatory slide. There's all sorts of other services out there that this touches and makes it really easy to integrate those back into whatever it is that you're working on. So if you do want to pull that out. So the elastic load balancing is a really easy one to point out how that interfaces with this because each one of your containers is going to present either TCP or unit P from from this from the container instance, each one of those ports can then be attached to a single elastic load balancer. And you can actually scale out your service, whatever your task definition is, can actually scale that out behind a single load balancer. So then you're not just provisioning an instance, you're provisioning a container behind a load balancer. So the load balancer itself is just making communications with the single container that's that's driving whatever service it is that you're accessing from there. That service, then if it becomes non responsive, the container is the task itself is actually destroyed, not the container instance. If a container instance is destroyed, the tasks are then actually reattached to the ELB when they're when they're expanded, they're scaled out based on the scaling group. So this is where you go to see the open source side of this. The CLI here is completely open source. The ECS agent is completely available and I'm really looking forward to hearing your feedback and finding out how we can better serve the atomic community. So the other thing about having those as open source and makes it very easy to sort of pull them into a standard CICD process. If you have something that's already going with your Docker files through GitHub, whatever can actually roll that right into your process and have those instances just turn actually you can actually provision based upon your process and you can have multiple clusters that do different things. So if you had like a test versus a prod cluster, you could have one that was that had a scaling group that stayed at one container instance versus a production one which scaled out to a much larger audience. Okay, so now we'll look a little bit closer at what the task definition is. So like I said, a task is going to be something that's actually handled by a container. So if you're looking at how that your task is going to run, that's going to be fine. You're going to find that in the task definition. The task definition also includes storage. So you can actually identify how your storage is associated with this particular container and then your run command in that definition altogether will then define what resources will be allocated by the scheduler for any given instance of that task. Yes. And the storage can be backed by EBS or Elastic File System. Yeah, no S3. I mean, you can access S3 programmatically, but then that's not the same as having a block storage. Can someone bring their own? So like for example, somebody wanted to run Wester and you can do that too? Absolutely. So technically you could run SAF. I don't know why you can call it. I want to in this context, but you could do it. And Gluster is a great fit. So just looking in the graphic interface at what you're actually defining for each one of these individual tasks, you can see just the defining metadata, the port mappings, which are really critical. And then down here you're actually associating a specific number of CPU units. There are 1024 to start with, and you're basically going to divide that up accordingly as you see fit. So it can be a little bit more granular. You can actually make some stronger definitions around the environment and your Docker run commands. But that's typically not necessary, as long as you allocate what resources you need. If you've got a defined container with your standard commands in there, you'll get what you need with the run command. If you want to override your run, you can do that in the advanced configuration. And yeah, it comes out in JSON. You can define it that way too. Okay, so each one of these tasks is scheduled on to a container instance. And that is essentially what happens here. So a container instance itself isn't necessarily consumed by a single task, right? You've got 1024 units to divide up. But you can identify how many units you're going to associate with any given task. And then auto scale accordingly. So we're looking at it from that perspective, right? So a task actually defines a unit of work. This is going to be associated with specific containers one or more, and then the actual resources that you're expecting to associate with that particular target. Okay, so now here's what I said. For a long running application, you're going to create specifically a service. And the service is going to refer to something that you want to keep running at all times, right? So now we can associate that service then with a specific load balancer. And that load balancer then will scale out the back end to make sure that it's actually providing sufficient workload, right? So you can have a load balancer, auto scaling group, and that load balancer then will have whatever triggers you actually identify are going to make it possible for you to maintain the serviceability of this application, or this task group. It re-iterates exactly what I just said. So another thing is you can actually have a container instance configuration that you've defined, then you re-iterate that. This is kind of very helpful in the context of atomic, right? So we're actually going in with the OS tree and literally moving to a different version of the environment, right? So we can take a machine image that we've created for our new environment and add that to our stack. So now we have actually have sort of a standard kind of red light, green light deployment here because we've got multiple auto scaling groups. Each one of those auto scaling groups is associated with a particular version of the OS. We can drain connections out of the old systems. And once we have those completely drained, we can actually go back to a full, full new deployment. So now all of this is the new infrastructure. Nothing is left of the old infrastructure. It's all scaled down. None of the connections were actually dropped, right? Because the old infrastructure was there at the same time. We just removed the DNS entries that are associated with the older instances. So anything that's still running is still serviced. Once there's no service load on the old instances, they can just go away. That makes sense. Alright, so now we're just going to look at code. Just relax. So for grabbing the most recent atomic instance, I use the JMS path query that's associated with the AWS CLI. So grabbing minus one is the fastest and easiest way to get it if you sort by creation date. So once I have that machine imaged, I'm going to do a run instance command. The run instance command I'm going to associate with whatever size instance I think is representative of my test environment. So it could be just a T2 micro. But I generally choose a small because they're a small number. So I can choose a small or a large. It gives you a sustained bandwidth. And I prefer to have the sustained bandwidth or something measurable rather than having something that's just subject to steel time. How do they stop you as an Amazon employee from always grabbing the largest instance that you possibly want? I'm just curious. Where is it up to your personal discretion? Well, so it's yeah. So one of our basic principles. What's that? Oh, so how do they stop just an Amazon employee from grabbing the biggest instance they possibly get? Yeah. So why am I not running everything on a GP2 and an I2 instances? So frugality that's the Amazon value that I would say associates mostly with that. Don't want the phone call from Jeff. We see a big difference in GP2. Well, I have my own billing alerts set up so that I know what I am spending. So it certainly don't want to spend much time. So you can, if you want to, schedule it so that all your instances just are turned off at a specific time every day. That's totally possible. Not that I do that, but I do find myself occasionally on my phone looking and thinking, oh. Okay, so once I have that, once I have a running instance, I'm going to attach to that instance and just update it, right? So just pull, just update the OS tree. I'm going to go to latest. Then I'm going to grab the ECS agent from the Docker registry. Easy enough to do. Once I have that, now I can create the ECS Optimize on. You can see that command there, but maybe it's on the next one. No. Okay, so I used a very, like a couple of different directions with the CLI so you can see a couple things here. There's a create image call that I made that I don't think I actually recorded on this slide. So at this point, you would want to run a create image so that you have the current version of atomic, the current version of the ECS agent all together. So we'll call that one, our machine image is going to maintain that, right? So there's a couple of other commands that are associated there on the GitHub repo if you read the notes. There's a couple of directories that have to be created for logging in the config. But once you have those created, you want to create the image. So pull all the new, create the two directories that are necessary, and then run a create image for this particular instance. So actually here, when I did a run instance, I made sure to change my instance initiated shutdown behavior to stop instead of terminate so that in the event that something went wrong, I still have the instance there available for me for creating the image. Okay, so here I've actually generated a small CLI skeleton, and that's just a JSON structure for defining any of the things that are associated with a particular command. So any of the parameters that are associated can be collected into one of those skeletons, and then you can place that in a template. That template then can be used to run the command consistently, and you can just make updates to that template rather than actually having to type this out. And those work great with BOTO. So you can manage them programmatically after that. Once you have a skeleton or a CLI configuration that you've generated, you can then make modifications to that programmatically. Okay, so now using that in the CLI input JSON parameter, I've actually created a cluster. I've given this cluster a name of production, because that's what we all do. Okay, there we go. Yeah, so taking that instance that we actually updated, we're going to run this create image to get that image, the ID over there that's associated with the volume and the instance definition. So the great thing about this is if you decide to allocate more storage, like you add more devices into your configuration, then those devices will be recreated with each one of those images. So if you're using, let's say you've identified devs that you put into your Docker storage setup, and then you add like SDB, SDC, whatever, that configuration can remain. Each time you create an instance of this machine image, you actually end up with the same storage configuration. So you're keeping consistency with your own image. So I'm creating the image, or I'm creating an actual instance here. I could have done this through an auto scaling group, but I chose to do it as a single instance, because it's easier, more clear. So now I have one instance with the default configuration for this particular region. So I shell in, start the container, so do a Docker run on the actual configuration. Now, this is the important part. There is a connection to the Unix socket there, and that requires you to run privileged. So thanks, Dan Walsh, for fixing that, so that we could actually do it. This is not a solution. This is a workaround. But the dash dash privileged dash dash net equals host is a requirement for running the actual agent. If you don't have that in place, then the agent will just fail, and it'll fail repeatedly. And we won't have an attached instance. Once that ECS agent is running, it's associated with the, oh, I'm sorry, there's one more thing that's associated here. The ECS underscore cluster environment variable, there's a ton of environment variables in the read me for the actual agent. But this one in particular associates with a defined cluster. So if you have, you always have a default cluster. If you just start instances that are associated with the EC2 container service, then they will, by definition, associate with the default cluster, and that's what its name is. So if you just start up and run the ECS agent, you're running on the default cluster, you can actually define services that are associated with that. But if you do, like I did, increase your own cluster, then you need to actually specify in the run command what cluster they should attach to, what cluster of those instances should attach to. So here you see a specific container instance. This is really where we're ready to define our tasks, associate those with services, and then provide metrics for which they can scale. Any questions? So the way you walk through, look fairly manual, having the SSH into the thing and said everything, you can do that programatically. Yeah, so there's two ways to handle that. The first way, and the most standard way, the atomic way, is via the cloud-on-yet. So configuring the actual Docker command I expect to be done in cloud-on-yet. Yeah, the second way, and the way that I feel like is gaining traction, and I'm really excited about, is the simple services manager. So simple service management actually allows you to gain programmatic access to the instance without having to provide an SSH login. So it's done through the metadata. The ELB configuration, you said that you associate that with the service. It's running on Fedora atomic instances. So the service would be defined in a task, those tasks would be running on the Fedora atomic instances. So it scales the service, but does it also match the back-end Fedora atomic instances for scale? So the cluster would actually have its own CloudWatch metrics that were associated with that. So if you were running low on resources, you could actually scale the scaling group. So when you define instances, you can define instances one of two ways for the EC2 container service. You either define them individually, or you create an auto-scaling group that has a scaling configuration that includes that image type. Then you provide the, so the launch configuration has the instance and the associated only, and then you're, the auto-scaling group actually has the metrics on this. So there's just some coordination happening. Yes. It's also a cluster scheduler, open source, or is it like your... Yeah, that's the special, that's the abstracted part that's sort of the Cloud Service, right? Is the end, while the API itself is open and available for use, the key value store is the secret sauce. So you're presented running Fedora atomic host, yes, with your agent. So my question is, it's possible to do a hybrid docket cluster with your Amazon container service, because you're providing just containers, yes, in a maintenance, or I am in a standard? No, we provide Zen virtual machines. In EC2. In EC2. Yeah. Yeah, not strictly containers. Oh, okay. So your scheduler and key value store are... So the scheduler itself is not disclosed in itself for the service, the associated services, but the API is available to actually put whatever scheduler you want to put in place. So if you had something that was going to be unpacking, right, Mesos, right, you wanted to add that on top? Or even if... So I was going to say, like, what if I'm using Amazon along with something else and my environment's based on EC2, or sorry, ECB and Kubernetes, Zach and I, how do these play together? So they kind of play in the same space, right? So this is the... So without getting too much into comparison, what I will say is that when the EC2 container service was being developed, we had a lot of customers that we were talking to who were saying, okay, so we want to get into the container space, but we don't want to be in the business of managing our cluster, right? So this was a decided move to create an abstraction so that we could eliminate the requirement to do both. But provide an open API so that if you decided you wanted to do both, you would have the ability to actually provide back some numbers, right? So some details of the scheduling. Now, as far as the EC2 goes, there's already a key value store here. You could leverage that and wouldn't necessarily require it. Do you see people plug Kubernetes into your key value? Personally, I have not. But I'm looking forward to it. Yes? It's just a general observation. I'm not necessarily signing anything up for work, but I would be interested to speak to you. It would be interesting for me from an atomic perspective to help eliminate the stuff that you had to go through where the doctor pulled the latest version of your agent and then I have to save a new image. Okay, you're nodding as well. It would be nice if there was an AMI that was just atomic and we have ways of potentially caching those. That's fantastic. Yeah, that's great. And that's actually something that I was really... I mean, that's really why I wanted to talk about this is because I think there's a great opportunity here for some strong momentum in ECS-optimized... in an ECS-optimized image. We have that in a couple of different places, right? So the way that... So obviously there's a lot of strong tie in between ECS-optimized. Yeah, I know what I mean. Surprise. Thank you for an open source perspective. Yeah, exactly. And that's exactly where I'm coming from as well, right? Is that I want to see this strong adoption of fedora atomic in this space. We have people who remind us that every additional step to get on board, you know, you lose 90% of people. Precisely. And that's exactly what I want to do, is take this opportunity to provide people with a very simple level playing field for defining Docker-related tasks. How big... ...community behind the open source project? Or is it all... So obviously we maintain the repo, but pull requests are accepted. Yeah. Now, very happy to have strong participation. We have probably about 15 strong contributors right now, I think, from the outside, and then a dedicated team. Have you also tried this on stock fedora as well? Yes. So stock fedora was the first thing that I actually deployed EC2 container service with. And that was just super easy. Obviously, same problem exists, right? Which is that you have multiple steps that you have to go through in order to get there. And ultimately, I would like to see any CS optimized image. We would love to have somebody from Amazon participate in the blog work. I'm right here. I'm right here. We... ...access accepted, as they say. By the way, you have five minutes to take more questions. Excellent. Any general questions? I mean, while I'm standing up here. Why does the ECS agent run as a Docker image? And not just a regular... So it'll run both ways. And it could literally be included in the base OS. There is a configuration for that. It's just a go package, right? Right. So it lends itself to being run as a container. Because this is Fedora Atomic and not Fedora proper, right? The idea was to follow Colin's tenets, right? And ensure that it's running in the appropriate package model. But it's all open source. So we wanted to roll that into the image. Do you have any idea how much that would add size-wise? I think 11, maybe. So not inconsiderable, not demaxed. Yeah, not demaxed. So that's your garden stick? Yes. I'm just basing this on a container size off the top of my head. But it might be smaller if you've got all of your libraries in one place. That's a guess. Anything else? Cool. Thanks, guys.