 Okay, we're about to begin. It's good to see that the room is not, you know, overbooked, so no one's getting dragged out of the room. Welcome to Engine Yard's sponsored talk, Deep Dive into Docker Containers for Rails developers. So that's a mouthful, so let's take a look at the title. Deep Dive, this is me and my wife, scuba diving in the Philippines, where advanced open water is certified and it's beautiful underwater and when you go deeper, it's actually even more beautiful. So we're going to talk about Docker containers. Who among you have used Docker before? It's sort of good, it's more than half, but who among you has used a container but not Docker? Okay, we got one, two. Okay, so this is not an introduction to Docker talk, but we will look into container internals, right? What are containers made of? So, and then I have to be specific, this is where I have to make sure this is for Rails developers because when they announced Rails Conf will be in Phoenix, I was just thinking, oh no, a lot of Phoenix jokes, right? So you've probably heard a lot of these jokes already, right? So this is seven Phoenix, one of the organizers and the Phoenix framework. Some people have moved on to other languages or framework that's fine, but we're here to say that we use Rails and a lot of people still do. This is sponsored by EngineYard where I work for and we're celebrating our 10 years this year, so please join us tonight. There will be a party tonight at seven p.m., so please join us and we also have a booth tomorrow and on a Thursday. So EngineYard's a great place to run your Rails applications where you can easily scale from one to hundreds of servers, right? We have 10 years of Ruby and Rails optimizations on top of AWS and we have top notch 24-7 support. So let's get into the talk. These are the topics that we're going to talk about. The reasons for using containers, what are containers made of and how do you run containers in production? So there are a lot of uses for containers but here we're going to focus specifically about on deploying your Rails app in a container. I remember when I started Ruby back in 2006 or a few years after, one of the most popular deployment tool back then was Capistrana and it probably still is in some shape or form, we still use that the Capistrana way of doing things at EngineYard. We have deployed a lot of Rails applications using Capistrana, you know, big customers, big applications and it works and even still works until now but here I'm going to try to discuss why you should put your Rails app in a container. So with Capistrana, as a station to a server, if you're using Git, you're going to do a Git clone, Git pull, install the gems, pre-compile assets and maybe run migrations and it's fine, it works. We have big apps using that approach and it works but sometimes when GitHub goes down then no one would be able to deploy it, right? This is not enough on GitHub, we use them, it's a great service but when they go down, a lot of people notice, right? Because a lot of people use them so we get a lot of tickets actually when GitHub goes down, nothing's wrong with the EngineYard platform but when GitHub goes down, a lot of our customers can't deploy. That's only a small reason though why you should use a container but let's take a look at what's involved in using a container. Here you would see that you still need to install Ruby, install the packages, copy your code, install the gems, pre-compile assets, it's very similar to Capistrana, right? So you're not, it's not a silver bullet that would remove all these steps, right? You're still doing it but now you're putting it in a container and once you have that container, your server needs only to know how to run that container, right? It doesn't even know what's inside it, just run that container and you could run it with other containers. It could be another Rails app if you have another one, you could run it on the same server or it could even be something like Redis or a database, although our DBA is here and he wouldn't like that. You shouldn't run your database in a container but it is possible, right? Whatever you put inside it and your host knows how to run it then it should work. Then you could also have multiple servers. There's no real world analogy to this but you could duplicate a container easily. You could run it on multiple servers so now when you try to scale and you know that Rails can scale, right? You just run a lot of different servers and then on those servers you run your containers. So containers start faster and you'll be able to easily run any code that you could put in a container which makes the whole process faster, like your developers would be able to release code faster in staging or in production and you get to focus on your business problems but what are containers? There are a few descriptions that I keep on hearing when people discuss containers. Well, first is lightweight VMs and a lot of people don't like this description and because technically a container is not a virtual machine. When you have a virtual machine you could have a host for example that's using the surrounding Linux and you could have a virtual machine that is a Windows box, right? You could have a guest that is different from your host but with containers, when you have a Linux host you could only have Linux containers. There are Windows containers but we're not going to discuss them that's outside the scope of the talk so we're specifically looking at Linux containers but I like this description that it's a lightweight VM because of what I described earlier that in a container you could put everything on it. In fact, you need to put Ruby, you need to put your packages like if you have MySQL client libraries you need to put those inside your container. So for me it's a good description, it's a lightweight VM. Next is chroot on steroids. So chroot, if you have a directory for example you could make that your new root. You would still be using the same Linux kernel so that means it's technically one OS but if you have different sub-directories and you change your root into that you could do a lot of interesting things. Like let's take a look at this. So here I have an Ubuntu directory and let me just pause that and you could see that the directories on that Ubuntu 17.04 are just, you know, they're similar to what you can see in your Linux box, right? But here they're just sub-directories and you could run chroot, you could run chroot, so let's just run it again. So you have an Ubuntu directory, you could chroot into that and now you're inside a different OS, right? You think you're inside 17.04. So I'll check slash RailsConf, it exists on the host but not on the new root. So here I also have a CentOS 7 sub-directory and I could chroot into that and you would see that it's in its own, you could see the version of the OS but since it's a CentOS root I now have YUM inside it. So I have an Ubuntu box but I have YUM running. So it all shares the same Linux kernel but you could see that you could run whatever distro you want. So here at the end I just have another directory, Debian and you could see the version. So now I have one Ubuntu, I think it's a 16.04 LTS version but I've showed you three other distros that I could run by using chroot and chroot is one of the things that a container uses. You have file system isolation where you're inside it, you can't see anything outside of it. However, it's not built for isolation. So you could not see different files outside but you could see other processes as I will show you later on. But this was, this is a very old technology like released in 1982 and it was used mainly for testing or for building software where you don't want to use any dependencies. So it's like having your pristine OS inside your existing OS. So the third description is namespaces and c-groups and this is the meat of the topic and what containers really are, are namespaces and c-groups. These are kernel features. So if you've heard about namespaces and c-groups, namespaces, when your processes run inside a namespace, they think they're on their own system. They don't think that there's another system, they don't see the host, they see their own system. So a container, you could look at it as a different route, a namespace and a c-group. So there are tools to create namespaces but we'll look first at the higher level, higher level tools that create namespaces and these are the things that people are familiar with. Calling the container runtimes, LXC for example. It has been popular and it has existed before Docker, right? Docker at the beginning was using LXC to create a container. So it was just a wrapper, for sure it provides a lot of different advantages but at the beginning it was using LXC. Then you also have Rocket, SystemD and Spawn. But at the end you're just creating namespaces and c-groups. So none of these tools added new features to the kernel. They are using namespaces and c-groups. So when you're in a container, there's an illusion to the user that you are on a different OS. As I've showed you earlier, you think the process thinks it's in its own OS. So then that is the goal for the containers, right? So here we'll see the ch-root again. I'm using Ubuntu 17.04 and you would see that inside it, I could see all the different processes that are running. I just cleared the screen very quickly but I could grab for top, I could see that process inside that root and I could kill it, right? So if someone in the host was running top and I'm inside the new root and I killed it, then well, I'm sorry to that person running top. So what namespace does, right? Namespaces, what they do is provide you that isolation. So first let's look at the Pid namespace. So I'm going to introduce a tool called Unshare or a program called Unshare that would create the namespace. So I'm going to combine that with ch-root. So I'm going to say Unshare, make a new namespace for a Pid namespace, ch-root Ubuntu 17.04, I'm using the same thing. Going to mount the proc file system and after I run PS, you would see that I only see the batch process and the PS process. So now inside it, well, it thinks it's Pid number one. But in fact, it's not process number one in the host system. It's something else, so it's just mapped to something else but inside that namespace which we created using Unshare, it thinks it is Pid number one. So now you've created a namespace that can't kill the processes that are running on the host and why is this important? When people run containers that were created by someone else, you don't want that container to be able to go to the host and just kill any process, right? So next is the mount namespace. So when you create the mount namespace, you inherit all the mount points of the server, of the host, but then when you make changes to it, the host won't be affected. So why is that important? So when you create a new container or a new namespace, Docker, for example, changes the mount points for proc, season, dev, and so the containers won't have access to the hosts. For example, here, the container won't have access to the disk, why is that important? Well, if you have access to the disk, then you could corrupt it and every container running on that host would have a problem. So you don't want your containers to be able to access certain mount points and that's where the mount namespace would help. Another namespace that we'll look at is username space. And this is actually relatively new and even Docker only added this maybe a few years ago. So, but this is like PID mapping wherein when you're running inside a container, when you're running as a user on a container, you are actually a different user on the host. So it's like PID mapping. So a lot of containers run as root inside, you run as the root user inside a container and that could be a problem because when you're running as root without username space, you're also running as root on the host and you know why that's not good, right? Because if you have privileges on the host then you could do a lot of different things. So when you enable username space, you'll have root inside a container but you won't be root outside. So you won't be root on the host. Next is the network namespace and inside a container you will use your own network interfaces. So it won't have any connection but what Docker does, for example, is create vif pairs and use a bridge on the host. So now you have one pair on the container, one pair on the host and so you'll be able to have your network connection and we will show later on how that works. And there are seven namespaces right now. So we started with mount and the latest is the cgroup namespace and this is actually more than 10 years in the making, right? So mount was added at the root kernel 2.4 and user, for example, was added in 3.8 and cgroup recently was added on the 4.6 kernel. So it wasn't, there wasn't a just one time we're in, okay, we're releasing containers. They release namespaces and they release it incrementally. So let's take a look at how you're going to use everything, how you're going to combine everything to create your own container and run Rails inside it. So we're going back to our same example, unshare but now I'm just showing here that you have a typical Rails app on user source app. So we're going to create namespaces using unshare but now we're going to pass mount, UTS, IPC, Net, PID and run chroot so that it's what we've been running this whole talk and we're going to mount the proc and then next I'm going to add a lot of environment variables but these are just needed by my Rails app. Like they have database URL and secret key base. I'm going to create just so it's easier to see and now I'm going to run bundle exact Rails server to run my Rails app. So I'm now inside a container and running a Rails app. So I'm going to try to curl and see if I could access that and you would see that it would fail because I haven't set up the network with pairs that I mentioned, you would see here there's only one loopback interface. So now I have to create those with pairs, right? So I'm on the second tab on the host and I'm going to create the with pairs using the IP command, you just use H for the host HP ID and then C for the other pair. So now I have two pairs, I put the C one and put it on the pro society. So that's the container part. Then I put the H5140 on the Docker bridge that is running on the host. So now you would see that there are two network interfaces, right? So now I'm going to bring up those interfaces. So bring up the loopback interface. Going to bring up the other, the one pair, one end of the pair, name it if zero inside the container. Here I'm just going to add an IP address. Of course you want to be able to connect to your container using an IP from the bridge that I just chose randomly. And I'm going to add a route to be able to have connection routing it through the bridge and after that I would be able to curl the real stuff inside the container. But note that I'm using the local host, so 127001 inside, but outside of it in the host you need to use the IP address that I use. So here you would see that the Puma process is running. That's the default now with 5.1 and you would see that it's PID 9 inside the container but it's a different PID on the host. So this is the PID namespace at work. So next is C groups. So C groups are used to limit resources. Like you could set a limit, a memory limit, a CPU limit or even access to devices. You could also set a limit to the number of processes you can fork because you don't want to exhaust all the process, the number of process you could run. And these were, C groups were added on the 2.6 kernel. So let's take a look at how you're going to set a memory limit to that. So at the beginning it's just the same. We just create the namespaces. So we're doing the same thing at the beginning, creating the mount UTS namespaces. And then I'm going to mount the proc and then the environment variables that we also need. But before running Puma, we're going to use C groups to set up a memory limit. So here I'm using C groups. So here I'm using the CsFS C group memory which is the C group file system. It's already mounted on my box. I think it was done by system D. So unlike namespaces wherein you use unshare as the program that create namespaces with C groups, you actually just interface with a file system, with the C group file system. So I create the directory, create the Rails directory, and you would see that if you, I just created a directory but after creating it, it creates all these files for me. And those are the limits that I could use. You would see memory limit there and other things. So what I need to do now is get the process ID of my container. So I'll get the process ID of bash. So there's 1045A there and I'm going to put it inside Rails slash tasks. And tasks on C groups are the processes, right? So I'm saying process1045A should be under the Rails C group. So there's nothing special with this. I created the Rails C group. And I'm going to say 40 megabytes will go to Rails memory dot limit in bytes. So who wants to guess if that's enough for a Rails application? It's a very basic Rails application. So now I'm back to my container and I'm going to run Puma. So I'm going to run bundle.exe Rails server and it says it's killed, right? So with a limit of 40 megabytes, our Puma process can start. So now I'm going to increase that to 80 megabytes and let's see if it works. So this is a new Rails app, so I think this would work, right? So now you could run that process and you would see here that Puma is running. So that's how you use C groups with your Rails app. So next description and the last one and this is the most accurate description is containers or processes. So you might have heard of this. They're not VMs, they are processes and this is the correct description. And if you take away nothing else from the stock is you could run a lot of processes as you know but containers make it easier to run those processes together on the same host. So let's take a look at this next video. You can see that I have a lot of Puma processes, right? So and then I'm just showing you that the PID, I'm not sure if that's easy to read, but the PID namespaces, so you could check the namespaces on the proc file system, they're all different. So I'm just showing you that this processes are all in different namespaces, right? But they are namespaces and what's interesting is I have a lot of Puma processes running and I don't have even Ruby installed on the host, right? So your host doesn't need to have anything. In fact, there's an OS, core OS, or I think they've renamed it to container Linux, but it doesn't even have a package manager because they want you to run everything in containers. So here I'm trying, okay, run all the Puma processes you want. I think I'm using the same version so this is the same container, but you could run whatever Ruby version you want, whatever app server you want, you could mix and match Puma, Unicorn, and containers make it all easier to do all of that. So containers are processes, but containers being a new root, having namespace and cgroups, they're not actually enough. We have, whenever you create containers, you have to make sure you know how to secure them. So let's talk about container security. The way security works with containers is you apply layers of them. There's just no one setting that would make all your containers secure. Like you have to run a number of different things to make sure they are secure. For example, we have app armor, this Linux security module, or if your host doesn't support it, SCP Linux, and it limit the actions that a given program can take. So it provides a lot of limitations on the container, but actually if you start using the username spaces, a username space, some of these restrictions from app armor are not needed anymore, but you still keep them, so you just have another layer of security. So next is capabilities. In the beginning, there's only a root and non-root. If you're a regular user, you don't have access or you don't have privileges to do a lot of things. And Linux introduced capabilities so that a regular user would be able to do something if it has privileges. Some privileges, some capabilities, but not a full-fledged root user. So containers need some capabilities, but you don't want to give them all the capabilities. So that's why you also shouldn't run your containers as root. And while when limiting capabilities for some containers, then you limit what those containers can do. However, how do you know which capabilities to restrict containers and which capabilities not to restrict? In fact, when you search GitHub, for example, on Docker, there's a lot of discussion on what capabilities to allow or to deny. So there's no one answer. When you use LXC, they give you some set of capabilities and when you use Docker, they give you another set. So it's different. And the other is Seccom. So this is a Linux kernel feature and it filters system calls. And Docker, for example, disables 44 system calls out of 300 plus. Like one example of a system call it blocks is opened by handle add. Because when you use that, you could escape the container. So then the solution is just to disable that system call. But again, which settings should you block or should you disable? So those 44 system calls, how did they arrive at those lists? It comes from years of running the Docker project and to know which system calls to block. Like at a beginning, if there's a vulnerability, some system calls will have to be disabled. So the last part is running containers in production. So I've shown you namespaces and C-groups. So I hope I've convinced you to look at namespaces and C-groups or containers to run your Rails app. But I hope you don't go from this talk, creating namespaces and C-groups on your own, like running on share. Because most likely that would be not secured and will have a lot of bugs. For example, I've shown you CH-group, but that's not even actually what Docker is using. They're using pivot-root, which is more secure than CH-root because as I said, CH-root wasn't meant for isolation. So you don't write your own, it's like, I think it's like cryptography, right? You don't write your own, you let the pros do it. So you use a container runtime. I've shown you Docker and Rocket and that's actually good. If you're going to start running containers in production, that's the first step because they would create the namespaces, C-groups, and they would have default security for you. But then you'll also have other problems, right? What if the Docker daemon dies and, you know, I've had to restart Docker a lot of times and all your containers are gone? Like, well, what do you do with that? The site would be down and it would be bad. So you use something on top of it, you know, an orchestration system, and here you would have Kubernetes, messes, Docker, Swarm. You could choose, I mean, we like Kubernetes. When you run your containers, this system would choose to host with resources, right? So if you have 10 servers and you'd say, hey, I want to run this Rails app, then or this container with the Rails app and then Kubernetes would choose, okay, you run it on this host because it has memory. And then when a host dies, when a server reboots or becomes inaccessible, Kubernetes would then, oh, all the containers that are running on this host, I'm going to move them to another host. So with just the Docker, with just the container runtime, you'd have to manage that yourself, right? So that's why even Docker has Swarm because they know it's just running Docker on its own and one server is not enough. Then Kubernetes also provides your downtime deploy. If you have containers, then you want to be able to create new containers and with newer versions. But all of this, you still need an image, right? Which I didn't talk or gave the technical details. You still have to create that image, right? I tell you, install Ruby, install the packages, copy your code, install the gems, but how do you do that? And some people, they just don't want to do that. Of course you could automate this, right? A lot of you are using or more than half are using Docker or containers with Docker so you could use Docker build. And there's a lot of automation that you could use. You could tie them up with your CI, for example, and you could have an image. But what if you don't want to think about all this, right? That's like when you're a developer, you don't want to think about containers, C groups, namespaces. Then you could actually use a platform, right? There are a lot of open source projects for this, days, open shift, wherein you just, you don't need an image, you just push, right? You run a command like get push or the Cloud Foundry CF push and your app will be sent to the platform and it would run containers for you. But in that case, the containers are just implementation details, right? Like you don't care that they're running containers. I just care that it works and I just care that if I push my app, I would see the new version and scale automatically. And yeah, that is the goal. So you now know about namespaces and C groups but you don't even have to use them. And in fact, engineering, so this is just a plug, engineering has a platform that does that or we'll have a platform that does that and there would be an announcement though we have a keynote on Thursday that where you will hear more about it. So we work on that level. Actually, we had a workshop at Kubernetes so we could actually also work at the orchestration level but you know, most people would just like to push their app and just be done with it. So yeah, in closing, I mean, deploy your Rails app in a container. Looking to the technology, it's mature enough. A lot of people are using containers. It also has a long way to go, like databases. I think you should not run your databases yet in containers, it is possible but it's still early. And that's it, thank you.