 Yeah. All right, there's a bunch of seats up front that I happen to notice. And so if nobody moves from the back row up to the front row, I'm going to be forced to yell into the microphone, which might just be unpleasant for everyone. Any takers? Back row? Second or last row? This talk is going to be entertaining no matter what. So, oh, that's good. Okay. Thank you so much. Okay. Well, I think we'll just jump into it. And anyone that comes late is going to miss the awesome intro. So thank you for coming to most servers, no problems. My name is Nick Stilao. I work at Pantheon. And I firmly believe that the more servers you have, the more problems you have. And I'm going to try to convince you of that and talk a little bit about some alternatives and links containers and what they can do for you. So really, this talk is about containers versus VMs. But we're not going to worry about that too much right now. I'll convince you of that in a little bit. So we're just going to start off with kind of what kind of problems am I talking about? They're not kind of these existential worries kind of out there. I'm talking about real problems that are paying for site owners, I'm talking about downtime. And I'm particularly talking about problems for kind of operators, systems administrators, the people who might be getting alerts from such things as Pingdom, PedroDuty, Desk.com, New Relic, you name it, problems that are waking you up at night, bothering you on the weekends, and not keeping the beautiful sites that people are creating online. So really, this talk is going to be about how using containers and VMs can increase your uptime and decrease the number of problems you encounter, specifically those problems encounter human interaction, which are a special kind of annoying problem, which I'll get into shortly. I don't see Mark, but he wanted me to talk about the shadow cloud in this talk, and this is about as close as it's going to get. There's the clouds, and there are shadows under it, but I had to crop them out to make it pretty. So we can save shadow cloud for Q&A. And I want to remind us that we are living in the future, and there's a lot of cool stuff out there. Cloud's been around for a while, containers, like kind of big buzzword, and I want to convince you guys that this is something like we should all learn more about and figure out how it can serve ourselves and the people we work with and the site owners and everyone around us. And then lastly, I'll promise that we will get our hands a little bit dirty in this talk, digging into some kind of C-group stuff, lower level. Let's see, who's like a sysadmin or like an operator, or like a cloud person? Awesome, maybe half, maybe more. Who's not? What are you guys? Developer? Cool. So a lot of this is like on both sides of the fence, sysadmins or whoever, operators are often the ones dealing with this, but developers are often the ones that have the great ideas and just can't get it deployed out, so this should be some good background of what's going on. Okay, so up first, I'm going to really try my hardest to convince you guys that more serve as you have, the more problems you have. A lot of people, you kind of like get that nice new quad core or whatever, and it seems like this awesome thing, maybe by the end of this presentation, you'll kind of not be so excited the next time you get budget to buy some servers. So on a high level, most of the people in this room probably make money by managing sites. That's kind of everyone is operating, keeping the stuff online, that sort of thing. And so kind of graph this out with kind of a fake graph, show some trends. If you're not managing any sites, you have one problem which is that nobody's paying you any money, so you want to get rid of that. And then you start managing some sites and you get some problems, and you start managing some more sites and you get some more problems. And the more sites, keeping sites online isn't easy. There's a lot of things that can go wrong. There's about a million things that can go wrong even with a single Drupal site. The weirdest things can take stuff down. So you can't get rid of your problems entirely, but what you can do is kind of think about kind of methodologies and technologies and stuff like that that can let you grow the number of sites that you manage while having fewer problems. And I think there's no numbers on these axes, and I think it is relevant that certainly at scale, this becomes important, but I would argue that, and we'll argue that even at a small scale of one server or a few servers, all this stuff is equally applicable. And we want to be thinking about what happens when we start managing more sites. So the first thing, the first way I'm going to try to persuade you is to tell you a little bit of story. And this story took place over the past couple of years at Pantheon, where we have firsthand knowledge of kind of trying kind of a VM-based architecture and then kind of a new multi-tenant containerized architecture and where that's led us. So these guys have launched a lot of Drupal sites like many people in this room, hundreds, right? They've done the same thing over and over again and yet still kind of we see sites failing, going off that kind of thing. So around maybe late 2010, these guys had an idea, you know, let's get all this stuff that people love, get the whole kind of lamb stack going, get kind of Redis or Memcache in there, get some Jenkins to run some jobs on the side, that sort of thing, give it a little dashboard so you can click around and it's pretty, slap it on a VM and charge lots of money and get super rich. And well, except for that last part, we did it. We got 300 clients. We got one client per VM, so we had about 300 virtual machines. These were all in Rackspace and we had 300 problems. This was a nightmare to support. This was like doing OS upgrades were the worst, like dealing with Rackspace, like dealing with individual outages with Rackspace, trying to communicate status to customers when you're just pagers going off and that kind of stuff. Having Dev influence live, all these things. So we learned firsthand that that wasn't that great idea and sat down to noodle it over a little bit and had this awesome idea where, okay, let's take 300, like one gig VMs, kind of do some platform magic and turn it into maybe like 10, 30 gig VMs. The way some of these cloud things work, that actually cost the same. If the pricing is linear on RAM and we just need to kind of make this magic happen. So we kind of were figuring out what that magic was. So we got these kind of bigger servers, multi-tenant kind of containerized with solar on some and PHP on the others and Redis on the others and MariaDB on the others. The time was my sequel. And so we were kind of figuring out this out and it was going pretty well. So kind of thinking about this talk and thinking about this made-up graph that I made and I was thinking like, okay, let's do some math because I went to school and I did math in school and that's cool. So I'm like, okay, well, when do I get more problems? And I'm like, okay, so it correlates. I'm going to do linear correlation just to keep it easy. It correlates with a number of sites but there seems to be this rate of growth that I just coined that term, the PETA coefficient. So this is the pain in the ass coefficient but it can also be represented by a stack of PETAs or I chose Omicron because nothing out there uses Omicron to denote it. And then I realized why because that just looks like zero and it makes all your equations kind of confusing. But rest assured that is the PETA coefficient which is the problems per month that any one thing gives you. So it's nice and abstract which means maybe that means it can't be wrong. So yeah, let's all just take this for fact. PETA coefficient. So I was like writing some numbers for Pantheon and so I'm like, okay, how many problems a month? So I looked at all those services I mentioned earlier like desk.com tickets, Pingdom alerts, pager duty alerts, like all these kind of like automated systems and actually interacting with customers and humans and stuff and trying to figure out kind of exactly how much of a pain like these different things represent. And so came out with like via, you know, like per server, servers are like way bigger a pain than anything else per kind of active Drupal developer who's banging on stuff per Drupal user who's logging in and running crazy admin queries and stuff per container, per page view. And so like, oh, that's pretty clever. I kind of proved like I proved my point. How clever. But if anybody actually knows math or is just paying attention, that's kind of like tautologically true what I just like did there because I don't have that many servers relative to users. So of course they're going to represent more problems. So thinking about it a little bit more. So risk, you know, kind of can be defined as kind of the probability or likelihood of something happening times the impact or consequence of that thing. So if we're kind of talking about VMs and containers here, you can kind of look at, okay, what if we have two containers on one VM, right? We have half as many things that can break. So maybe that's half as much likelihood. But then if something does break, say the disk, we have double the consequences. And then conversely, if we had two VMs, right, risk, you might have twice as many disks, things that can break one half the consequences because only one site would be down or something like that. So I haven't really, so according to kind of this math, there's no real difference between these two. So I was thinking about a little bit more. So thinking about kind of the problems like you see when you're on call, right? So we're dealing with servers. Some things are kind of like self-healing problems. And these of course are my favorite kind of problems because you kind of just wake up in the morning and there might be an alert in your inbox or you might kind of see like a minute of downtime if there was some kind of flakiness or something, but you didn't have to wake up. You didn't have to like make any decisions, that kind of stuff. And then we start talking about problems that require some manual intervention, right? So this might be basic button pushing where you kind of have to go on, click a button, you know, it ought to resolve the issue. Then kind of maybe some real decisions have to be made. Sometimes you even find issues that require coding to solve or maybe like hard business decision trade-offs. Whether you should kind of like data integrity versus availability maybe like that, right? And if you're making those decisions on the fly, you're making human decisions and under more stress and more time constraint and stuff that you are, you're going to be making probably on average worse and worse decisions. And so if we go back to kind of the simple equation of risk equals likelihood times consequence, that really doesn't take into effect human decisions and human interactions during that process. Because each decision and each trade-off you're evaluating, especially when it's kind of on the fly and you're trying to get the server up and like figure out what is going on with like the disk or the network or whatever swap, you know, at any of these million things, the more likely you are to actually incur additional risks and make additional problems for yourself. So I think in that sense, going back to kind of these two equations, if any of the consequences involve human decisions, they are actually, they can have bigger consequences than they look like. So in that sense, kind of rolling with containers makes you make fewer of those hard human decisions, which means you're introducing fewer kind of human error and under-the-gun decisions. So another thing we can talk about is like network failure paths, right? So each server you bring online, if you're kind of in a cluster, that server needs to talk to every other server. Maybe you're on, you know, it depends on your infrastructure. You might have like an internal private network and you might have a public network, right, which compounds things even further, because both of those things can like fail separately. Maybe you're on hardware, maybe you're not, but maybe you're on the cloud. But in this case, because of that kind of N-squared growth of edges in a complete graph, if we go from five to six servers, we actually, we got one more server, but we got five more problems, five more network paths that can break and prevent your site from being online or your data from being backed up or replicated or any of those things. And like largely, like the network, like the network works most of the time, right? So there's this great blog post by a fear called the network is reliable and this thing kind of goes into case studies from many different data center providers showing that the network is indeed not reliable. And it might be reliable if you have a few VMs, like over some time, but if you look at any enterprise-grade data center on down to anything else, there are like network partitions are a very real thing that happen and so it makes this problem definitely a real thing that adding one server adds you a bunch more than one network pass that could fail and probably will fail at some point. So if you want fewer problems, there's a couple of different things you can do. Some people who are more like, Opsi might know these terms, which are pretty common, mean time between failure and mean time to resolution. So if you want fewer problems, there's a thing of some different ways you can do that, right? You can get more reliable things, right? Maybe we're talking about disks, maybe we're talking about network, maybe we're talking about any of those things and that one's kind of a toughy. I mean, you could probably pay more maybe and get more reliable stuff, but really I'm gonna let the network guy do the network and the disk manufacturers do the disk stuff and the real thing that I can control is to get fewer things, get fewer servers specifically. And then of course you want to decrease the time to resolution. So you can speed up detection in-site resolution. Some of the other great DevOps talks are gonna touch on some of this stuff for automated monitoring and telling when stuff is wrong and when performance does spike up, kind of trying to figure out what's going on, get that site back online quick. And then lastly, as we were just talking about, reduce the reliance on human decisions. Again, human decisions are gonna compound risk and introduce additional risks when you're just trying to fix something in the first place. Which brings me to Chief Chirpa. So Chief Chirpa was, well, he was the leader of the E-Walks on the forest moon Endor, but also he was the first, it was the name of the first server I worked on and deployed code to out of college, or I was in college, and I love Chief Chirpa and he was a great server. I'd push up my code. I kind of felt like he was one of my friends sometimes, but in the end, Chief Chirpa burned me, and so that was kind of one of the, one of the few long road of lessons that now I can say Chief Chirpa sucks and would rather drink beer with my friends and hang out with my real human friends and treat my servers like stuff like endpoint 9A71A1EF. So hopefully I just convince you a little bit that servers can power the internet, but they also can bring a lot of problems, especially for people that are tasked with making sure they're online performing and doing what they should be. So promise to get in containers a little bit, so I'm gonna go on from there. So who's heard of like containers and containerization and Linux and stuff, right? So maybe that's, yeah, Linux and stuff. Great, everybody. Yeah, so we talk about containers a bunch, but it took me like two years or, I don't know, something, and then I realized, like, oh, that's what you mean by containers? And so partly what I want to share with you guys is containers are kind of no big deal. It's what you can do with containers. So I want to talk a little bit pretty straight up about containers and what they are and how they can dispel any myths or everybody should hopefully leave the room knowing what containers are or having a better idea. So containers, yeah, it's a buzzword. It gets thrown around. It's kind of like no sequel, right? Like a couple of years ago and everything's no sequel. And then someone redefines no sequel. It's not only sequel, and that is kind of even more confusing. And so to me, containers are right in that buzzword territory of meaning something very specific, meaning something between a very specific thing and kind of nothing at all. So for example, at Pantheon, this is what our marketers say we built. So this is on our how it works page. And this is all factually correct, but at the bottom you'll see something that looks like a cross between like a nano-cat and kind of a squid. And then up top, I think what's going on there is that some HTTP requests are trying to inseminate a runtime agent. And we might have twins, I don't really know. There's a cache yet involved. So this stuff doesn't really help like disability miss about containers. This probably actually makes it worse. But kind of when you zoom up to 10,000 feet, everything kind of, you know, it's a little hard to see what's going on. So there'll be lots of stuff like this out there, but also hopefully more kind of hands-on stuff that gives a better idea of what's going on. So containers are very simply just resource constrained, system isolated, metered processes. I could even just say they're just processes. When people are confused about containers, I was looking at these water bottle containers, right? And nobody wonders what a water bottle container is. It's just a piece of thin plastic that holds stuff, right? And the only difference is before you had plastic water bottles, you were all wet when you tried to carry your drink somewhere. And that's basically what's going on here. Containers are like very obvious. It's what they enable you to do. Carry your water and your Gatorade with you around. So basically they're just processes with some constraints on them and isolated. One cool thing about containers, so this is some code we'll get into in a little bit, is that they're really fast to spin up, like a couple of milliseconds. So a lot of ways, if you're asking, which is common question, how are containers different from VMs? Well, they're not really all that different, except that one takes a few milliseconds, maybe seconds, something to spin up. The other might take, I don't know, minutes, right? So it's not that it's totally different, it's just that it's happening a thousand times faster, right? And that enables some cool use cases even that kind of VMs don't work quite so well on. Some of those use cases are very applicable, kind of people are doing kind of cutting edge stuff, and I think, but in general, just the speed of containers versus VMs is something that will enable a lot of great stuff. We'll see in the years to come. So yeah, some people, even if you, you know, I just run this one server, this like, I forget what this one is, two, E, Mac three, Macintosh, oh, the first Macintosh, and you know, you should probably upgrade your hardware because Drupal's not really going to scream on this thing, but even if you just say you have one server, I'd be willing to bet that you actually have more. I'm willing to bet that you have like a laptop that you work on, maybe a desktop, something like that, and then you have this one server in the cloud. So even if you think you have one, like kind of production server or something, maybe even if you're just one, you know, freelancer, you're willing to bet you actually have two servers and these portability wins are going to be good for you as much as some of the big scale thing. So like some other cool stuff that containers enable. So OS upgrades suck, right, when you have to kind of upgrade the kernel for security reasons, you know, you don't want to delay this stuff, you need to do it, but it's kind of a little scary to go on a running server, you're not really sure what's going to happen out of packages, different stuff. Oh yeah, cloud VMs kind of get weird sometimes, like you can't really, you don't have too much visibility into what's going on, so you're like, ah, maybe I just want to ditch that thing entirely. So when you're kind of operating with containers, you have a nice layer of abstraction between the server and between the container that's actually running your Drupal site or PHP processor or whatever it is. So you can just migrate those containers on the application layer to another server and then you can just kill the old server. And so this is one technique for not even dealing with in-place server upgrades, you just spin up new servers that have the new kernel, have the new OS distribution, whatever, migrate containers there and kill the old ones. So in this case, this is kind of a graph of after doing kind of wholesale migrations and new servers and deletions, kind of the average age of servers kind of drops, so they're nice and fresh and didn't have to deal with upgrading any of the old ones in place. So I've had experience doing this with Fedora 15, 16, 17, 18, 19 and now we're on 20. So I love this little logo here with the mustard is, I think his name is Beefy Miracle and he's the Fedora logo, or like the Fedora mascot maybe. But he says, end of life is a way of life and Fedora has one of the shortest timelines of any server and it kind of forces you to think through this stuff and embrace the portability and see what we can make happen. So another little anecdote is kind of like so the kind of MySQL was going up and kind of got bought by Son and then got bought by Oracle and then there's this like new fork and nobody knows what's going on for a bit and then sometimes someone's kind of like oh yeah, MariaDB's pretty cool, let's roll with it. So with the using containers, it's also in focusing on this portability and architecting for this portability. It makes it really easy to do stuff like migrate thousands of MySQL servers over to a new box and turn them onto MariaDB containers. So that was pretty cool and I love the new yak that he's roasting that dolphin in the MySQL logo. Cool, that was easy. So that's some things that containers enable to too, especially if you're thinking about the portability. So now we're going to kind of really, this is where like we roll up our sleeves a little bit and we're going to, probably you've got, some people probably know this really well but it's kind of weird and cool so we'll go into it a little bit but proc file system, look about C groups, talk about kernel namespaces. So on a technical level, containers are based, in the Linux world, containers are based on C groups and namespaces to piece of functionality in the Linux kernel. So these are, so we're going to go into each of these a little bit. C groups, look at how much fun these people are having with C groups. Everybody looks like they're having a pretty good time and C groups is another one that's really simple to think about. It's just a way of categorizing processes. So maybe one reasonable way would be to like have this root like okay all processes, and then on the, maybe I'll set this aside into my production processes and maybe my development processes and down there you have the actual stuff that you might be running, PHP, FEM, Drush, R-Sync, stuff like that. Maybe you want to kind of wait these a little bit, we'll get into that. And then like a slightly less reasonable more kind of bastard operator from hell thing would be like okay processes for people I like, yeah they get all the juice of course and then processes for the people I don't like not getting very much juice. But really it's just, and you can make these, C groups lets you make these hierarchies as arbitrarily complex or deep or wide or like tailored to your use case as you want, but really there's some kind of pretty easy kind of industry best standards that I'll talk about a little bit going forward. So a lot of people associate C groups with resource control right, but really C groups is just a way to categorize stuff, but then there are these C groups, sub modules and controllers. Oh yeah, so C groups, I have no idea what the capitalization is and it's sure for control groups, but it's kind of one of those things that's hard to Google for because well the capitalization is just all over the place and no one really calls it control groups and I think Google at least now like knows, doesn't try to auto correct you when you say C groups. So there are these sub modules right, for memory, maybe you want to limit the memory for one of those groups of processes that you defined earlier. Maybe you want to pick the CPUs if you have like a quad core, maybe you want one core for dev and the other three for production or something like that. You want to keep better accounting of what's going on there. Maybe you want some devices, physical devices to be accessible to this set of processes, not these processes. Limit the block IO, the kind of reading and writing you're doing from disks, stuff like that, tag network packets based on which process they're coming from, that sort of things. So even though this isn't core to C groups, this is generally what people are talking about when they say C groups. It's kind of this actual physical resource, CPU, memory, block IO, that kind of thing, that kind of those kind of constraints. So again, back to Linux and stuff, so there are these two kind of virtual file systems, PROC and like CISFS. So people have seen this kind of before, right? So these are some kind of funny things. They look like a file system, they act like a file system, but they're actually not really a file system in the way we commonly think of file systems. Really they're a way for interacting with the kernel. So in Linux we have the kernel and then we have user space, which is everything outside the kernel. So these file systems, which you can see on any Linux server, look like a file system, but they're actually a way of interacting with the kernel. And I'm not really sure why this happened. My theory, or I was talking about it earlier, and I think someone called it a philosophy, but I think it probably just happened because someone's like, how are we going to interact with the kernel? And someone else who probably wrote file systems was like, let's make a virtual file system. I think this might even work. But so stuff like configuring kind of your, how the kernel manages IP forwarding or something like that. This thing looks like a file, but actually reading and writing to it is not merely writing to a file on disk, it's actually interacting with the kernel and telling it to change the way that it's managing the network, in this case. So this is kind of a cool like proc file system hack. I don't know if you guys have seen it, but so say we're running Nginx, right? Web server, right? It has access logs, stuff like that. So if we look in the proc directory under the Nginx process ID and in this FD for file descriptors directory, we might see a handful of file descriptors, right? One might be to DevNull for doing, you know, black-holing some data. One might be a socket that Nginx is listening on or something like that. And one might be to an actual real file like the access log, right? So you can rmrf that access log and you don't see it in your shell anymore, right? If you ls, that access log is gone. But according, like, as far as that process knows and the kernels knows, that process is still holding on to that descriptor for that file and still writing to it. So even after you rmrf this file, you can kind of through the proc file system go in and actually tail it and read it and interact with it. So sometimes this can get you out of a jam if you accidentally remove something that's still being held onto some process. All this is to say proc sysfs are not your normal file system stuff. And we're going into this because cgroups relies on both of these file systems to interact with the kernel. So back to cgroups. We'll do something nice and easy to start. We'll make a new cgroup kind of under the memory tree and call it AA, which is pretty uninspired, but it makes it easy to fit on the slides. So we go in right, everyone can rock this on the command line, makedir, you know, created directory sysfscgroups memory AA. And now if we list that right after making it magically, there are all these other files in there. cgroup.clonechildrenmemory.kmem.usageinbytes memory, oh no, it wrapped, memory.limitsinbytes and stuff like that. So what we did there, it looked like we were making directory, but we were really interacting with the kernel and telling it to create a new cgroup called AA. So now, just like we kind of did for managing the network earlier, we're just going to echo 100 and put that into the file called memory.limitsinbytes in this AA cgroup. So that's pretty easy. We already made a cgroup and now we're kind of setting some constraints on physical resources on memory just with like a couple lines of bash that anyone could rock. So that's like a little, that's cool, but it's a little weird, right? So maybe we want to like use some tools that are a little higher level, right? Because that feels a little, always feels a little dangerous and weird to me. So libcgroups is a library for interacting with cgroups with the kernel and it comes with a couple of tools, a couple of helper tools. So we'll kind of do that same thing we just did. We'll just create, except for the CPU this time, we'll create the CPU AA cgroup. And then we'll set the cpu.shares value to 100. Sounds nice and arbitrary. We'll rock it. And then we'll use cgexec to run this little Python script while telling the Python script to run in the cgroup AA that we just made. So I have a bunch of cool examples online on GitHub, so we can get, I don't think we'll have time today, but we can get those together and they're just kind of fun ways to like play around with cgroups. So I thought this was kind of a funny way. So we'll create a memory, a cgroup called tnc in memory and then we'll use cgset and set the limit in bytes to 100 bytes for the tnc process. And then this is a little command line magic, but you're getting the dollar sign, dollar sign is getting the process ID of the current process, which is this bash shell that we're presumably in. And we'll, again, use just this kind of like redirection to write that process file to this tasks file and thereby associating the current shell we're in with the tnc memory thing. So what can you do with 100 bytes of memory? Pretty much nothing. So at this point, we've locked down the shell we're working in to 100 bytes of memory and if you try to run anything, it will kill your shell. So that's nice and fun because your shell dies and then you have to make a new one but you kind of get the point that the C-group memory limit kind of was acting upon your shell. So CPU shares, right? CPU shares is just a way for kind of prioritizing CPU resources. So maybe we make one CPU, C-group called AA, we tell it that it shares our 100 and another one that it shares our 10 and we kind of run a CPU intensive thing and kind of if you look at top, you'll see that one is using roughly 10x what the other is. And the limit in bytes is limit in bytes actually has units. CPU shares is different that it's just kind of relative. So these could be 1 and 10 or 10 and 100 or 3 and 30 and you'd get the kind of same output here. So as I mentioned, additional C-groups, kernel namespaces are the other critical part of containers. So all C-groups are categorizing and have these submodules for limiting the physical resources. Kernel namespaces are ways of in software isolating some processes for the other. So for example, IBC inter-process communication, if you really want secure isolation, you don't want one process to be able to kind of chat with the other process. So you can use this IBC namespace to isolate them. Network namespace, if you have again two processes that you want to kind of run on the same server, but you don't really want to talk together, you can use a network namespace. And that will essentially, as far as the kernel is concerned, as far as the process is concerned, launch those on two totally separate different networks. And there's another, a number of others here, the mount PID user UTS, but that's the basic gist of it. So as sharebear, sharebear looks pretty cute, but he's kind of a badass. And one time he said, you know, before you can share, you must first unshare, which I thought was pretty deep. Unshare is a system call and it's also a utility that you can run. And so this is just an example here. Unshare, so what unshare is doing is it's unsharing it from the rest of the processes. It's creating a namespace for whatever you run after it. So we're unsharing and we're saying specifically create a network namespace for bin bash. And so if you just run this command, you will be in a bash shell which cannot talk with any other network at that point. And so that's kind of the simplest way to create a network namespace for a process. So great. I'm glad we kind of got in there and dirty. And now we can kind of all really forget that knowledge because unless you want to explore and that kind of thing, I certainly don't really want to be managing kind of droopled, like back to droopled world and awesome sites world, that's kind of way too low level for the day to day. But I wanted to show you guys a little bit about what's going on, kind of, you know, lower level on the Linux side current sign. Well, there's got to be a better way to do this, right? There's got to be a better way of portability, the gains we want without really worrying about this stuff on a low level. And so kind of moving up the chain from some of the work we just did, we looked at kind of using the virtual file systems and then we kind of looked at some helper tools that were out there. This is kind of going up the stack a whole nother layer. So there's stuff like Docker which probably are in sort of OpenVZ, Libvert, let me containerize that for you. It's this thing from Google I'll talk about and then just a generic container with the Linux penguin on it. I think like probably everyone who's heard of Docker, probably, yeah. So Docker's awesome and I think it's pretty easy to tell like which one of these are going to be like, are well marketed because they have logos. So a lot of this stuff like the slides aren't very pretty because like no one, a lot of kernel hackers, not many logo designers which I guess is a fact of the world that helps projects get out there and get well known. So this is on GitHub. This kind of goes through kind of comparing some of these different tools that let you manage this stuff. So Docker is in there. Another one called SystemD, SystemD N-Spawn, LXC, Libvert LXC, LMCTFY which is let me containerize that for you or contain that for you which is Google's and OpenVZ. So all of these are a little different, right? They're probably coming from different places, like different, they might be just pure open source, they might be kind of spurred development at a single organization or like kind of pure open source like SystemD is kind of from the ground up like created. But basically they have kind of some different things and these might like socket activation is kind of a cool one where you can have a container that's using no resources and as soon as someone kind of wants to interact with that container maybe over a network socket or something like that it'll just spin up into existence and let you use it and kind of some other details here. So this is a cool GitHub repo that's kind of a community moderated matrix of kind of which one of these which one of these supports different features. So LXC, there is no logo. LXC is a wrapper around the things we're just talking about. See groups and namespaces. It is part of... LibLXC is kind of a wrapper around those things. It has binding so you can use it from different languages. A set of tools, decent docs on this as well as container templates. I'll save my recommendation for the end which is going to be Docker but I still want to kind of go through this stuff. I don't want to spoil it. So LemCutV definitely wins like they don't have a logo but like anything with Google you can probably get something with like these colored blocks so that counts for something although it's not a real logo and let me contain that for you it's like a pretty good name I'm not sure who came up with it or who let them come up with it but LemCutV is a little it's kind of fun to say as well. So it's created by Google it's kind of open source. I was hoping to get it to build on the demo vagrant box I have but it's a little bit of a pain to build and it's really cool Google's like putting resources into open sourcing it but I think it's not but they might have a hard time getting adoption because it is kind of has one foot firmly inside there inside their firewall but every single process that Google runs in a LemCutV container so that's pretty cool Google data centers full of servers servers churning away every single one of them is using LemCutV to containerize it and largely what this is doing at Google is trying to achieve 100% utilization so you have jobs queued up indefinitely and as soon as a server has a little bit of resources it will start working on a slow job it can get killed if someone types in Linux containers in Google and presses enter and that server needs to be used to serve that real-time request and that's all done through namespaces and stuff it's nested containers which is kind of cool so a lot of the C-groups and namespaces is about isolating the process but some of it is just for the processes sake like the PID namespace if you put something in the process identify our namespace it will think that it's the only process running on the box and I like to think that that is more for that process than the other processes and it just makes it feel good like this whole server I'm just going to do what I do but so that's LemCutV middle of the road log out so systemd nspawn nspawn is also a cool name I might be giving away some of my criteria for evaluating these things but there's also not a logo it comes out of the systemd project who knows about systemd yeah, some people so systemd is shipping with fedora it's going to get caught on to rel shuttle worth acknowledged it'll be part of Ubuntu coming up and systemd basically replaces sys5 and nit I don't know if that means anything but instead of kind of a nit scripts and PID files, systemd is a process manager that runs as the first process when your Linux server boots it helps you keep track of all this stuff all these processes see groups stuff like that it also has this systemd nspawn thing for creating kind of real real containers so this is going to ship with like fedora and rel and stuff and it will ship even if you guys don't know about it my colleague Joe Miller gave a cool talk sent to us dojo about kind of living the future because we've been on fedora on systemd for a while but it's definitely something to kind of look into because it it's a little different than nit scripts and stuff like that and we'll see more and more of it for sure so this is like one of the little demos so I was playing with vagrant in a fedora box I'm like, okay launch vagrant, that's cool virtual box, right, osx virtual box fedora thing like okay, dbootstrap it's a debian command that gives you a very minimal debian install and then you're like systemd nspawn that debian box and so I'm playing around with this and I'm like, oh my god I'm on osx and I have a fedora VM virtual box and then I made a container and it's just, no that's not enough we have to go deeper more levels and then I actually just stopped and kept, I just went to bed or something, but pretty cool right, you can kind of that's what containers do, they just let you go kind of deeper and deeper in those processes, might think that they're the only ones running on the box this is like just a little tidbit, pretty much the same thing but you can be like dash dash read only and then nobody can write to that file system I don't know, for some reason launching a whole container that can't write just kind of gets my I'm like, huh I wonder what I could do with that nobody could write anything they couldn't break anything sure you can't use it but it will never break and then docker so a lot of you have heard about docker it came out of a company called dotcloud and I kind of like this I kind of like this quote from a little apropos of the talk docker talking about in its early age, the dotcloud platform used plain LXE which we discussed, has no logo the platform evolved they were like patching the kernel kind of doing this different stuff and then eventually they're like, whoa this is like not this is really different than LXE this is totally different let's get an awesome logo and tell everyone about this there are a lot of people probably in this room I know more about docker and Drupal and how this can make your developers happy and your life happy and stuff and I think check out Ricardo Amaro's talk from Drupal Conprog I thought that was a pretty awesome one pretty good overlap with this talk but specifically some cool docker stuff so that's a good place to get started and ask around maybe we could offer something because I'm sure some people have been playing around I mean so dockcloud kind of platform as a service which may or may not still be around but now realize that oh my gosh people don't really need the platform they just need this wrapper around all these things they're not using proc and sysfs to do this cool stuff they can keep working on the important things so right there's a spectrum right we started off virtual file systems lemcuffy is kind of weird LXE is still pretty low level doesn't feel super helpful when you use it it got me to that inception kind of idea and I think docker is probably one of the best marketed, best prepared best documented, kind of like highest momentum ways that you can actually leverage this in your day to day whether you're a developer or an operator or a systems administrator or a small shop or a big shop or working open source or on private stuff pretty exciting stuff and then just to set up a talk for like next triple con once you get a bunch of containers running on a server you need to manage those containers across multiple servers for HA or even data centers or whatever and there's this cool project core OS that is kind of I think they have a product now but it's not really a product you can use I think they're really setting their sites once down the road when everyone's like yeah docker no sweat and now how do I use docker on like you know 100 machines at once but that's cool and I have good hopes for their success because of their logo and yeah because of their logo that's cool one to check out so no you know what else says no problem you know most there is no problems no problem says Alf so servers are great they solve problems they power the internet they also create problems containers really can kind of get this kind of agile portability whether you're working you know with a few servers or many servers or a few sites or many sites small teams big teams a lot of cool stuff going on there don't worry about containers you already know what they are you can tell everyone you know exactly what they are they're just isolated processes and on the technical level they're implemented with C groups and name spaces and you know please please use an awesome tool to manage this stuff otherwise you're probably just going to be typing on the command line a little too much and the future is now I think this stuff is really happening and I would encourage everyone and be excited to chat with everyone about what opportunities this kind of stuff opens up and where it's going to go next and kind of how you can leverage it to make your life better because technology just isn't about technology it's about how to make our lives better and I think these technologies can do that so let's go out there let's make some more Drupal let's keep some sites online and yeah thank you everyone for listening we have a few minutes for Q and A I'm happy to kind of just like get together after or online or whatever I put on GitHub some of these like container demos if you just want kind of like the easiest way to play around with some of this that always helps me hacking on the command line always helps me kind of get my head around a little bit not sure if it will you and finishing up with a calming manatee here might mark the yeah end of the end of the sessions at DrupalCon so thank you guys so much any questions what up so I'm guessing that you guys are probably using systemd and nspawn at Pantheon use it a little bit like primarily systemd has other ways of letting you interact with cgroups and namespaces that are using nspawn just through the unit configuration directives and so that's kind of an even lighter way of managing those cgroups and namespaces okay and what's nspawn's role there so nspawn is basically a wrapper that kind of helps you actually mimic kind of like a boot as opposed to just creating a process so nspawn helps you create like a whole it looks like I have a whole server to myself as opposed to like I'm just a process and I have this much memory and this much cpu or whatever and I'm guessing that you guys need some of that stuff for what you're doing I guess sorry this is sort of segwaying into a question about whether if you were starting today whether you'd probably build on docker or whether you'd sort of stick with systemd I mean if you were doing if you wanted to build a big platform which like people there's going to be amazing innovation in this space and I would look straight to coro s because these guys are rethinking like from the operating system level up how this stuff works so coro s all the way and I yeah but it's not quite there yet and then if I was a developer or a operator I would look largely to docker right here now thanks I hope to yeah so I was just going to ask because more servers more problems but you do need servers because you need at least one so what kind of goes into this decision on whether to containerize something or spin up a new VM for it so like if I could make a million dollars running one line of code on a raspberry pi I would have already done that but I can't and I need to use servers to like power the experiences and solutions I'm helping people get to the cool thing about containerization is once you're thinking about the water bottle it doesn't matter if you have a whole say you have two servers right whatever two big bp servers once you're thinking about the water bottle it doesn't really matter if you have one water bottle on that server many so yeah I think it's all about architecting for the portability and once you have the idea of containers they're portable and you can kind of move them around more easily then that's really kind of where you want to get to because then it doesn't really matter what physical it's an abstraction later abstracting what you're running from where it's running so at that point it just becomes a application type thing mostly what servers do I have available and what kind of processing needs do I have and how can I kind of allocate them which is the whole lem cut fee yeah and like specifically lem cut fee has like this pretty cool containerization thing but what they don't do what they don't open source is that like resource allocation thing which is like the big algorithmic kind of challenge there it's their money I started walking out and I realized I might have missed a very basic thing here so like what part of the stack are we containerizing I mean are you saying you know like for each website run Apache and Drupal in its own container as opposed to having multiple ones running together yeah how far you know how what would you couple in there and decouple would you put like you know would you put reticin there too or would you have that somewhere else elaborate on that a little bit please so as far as the technical constraints you can kind of do whatever you want right in terms of what I guess my experience is with is containerized resource so reticin is in its own kind of C group container and PHP is in its own C group PHP FPM well actually so that's a good point reticin is in its own container because we really just need one process to run it for application containers we pair Nginx and PHP FPM in the same container so in that case because those are kind of intimately tied we'll use the same containerization there and yeah I mean I think generally just you can think of like okay I want to run these two things do I want thing A to impact thing B or do like or do I not want them to impact each other and if you don't want to impact each other it's a case for containerizing them separately good question though and I think that's another kind of open question there are different ways to do it right you can containerize a whole Drupal stack and kind of one you know one server what we do is more kind of pantheon is more like more spread out so we'll have and this is kind of our decision that doesn't just kind of how we did it it's like we'll have servers that have all application containers on it and then servers that have all database servers on it and then servers that have all Redis servers on it Redis containers sorry thank you yeah and I think Docker has some kind of notions of how they're going to want that to work for you and those are probably good places to start for sure I noticed that you did a D bootstrap on a directory before you started a container so when you're trying to reduce the number of servers you've got doesn't actually lead to you having more servers because you now got to maintain and upgrade all those individual D bootstrap environments yeah that's a good point so I think in some ways definitely like if depending on your containerization strategy that you can get that complexity like you have as many like Linux like operating system like things right but largely you know if you go back to the network example there are physical network things that can fail whereas in the reliability of that is probably considerably lower than the reliability of like a loopback interface that you're configuring on the host so in that case although you are not completely reducing the complexity of kind of operating system like stuff you are kind of choosing for a higher reliability kind of in that case network you are talking about servers but does this containerization work in virtual machines for example instead of buying servers I would choose Amazon web services and just buy a couple virtual machines or buy from Rackspace totally yeah so like Pantheon largely runs Rackspace cloud VMs and also some hardware and other stuff and that's like the cool thing kind of with this example like of SystemDnspan where I have like OSX and then Fedora and then this other stuff that kind of by definition containers kind of don't really care what they're running on and you can do it on a VM just as easily one technical difference between a VM and a container is that in the VM you can actually run different kernels because you're emulating the processor but in containers you have to use whatever kernel the host kernel is using because you literally are using the host kernel so the previous question brought up a question for me you know there's still configuration management with all this stuff like with the example that we were talking about earlier with two questions ago with you know we have a Drupal stack engine x and fpm that is also dependent upon Redis so somewhere that container needs to know where to go to get its betters objects from and basically how do you manage all that so that's a great question and I think that's kind of what I was alluding to with the core OS which is and again starting kind of with the virtual file system like we use Chef we use a lot of Chef at Pantheon right that's like that configuration management is critical especially kind of if you're on one of the lower if you're on the lower end of the kind of spectrum here you're going to be probably doing more like hands-on configuration management if you're on the higher end you're probably a little less and so I think that's where my knowledge stuff like Docker and then core OS are trying like basically make it so you have to do a lot less of like the nitty-gritty kind of like make or set permissions like touch this file kind of stuff but depending on where you are in the spectrum you definitely need configuration management I imagine a little less so down the road and then the other thing you've got at is I mean you can it can fall under also fall under orchestration which is even if that reddest thing is perfectly configured and you're good to go until the PHP knows about that it's not going to be able to use it so that's kind of another piece which is kind of and depending on your architecture a different problem but notifying after you provision and it's ready to go notifying the PHP that hey here's this other resource you can connect to so do you know off the top of your head with any interaction between or modules in Ansible that work with these containers I do not I think there's I think that's something definitely worth looking into and like depending on what you're using Ansible with if you're using it with SystemD which you totally can then you can kind of pair those up pretty well so I think we'll do the last question here. Yeah real quick speaking of the Raspberry Pi if you have a 32 bit kernel is there any way of emulating a 64 bit using container I know you can do it the other way around Oh the other way around I don't know that's not really something that's been on my radar but yeah it might just be like a down down sampling kind of thing. Well if anyone wants to boff about it I'm intrigued but thank you very much. Okay last last. Hi it could be possible to dynamically assign resources to docker for example triggered by a number of heaps for a website without shut down the container. Yeah so I was just I was some resources on like the GitHub thing and like Google's kind of a great source for this because they like are definitely doing it at scale and I think I can't quite remember but something like one writes a second to C groups and 10 reads a second is what they were kind of you know saying was happening on their servers so as much as once a second you know you could be kind of reallocating CPU shares or something. I think that probably also depends on just the hierarchy you set up like how how dynamic you want to be changing those. Our use case at Pantheon doesn't really require very dynamic reshaping C groups. Thank you. Yeah. Alright thanks everyone.