 Welcome everybody. Everybody will come in and get us sat down. Erin is here to talk to us about Linux containers. Thanks. Yes, hi Erin. I've been doing security, ISEC. If you don't know who NCC Group is, ISEC, Montesano came together, we were brought by NCC Group from the UK. But same people, same place. I've given some talks before on some different things. So this is my first DEF CON talk, but it is my 12th DEF CON. This is my 10 second promotional motivational slide to not wait as long as I did and not be a slacker. And if you've got an idea and we've got lots of inspirational talks and cool talks here, keep that going and do research and submit to the CFP and give talks. Don't wait as long as I did. So I want to start with this story. So Bob, this is Bob here. He's got some web app or he's got something that's on the internet. It's on Linux. And somehow there's a bug. He gets popped. There's some kind of RCE. He gets arbitrary code execution, something like that. And he wants to add some security, but he's not really sure. So somebody says, well, you should use CH root. That's legit. Open SSH uses it. It must be good, right? But as we all know, it's broken if you have root. You essentially just CH root yourself and then you can go up to the root. So that's well well known. So somebody else comes along and says, oh, you should use SC Linux, right? That's easy. The NSA made it. It must be secure. If you know anything about SC Linux, it's a super complicated type based system. It's really designed for multi-level security. It's really designed for kind of government things when you've got lots of different things to classify. It's got really good support on Red Hat, but other than that, it's not so great. Some of those reasons, right? Complexity. It's really complicated to actually create it. It has the Linus Turvolds problem, which most of the other Linux security modules have where, you know, he doesn't really believe in any of them. The set and force zero problem where, you know, if you walk into a box that has SC Linux, the odds of it being disabled are high. And, you know, at the end of the day, the kernel's got a kernel, right? So if there's a kernel vulnerability and SC Linux is being enforced by the kernel, it just gets turned off. If you ever look at any of Spender's exploits, he loves to just do that first and then dig at them as he does. But speaking of Spender, so somebody else comes along to Bob and says, no, you should use GR security. That's the best. And yeah, it really is in a lot of ways. But that's not really solving some of the problem, right? If he has WordPress and he's got some arbitrary, you know, command injection, GR security is going to do nothing for that, right? You're doing a lot of memory corruption protection. You're doing a lot of other things. But unless you do all the kind of turn everything to 11 with RBAC and everything, it's not going to solve that. So then somebody else says, no, you should just use VMs. You know, that's how to be ultra secure. But at the end of the day, you know, VMs aren't perfect either and you've got a lot of other problems with configuration management and everything to do there. So the second story is Glenn. He knows a lot about security. He's a crypto cat. And so he talks to this potential source and he's really, really paranoid. He wants to run Linux because, you know, he doesn't trust OSX. He doesn't take that seriously and, you know, malware is on Windows or whatever. And, you know, all the cool kids run Linux and it's got to be super secure, right? But not really. I mean, he's one Tor browser bug exploit away from having his calculator popped or whatever may happen. You know, lib purple track record is awesome. Last time I updated my live CD, that didn't go so well. So the third story is Margaret. Margaret works at an IOT company. She is tired of getting these, everything running as root and everything getting popped and not having to be able to see any kind of security on this little platform. Then I actually inserted this slide after I saw Charlie and Chris's talk where, you know, maybe Margaret also works on the GPU connect team and doesn't want to run D bus as root and have just come, you know, no ability to control things. But really, you know, a lot of that's done just because money obviously takes time to add security. There's no, you know, there's no way to virtualize something that's on ARM or MIPS or, you know, this little embedded thing that's running Linux. You know, a lot of these other things are too heavy duty solutions. There's got to be something we can do. And really these stories have in common attack services, right? That's the worst attack service possible there at the bottom. But really, you know, attack service is really important. And sandboxes and containers, it would be great to see them being the norm, not the exception, right? If you look at Chrome and Adobe Reader X, Seatbell, you know, any of the like kind of modern sandboxing environments, that really should be everywhere. There's a battle that we're fighting with native code and you're going to lose, right? And so until we can win and everything's rewritten in some crazy rust or whatever it may be, we have to cut our losses and we can do that with containers. You can do that with sandboxing. You can do that with other isolation. So for Linux, you know, open source loves to reinvent the wheel. But how I'd like to kind of set this up is, you know, being in Vegas, one of my favorite movies, so there's this movie, Boondock Saints, where, you know, they come out of this house and they're staring at six guys with guns drawn and, you know, it's a firefight to paraphrase. And there's this kind of bumbling investigator that says, you know, what if it wasn't six guys with six guns, but one guy with six guns? And, you know, he's kind of an idiot, so William DeFoe is William DeFoe and he says, you know, shut up, you don't know what you're talking about. But really Greenlee was right. It was one guy with six guns. So, you know, the question is, what if it wasn't a whole bunch of kernels, but one kernel with multiple user lands? And that's really the core idea of containers, right? So this also is not a new idea. This has been done a long time before Docker and everything came along. So before we keep going, I just want to say that this is, if you are really scared about security or incredibly paranoid and you have something, you know, ridiculous system, you do not want to rely purely on any kind of containerization solution. You don't want to rely purely on any virtualization solution. You know, you need guys with guns and fences and, you know, air gaps, actual air gaps, not things connected by wires. And really, you know, I'm not up here to say that containers can do everything. But what I am here to say is that you don't want to depend on any one single method. And containers are getting a lot easier. They're getting a lot powerful, more powerful. And if you can do anything, you might as well just add things in layers, as we've seen. So to get that to how this actually works. So the first major area is namespaces. So within the kernel, just like a lot of any other cool stuff on Unix, it really started in plan nine, where they had the idea of namespaces in the first place. But the Linux kernel is split up into these kind of five namespaces. Network, Mount, user, UTS, which is host name, and pit and pit. And the way you get into one of these namespaces is you, when you clone, when you process clones, you know, spawn a child, it's kind of the new fancy fork. The kind of phrase that you get set up into is you enter a new kernel execution context. And the way you do that is by, when you run, when you call clone, you add these special flags at the end, depending on the namespaces that you want to enter. And so the Mount namespace was added first a long time ago. It basically lets your process have a per process view of what the file system is. IPC, we can keep going through these. You know, your processes can have their own host name. They can have their own view of the other processes. And they can be nested and they can, you know, there's some cool tricks you can do. If you look at how this actually works, you know, if you create a container and you run PS in it, you pretty much just see the processes that are in your container. Which leads me to the awesome snake oil solution. You can't hack what you can't see. So the new, there's also a network namespace that isolates your IP, your firewall, your routing table, all that kind of thing. And the username space, which is the newest one, which was added, that is really important for securing your containers and essentially lets you be root inside of a container, but that's still treated as a low rights user in the context of the kernel outside of the container. Which obviously is also a high risk area of the container. You're really controlling a lot of how the UID system works within the kernel in very sensitive areas. And so there have been a few vulnerabilities of, you know, using the username space to break things outside the context of containers. The way that it looks is, you know, if you attach to our foo container, if you, your root you can see inside, you run sleep and then outside of it you're actually UID 100,000. Essentially just shifts all your UIDs up by some offset. So the other major area of containers is capabilities. The idea of capabilities is you take the user root, right, who can do everything and you blow that up into a whole bunch of little pieces, you know, whether you need to bind to something lower than 1024, whether you need to be able to run ptrace, whether you need to be able to, you know, change your network settings. And capabilities are great, right, that lets us get rid of this just kind of god mode and split it up into a whole bunch of little pieces that really can do, you know, only what you want the process to be able to do with root. The bad thing is that they started lumping them together a bit in weird ways. And so you end up with kind of a capabilities model that somewhat works but is confusing, hard, and can be messed up. So some of the, you know, kind of dangerous capabilities or capabilities that you might commonly encounter are things like being able to receive raw packets or bind to something lower than 1024 or change resource controls or send kill. So everyone always asks, you know, what capabilities should be dropped, you know, should I drop this or that? And really you want to drop all of them. You want to set up everything for your container or your namespace ahead of time and then when you pivot into it, you want to throw away all your capabilities and just live in your little cocoon and you can't do anything at all. And then you get questions on forums or even just people as implementing this, you're thinking like what if I just leave this one capability enabled? What happens? And the answer is it depends. So if we go and look at an example of this, so we look at Ping. Ping is setUID root, right? Everybody from forever ago remembers that, you know, like CD record with setUID root and you could pop shells on Linux and do privsk by doing, you know, any kind of setUID root binary with a vulnerability you get instant root. So it's kind of ridiculous that Ping, you know, on these days still has root and setUID root. Obviously the attack surface is Ping and has been pounded on a bunch. If you copy Ping to somewhere else, you'll lose your setUID root bit. And if you try to run that Ping, you'll get operation not permitted because you need to be able to create raw sockets for Ping or ICMP. So the way to fix that is you can set a capability of CapNet raw on that new elf. If we, you know, look at that, we can see that we have CapNet raw. And then when you try to run it, even as a low rights user, now I can Ping, but that's the only thing I can do. That's the only capability that I have. So obviously there's a lot of dangerous capabilities that you don't want to give things. Being able to row to override, you know, the discretionary access controls, or turn off the manager access control system, you know, things like that. That's definitely things you don't want to have. The other bad thing is there's a capability CapSysAdmin. And obviously that sounds pretty important. It's basically root. There's a whole lot of things that CapSysAdmin can do. There's also a great post by Brad Spender that goes into all the detail and all the different capabilities and how to use to get root or do bad things. And that's the link to it. And so the other major aspect of containers is control groups. The basic idea of those is they're hierarchical and inheritable system for controlling resources across a set of processes. And that can be devices or CPU usage or physical CPU, memory amounts, I.O. rates of a certain device network. And it's really you limit on steroids if you want to think about it that way. And it's mainly used to kind of fill the gaps of some namespaces. So right now there is no DEV namespace or there's no namespace. So you kind of have to get around that a little bit by using C groups in a way that someone intended to do that but not really. They're typically controlled through a proctile style file system of pseudoFS. So it's just a directory and you put create directories and this corresponds to PIDs and things will read that directory. The good thing is that obviously it's a file system based and everything on Unix is a file that used to be at least following that model. But at the end of the day you can do tricky over mount attacks that is a side effect of having it be a file. So it's it can be controlled through CG manager but really most of the container platforms will abstract this away. So when you put all that together namespaces isolate the elements of the kernel, the capabilities help enforce those namespaces and limit the capabilities within that container and then the C groups limit the access. So really those three elements along with some other secrets, magic sauce will create containers and it is better than CH root. There's a lot of background to those. You get special mount options. You can do things like an overlay file system where part of the directory is shown to the container and part of it is shown to the host and there's some go in between. You can do crazy stuff with SSHFS now. So where are Linux containers being used now on servers? So a lot of platforms as a service systems, you know, EC2, you can do Docker, Google App Engine uses containers. Heroku has been doing it for way before anybody did and they, you know, Rackspace everybody probably more that don't have on that list. It's also being used in clients. So Chrome OS is a huge user of these technologies. They've done a lot of work hard hard work on making awesome sandboxing in Chrome and Chrome OS. It's somewhat used in Android. It's somewhat used on some Linux distributions. There's also some cool sandboxing tools if you're into those that use these without kind of going to full blown containers. So the kind of major one that started on the kind of path of Docker and the current trend was LXC. So LXC is kind of vanilla and I won't go into the details of this, but essentially it's controlled by a template, you know, you set various options, you know, what devices you want to have, some security, different, some security things like Set Comp and App Armor or, you know, what capabilities you want to keep or drop. There's also been some recent advancement and I say recent within like the last year or two of containers. So unprivileged containers, I mentioned that the username space, so that really creates the ability for container that they can then be root in but is all treated as a non-root user when it comes down to it. Obviously there's some weird things that can happen with that situation so there's more work and auditing needs to be done there, but that'll be, you know, a futuristic thing where, you know, a low rights user that doesn't have root and never ever will get your root will be able to create containers and do things. But you have this problem, right, we talked about attack surface. So syscalls are one of the main entry points to the system. There was a lot back in 2.2 and now there's a whole lot more. What are all those and does your app really need all those, you know, 300 of some odd syscalls? Probably not. So then this thing came along called Set Comp and the guys at Google Chrome team are doing a ton of awesome work to really push the boundaries of what you can do here in a very good way. This has been tried before, but the way they're doing it is much better than some of the methods that have been tried in the past. But it's important to point out that Set Comp by itself is not a sandbox. It's purely made to just limit the attack surface to things that your app really doesn't need to have. So you can also, not only can you filter on certain syscalls like this process, it's just crunching numbers. So it just needs to call, you know, return and open and that's it. But you can actually control the arguments to those syscalls and say, you know, it only needs to open this file or it can only do these flags on this file. And so that's pretty powerful, right? You do encounter a performance hit. So when you've got a lot of filters, there will be some impact. I think that we're still kind of figuring out what that is and how to optimize it. But it definitely is something to consider if you're going to be rolling it out. If you're going to use any jail directly or use mbox or something like that, you can't, if you're going to use docker with set comp, you have to use the slash contrib or wait till 1.8, which I think is when they're going to actually launch set comp support. But LXC has it right now and you can use it. It's typically done through the PR control. You basically can set one of two flags, the strict mode, which I think will be used for just like, you know, once you get your process into a spot where it's just going to be crunching numbers or doing something very basic, you can put that on there. Or you can filter, and by filter I mean a Berkeley packet filter, which everybody remembers from TCP dump. This is how you would use set comp in an actual program directly. And Berkeley packet filter is pretty cool. You can actually use TCP dump to dump out some of the things that I don't really know how it works underneath, but it's cool that you can see it. So where is set comp being used right now? Lots of places. You can turn it on in lots of places at least. It's not necessarily on by default. It is definitely in Chrome and they've been doing a lot of good work. It's also in Tor, but it's not on by default. So containers, you know, who's doing these, right? So docker, obviously, CoreOS, they have Rocket. There's lots of interesting companies that are either doing these in the background, or they're offering them as a service, or they're going to be offering those as a service. Big shout out to the sandstorm.io guys. They have a crazy customized system and they do GRSEC for everything. And they're super serious about security and everything else. And also the Rancher OS is an interesting idea where everything in that kind of minimal system runs in a container. It's also funny, I don't know if this has been fixed or not, but when you would Google that, it would actually bring up WAVOS Rancheros. Google would fix it. So yeah, they might have some SEO problems. But we're really here to talk about the big two, right? So docker. When I usually mention docker, this is the response. I'm from Silicon Valley, so this is usually the response I get from some devs. It's really packaging and kind of development focused. But one thing that is really good about that, I think, is that it's also very app focused. So the philosophy of docker is that it's not just a container with some things in it, it's ideally one process in that container and that's it. As a security person, I really like that because then not only is it that one process in that container, but it's that one process with just the libraries that it needs and just the files it needs and cutting off a lot of attack service there. And the big deal about docker and why it's gotten so popular is that it really makes containers easy, right? LXC was kind of doing it in kind of the harder mode whereas docker is you know, you just do docker pull and magic happens. But docker isn't just containers, right? So there's all these other things that they've done. It used to be based off of LXC and now they have something called libcontainers written in Go. They also have libchan and libswarm and all these other things. They have a REST API. It by default uses a Unix socket, but you can turn on a REST API. It also, one of the kind of downsides of it that they're working on fixing I think is related to the fact that you know there's this root daemon that runs and everything, all your containers and everything from that goes you have to be root to interact with it which then makes people run as root on the container host and that's never a good thing. So you can pull down your images, you can map things from the host into your container. There's also this idea of docker hub which is kind of like GitHub for containers. You can do commits and all this stuff. And there's all the orchestration, communication and management. I don't think the word orchestration really mentioned much all at all in the last 100 years except for maybe last year. So CoreOS is the other big one. It's really started out as a minimal OS for hosting containers. They're launching Rocket in this app container spec. They're trying to kind of distance themselves from LXC and docker. They're kind of main ideas that they're trying to be more secure and lean and be what docker should be or used to be. And why would anybody use any of these systems? And it's mostly just to make it easy or to package a lot of the functionality with it. And one way that you might kind of relate to it is you can run for EBSD and you can secure that but you kind of need to know what you're doing or you can run OS 10 and it's easy and it's kind of BSD still but you kind of have to take some things for granted about control. And so LXC, it's kind of vanilla, you can do whatever you want and you do get some bleeding edge there but you kind of have to know what you're doing in order to do everything correctly especially based on the defaults. Whereas docker, they do a lot of stuff for you. They drop most of the capabilities that you don't need but they still leave some around and you need to kind of explicitly disable those if your app doesn't need them. We'll get into that. So LXC, hard mode but you got flexibility. Docker, easy but you have some costs or some risks. It's very, very new. We'll get to that. But we're here to talk about attacks, right? That's all just stuff. So if you think about this, there's a lot of different ways that we can go about thinking about what are the attacks in a container or what should I be worried about. So there's containers to other containers. There's things in the container against itself essentially trying to elevate your privileges within a container if you're not just a single process. Container to the container host. Things supporting infrastructure whether that's orchestration and all that other stuff or whether that's other things on the network and you can go from there. So kind of starting at the top, the kernel obviously is doing all this virtualization stuff. So in the kernel, you've got all these old drivers, you've got all this old code, you've got weird file system support. Pretty much nobody compiles a kernel from scratch anymore that doesn't have all this bloaty stuff that you probably don't even need. Which means you're also inheriting those vulnerabilities of some random Canbus, you know special Canbus driver thing in Linux that you don't use on your server. The other way that this kind of typically goes bad is not dropping capabilities. So CapNet Admin is kind of a bad offender. There's been, I don't know how many different CVEs were of documentation and vulnerabilities where you can go from CapNet Admin to something else that you shouldn't be doing that isn't related to networking. The way that's happened in a lot of cases is being able to force CapNet Admin to load an arbitrary kernel module that you then control. That was, I think, one of the worst cases. But, you know, speaking of dropping capabilities, Docker was really stung by this. So they kept around for a long time the CapDac read search capability. And if you read the man page and maybe this is what they did, you know, it says something like, you know, you can read some files related to, you know, permissions. You can also invoke OpenHandleAt, which is a syscall that doesn't really have a ton of documentation. They probably thought, oh, whatever. But it turns out that if you actually use that syscall and you do some other tricks based on iNodes and other things, you can essentially brute force the iNode of EtsyShadow in the host and read it from inside the container. And obviously there's other attacks you can do related to that. But, you know, that was something that really gave the capability model a good example of why you need to drop the ones that you don't really need because they're complicated. And, you know, if you don't drop those capabilities, you have to rely on some other thing to enforce the rules on your container. And, you know, that's typically done through a manager access control system like SC Linux or AppArmor. But if you don't use that, the capabilities are your only line of defense because if you're root in a container, you're still root on a host unless you're using the user namespace which again doesn't have a ton of support yet or adoption. You also have the problem of limiting access. So we mentioned how the kernel there's not a namespace for PROC or there's not a namespace for DEV. And so what happens is you have things in the kernel or things in the container that are exposed from the host or that bridge of the host and the container. So things like PROC or where you can expose the kernel memory into the container. You have things like, you know, CISREC which everyone forgets about but stung, you know, LXC for a number of years where they, you know, were leaving that open. You also have things like being able to make new devices or even just CD message, right? That's not a ton of attack surface there but, you know, you're leaking things like maybe firewall rules or other stuff. And then unattain access to the network, right? So a lot of these container systems use a bridge system by default. If you remember networking 101 a bridge is a switch. So most of the default systems by I think almost all the containers will create a bridge and let your containers talk to each other. And that isn't something that most people realize when they deploy five containers on a host. They don't think that all those containers will be able to communicate but they can unless you've written firewall rules or switched to some other networking type. That's not limiting any resources. Obviously if you fork bomb a container with my favorite bash function there you're going to be fork bombing the host. And then kind of some other things relating to, you know, Docker has been stung by this a couple times of, you know, this idea of taking your application, packaging it all up with the libraries that you need, putting a lock on it and shipping it off everywhere. You know, that doesn't really happen. What happens when there's a vulnerability in one of those libraries and you need to update it? What do you do? You know, you can't just you're not supposed to run app get upgrade on your containers. It's kind of a no-no in the container world because they're supposed to be ephemeral and immutable. And so what do you do? And so you kind of need to have a system to adopt that, have a patching system. And then the lack of mandatory access control. So most of the container systems also have some form of app armor support or SE Linux support that works quite well. Obviously it's not perfect. There's been some bugs even very recently that were published there. But at the end of the day, you know, the kernel is still enforcing things. So this isn't a perfect solution either. And really it should just be one facet of many different things. And we mentioned that the networking defaults and how that can be bad. For LXC specifically, the defaults are quite bad for dropping capabilities. But that kind of relates to the way that the model of LXC is you have a minimal container, but it's not supposed to really just be a single process. It's supposed to be a few different things, maybe an SSH daemon or SystemD or something like that. Whereas Docker is the idea of one process. But speaking of Docker, so Docker does a pretty good job of dropping capabilities. But if you read the between the lines, they mention that they drop all the capabilities except for those required. But if they don't know what you're going to be running, how do they know the ones that are still required? And then, you know, you have to be root to use it. So that ends up being kind of bad. I've been on a number of engagements where there's been a, you know, people have been added to the Docker group. But, you know, if you're in the Docker group, you can essentially root the host trivially. There's also things like the REST API by default doesn't have authentication. So if you've turned that on and it's bound on problem where, you know, some outside attacker can get to it or, you know, some server side request forgery or some other thing where you can get to that system. And it could even be exposed to containers through that bridge system if you don't have firewall. And then it also has the kind of GitHub all the way down problem, which I think is not just Docker has the problem where, you know, you've got something like it's including these from these five vendors and they just accept commit requests from these five other people on GitHub. And all that code is just being chained down to some kid in the basement. And he doesn't know that the code he's committing is going into data centers and, you know, a crazy company. And all it would take would be one commit. There's been backdoored AMIs published before so we'll see what happens with GitHub and all that. It also, you know, I mentioned it doesn't drop all the capabilities, you know, by default it still keeps CapNetRAW, it still keeps the container. You can still make devices in your container. You're going to be limited by app armor, but, you know, most of the time your app probably isn't going to need those so you should drop them. When you bind a port to Docker by default it binds it to all the interfaces. And so if you have a multiple interfaces that are exposing to your container it's going to be on all of those. There's some other things related to, you know, base images are also really huge. So a lot of people in their file which is, you know, the main way of spinning up your Docker image they'll, you know, from Ubuntu or something like that but that pulls in like 250 megabytes of packages that, you know, you don't need. And, you know, that's kind of a problem where you think you're just running a single process in that container but really there's a whole bunch of other stuff there that could be used to try to attack or break out of that container. I mentioned giving the, you know, giving users access to the Docker user or the Docker group on a host, you know, essentially giving them root. It doesn't have support right now for SAC Comp or username spaces which are two, you know, really critical security benefits but it will very soon. I think 1.8 is what they've mentioned about that. And the other thing I've mentioned that I've seen on GitHub is people for trying to do introspection with their container if something in the container be able to see, you know, what's going on with them, they'll expose the Docker socket inside the container. Don't do that. That's giving your container root on your host. And speaking of the user name space, I have to kind of poke at them a little bit more. They mentioned on Hacker News that, you know, I'm the maintainer of Docker. You know, we will soon have support for user name spaces. When we feel comfortable, the Docker can run UID uninterested zero. We will say so clearly. We're going to go, no, probably more year than a year ago now. And they still don't have it. To give them a little credit, though, I think that one of the major reasons why they don't have the user name space is I was actually a limitation of go and not necessarily their fault, but they're on it now and so hopefully it will be soon. Also, every container talk has to have an image of some broken container. So I can check that box. It's still, you know, their whole selling point was that there's no root daemon. It's just a binary that you can run and it does this stuff, but there's no thing that's running on your host. It still requires root to run it. So I don't know if they've really solved the problem. It also doesn't drop almost any of the dangerous capabilities. It also doesn't support the user name space. Let's see, what else? Kind of supports SE Linux, but there's no documentation. For a long time, they had a bug where you couldn't actually run things inside a container without running them as root inside the container, which just seems crazy. They don't have a lot of limits they should on proc or proxys. So if you read the documentation, they have this very impressive design with this multi-stage system and it sounds really good, but unfortunately I think this is a competition. I hope it improves because I like to see competition, but I think right now this is where they're at. So then very recently, I think at DockerCon 15, they announced the open container project. I was really, really bummed that they didn't go with this logo because it's awesome and a great movie. So they actually changed their name to the open container initiative. And they really have this specification for containers and kind of to standardize the system and not have all this different kind of competing things. They launched something called Run-C, which is a very, very minimal container that essentially is a kind of competition for Rocket. It's very minimal. You just define your container with a Gison file or something like that. And then it's backed by LibContainer, which is supported by Docker and RoboCop. So all that stuff sounds kind of bad and messed up and what can we do? So there's a lot of recommendations I have. I actually have a ton of stuff in a white paper that I'm going to be publishing really soon. So I kind of designed my slides to be somewhat of high level of each area. But the white paper goes into a ton of detail on all this stuff. But really the big message is defense in depth. Nobody says that and I can tooth that horn a bunch. But really if you're trying to defend against things you want to build your defenses in layers and that's really the only way you're going to do it. So if we're going to do that we want to start with the kernel because it's got a kernel. So you want to do GR Security. Like just everyone should be running it. It's ridiculous, especially on a server where there's no special hardware. Sometimes on a laptop GR Security can be a little tricky to do with drivers or whatever it may be. But on a server when you're just running a few things or you're doing containers just do it. And so obviously hardening your kernel in other ways. Dropping all the capabilities that you don't want trying to design for the smallest possible set and also assuming the worst. So you're giving these capabilities out but you want to be careful about what they are and try to fully understand what each capability really can do. So if you have an app armor or a GR Security RBAC or a SMAC or some other thing if you can most of the container systems will have app armor by default enabled which is awesome. It also can be nested so that's kind of crazy to think about but you can create an app armor profile for a container and then in that container if you want to then have something be able to apply a secondary set of that first one you can do that. That's kind of neat. For Docker specifically you don't want to allow users to run Docker if you don't want to give them root on that host. You don't want to run containers that are privileges or containers that are root. As we've seen either using an IP table trick or using the CapNet bind service you do not need to be root to bind to lower the 1024. That's you know if you how many times you've been on a project I'm sure you guys know but you've been on a project or you've looked at some security of something and the only reason why they're running as root is because they need to bind to port 80 or 443 and that's just stupid and that doesn't exist anymore and nobody should be doing it. Using small base images is also cool. There's a really good talk about building a crazy tiny go Docker image and it's going to be documented. Using set comp if you can so right now with LXC you can use a whitelist of syscalls that you want to allow your app to be able to call. That can really cut off a ton of attack surface in ways that the kernel is typically accessed. You want to do all your normal system hardening that's a long list of things but there's a lot of stuff to do there and that's what should be done. You want to avoid things that are listing on all interfaces that cross container networking. The other thing that's cool about containers is if you have these different interfaces it's really easy to tell what's going in and out of my processes in that container and so it can be great for trace flow. The core picture is you've got this hardened application then under that you've got your user namespace you've dropped capabilities then you have mount protections. For maybe like the log files or something like that. Then you've got a minimal container distro and then you do sitcom filtering and then you've got GR security and then you can even do a hypervisor or some other kind of virtual machine to gain some actual hardware separation and then you isolate that whole thing by some trust that you give it and then do that differently for other things. For containers that are getting hit by your API that are on the front end of your system those should be treated with different trust than some container back end that has your database. You don't want to have those on the same system if you can avoid it even in separate hardware VMs that's not ideal. Where do we go from here? More namespaces hopefully will be coming that will help clear that up a little bit. Some of the edge cases where PROC right now is kind of a shit show there was a thing recently where Intel came out with this thing called clear containers which is essentially like a 20 megabyte hypervisor that will give you some hardware virtualization aspects that you can apply to containers. That's a really cool hybrid model that I think is going to be need to explore. More minimal container distros will probably be coming out soon or more refined going forward. And also because this is Linux on Linux and most of these features can be implemented anywhere Linux can run you can actually do this kind of stuff on embedded devices. You can do these kinds of isolation mechanisms or run containers on a phone or on whatever whereas virtualization or other things is less capable there. There's also, we've talked a lot about servers but really there's no reason why desktop Linux can't benefit from all these isolation mechanisms or using containers. So hopefully there's a lot more efforts there. I personally run everything in a container that speaks anything externally or that parses anything from the outside. So there's all of this microservices idea which is somewhat new. It's kind of cool. You end up something like this. I think one of the ways that's neat about it is that actually isn't a microservices picture. So if you think about it the way that microservices the idea of them splitting up everything into 100 different REST APIs that sounds like a pain in the ass but really it lets you do a lot of least privileges and least access and you can do a lot of crazy cool security where everyone always draws all these boxes on a white board of this is what our system looks like but at the end of the day there's a whole bunch of other pieces that get connected there just my auth service talks to the database it also could talk to some other thing because nobody writes firewall rules for all that stuff so the microservices model is this kind of splitting everything up into pieces running it all in little containers having message queues and all kinds of things unfortunately it starts off with something like this and ends up something like this when it gets implemented so really it's not about perfect security containers you can't just drop that and think you have security but it's improving the work the number of times that I've been on a pen test or some red team or something I've been stopped by popping something and then having it be inside a sandbox or a container is close to zero and that's just silly these days because it's not that hard to do containers it can secure a lot of existing systems that are Linux and the microservices architecture it's coming it's going to be adding that's going to be a lot more popular and I actually like it from the security aspects of it so coming soon so my white paper is coming out it covers everything that I've talked about in a whole lot more depth it covers all kinds of past attacks and looking at new areas and everything and hopefully it'll be released soon if you want to make sure that you get it I'm going to be writing everybody emails me in a text file you can follow me on twitter and I'll talk about it at some point any questions or comments yeah three and I'll be around after this is the last talk thanks a lot for hanging out it's kind of late but I appreciate it