 Keyf소�der, iawn. So, as Rachel said, I want to tell you a little story and I want to tell you a little story about Docker. You may have noticed for the last three years the Linux community has been embracing Docker fundamentally. People are shouting left right centre about how amazing it is to deploy containers, how it has changed the way that we can deploy applications at scale in a distributed system and we can finally start taking advantage of the Cloud. But like with many things on the Internet, not everyone agrees. Onw, while everyone is shouting about it, everyone is also kind of complaining a little bit. And so you get these really weird kind of like quite flamboyant conversations happening online about, is Docker secure? Is Docker nicely architected? And looking back on my slides, I think the final one doesn't actually relate to Docker, but you kind of like it's kind of similar. And so, I kind of was looking at this and I was like, really? Like Docker has got a huge amount of community support. It's got some very, very smart people work on it. We've already met one of them this morning. Like, how much of this is actually true? And in particular, how much of it around security is true? Like, how secure are containers? And that's what I want to talk about with you today. So my name is Ben Hall, as Rachel kindly introduced. I am the founder of the Othalot Uproar. We build products and we help companies move to containers and help them adopt Docker. Our main product is Cata Coda. It's a free interactive learning environment for software developers. We help people understand how Docker, Kubernetes, all of this cloud-native DevOps technologies are coming along and how you can actually take advantage of them or use an environment driven from your browser. But the side kind of side bit of this is when we started a project, we had a fundamental question. What happens when you give anonymous and restricted access to a Docker container and a Docker server in the browser? And what do people actually do with it? And this is the story of how we try to secure ourselves. And based on the end of the presentation, we'll see how good it is if whether it's still working or not. So if we take a step back and we actually think about what containers are in the first place. So a container in the real world is this huge, big massive metal shipping containers. And they have various key important properties. They're about this wide or about this high and they have doors in a certain location. But most importantly, they have hooks in every single corner that allow them to be connected together and that's how they create this very strong coherent unit which allows them to send billions of items all the way around the world. What's actually on the inside of a container doesn't really matter. It could be MySQL, it could be Ruby code, it could be Python. It could be whatever you want. What's important is that there's these standard ways that containers are built and there's a standard way that these containers are launched and started. Because we've got this standard and because there's a consistent way, we can also now use that to apply a consistent way to start securing them and locking them down in a more effective way than what we had previously. And so if we come back to Catechoda, so the first thing which people see when they log on is a terminal window. This has got a very interesting mindset. People kind of go or 30 developers, not everyone, but 13 developers, kind of have been probably watching Mr Robot the night before and they instantly go like, ah, let's break it. Let's see what I can do because obviously this can't be real, so let's see what happens. And they do a certain process. So the first thing they do is like, ah, which user am I running as? And they'll see that it's root and they're like, ah, even better. So they then start poking around a file system, they'll start exploring, can they get up to the root directory, can they install other packages to extend it and potentially add some additional functionality that they want to take care of. They then at some point kind of get bored and they start wanting to change passwords and then they get really bored and it's like, ah, I just delete everything. And it's kind of like a very interesting mindset. And the reason they do this is because they see their root and then they kind of think, I've got complete control over everything which I can do. And this is kind of true. By default, all Docker containers user which starts them is root. But it's very simple to change the process and change the process that the user is running as. And so in your Docker file, which basically is a list of instructions about how your application is deployed, you can add an instruction in there to say add a new user and then when this process starts, run as that particular user. Or if you're taking an image offline and taking one of the official ones, for example, MySQL, when you launch that container, you can say use an unprivileged, non-privileged, non-root user to launch this process. And so by that, you kind of are restricting what the users can do. They can't install packages. They can't update passwords inside the container. And so you're kind of locking it down and removing some of that privileges. And from our point of view, this helps people get more focused on learning and not trying to break it. But you know, some people still continue to progress and they still continue to explore. And so then they'll start poking around and trying to find out more information about the host operating system. Because Docker is in a virtualisation system which we would be used to in terms of like ESX or then or other hypervisors, Docker doesn't provide that level of virtualisation. Instead, Docker is a kernel virtualisation system. The containers are taking advantage of the host kernel, which is how we get all of the performance benefits and speed improvement because it's sharing the host operating system. But as a by-product of that, the container can see what the kernel is actually reporting. And so you can find out the kernel version numbers. You can find out the uptime of the host because that's reported by the kernel. And then you can kind of like find out free memory, disk space, what processes are happening. It's not malicious. It's just kind of like a little bit annoying. But as with security, like all information is bad potentially, they could use potentially use that as an attack vector. But so far from what we found, it's just kind of like annoying because they would prefer it if they didn't find out. But we'll see. And they eventually find out they can't do much with that and so they go like, huh, we'll just reboot or we'll just do shut down. And it's like, you can't do that. It's a container, like what are you going to do? You can't shut down a container. There's nothing to shut down. And that's kind of true if you run Docker defaults and the Docker default privilege levels. But if you start changing things and you start updating certain properties, for example, you changed a network in namespace from not being the container, but to being shared with the host operating system. What actually happens is when you call that reboot, it's broadcast a message out on the network bus, the D bus. And that's how you can feel like if you're connected to an SSH session and someone reboots it, you'll get that warning saying it's about to go down. Please save your work and then it just disappears. So that's how it works at the network level. And so when your container is also talking on the same network, it's like someone can reboot the host. And so you need to be aware of these interesting details of Docker. And we need to be very careful when we've implemented CataCoda to make sure that we haven't accidentally elevated the permission of a container. That would be embarrassing. But there's lots of reports about Docker's security and container security and comparing the different container run times and where they fit in terms of security. And as we can see from the middle, Docker has got a lot of green. Green is good. Green is good as we found out from the previous talk. So everything is strung by default. This is what Docker's mindset is. We want strung default security. But there's also some optional ones in there. And that's kind of interesting. So what does optional security mean and what are the side effects of that? And also I think this is probably what will happen if you run processes on a Linux box without Docker containers. It's kind of like having any of this at all. So the first one, which is potentially problematic, is this thing called a fork bomb. This is a beautiful bit of bash scripting if you break it down. So it starts by defining a new function in bash called colon. The body of this function is calling itself and piping the results of itself to itself and then releasing control. So it wins in the background. After it's finished defining itself, it calls itself. And then instantly, you can tell the effects of running this code because you have 4,000% system load, you have 7,000 processes running and that's kind of like the last you'll ever hear from that host until you do a hard reboot. This is problematic and it's basically just a DDoS attack of resource exhaustion trying to use all of the available resources and PID limits and open files and everything like that. This was problematic because there wasn't really good answer on how to protect against this. But in the recent version of Docker and the recent version of the Linux kernel, you've now got PID limits. And so when you start a container, you can specify saying that in normal operating system and normal boundaries, there should only be 10 processes, a maximum of 10 processes running into this container. Let's be honest, if you're running bash, you're probably going to have bash plus a few other processes, but you're never going to need 10,000 of them running in your system. So you set a same default kind of safe limit and then if people do get access to that container and they do try and execute a fork bomb, they'll just get this kind of like going, they can't do it because all of the PID limits are restricted. And this wasn't available before. And if you install Docker and you haven't got it the latest version, you will get this warning. And this is what I really like about the approach that Docker is taking. It's kind of opening up really hard, really deep levels of the kernel and making them accessible to everyone else just by adding a command line property when we start the container and when we start that process. Where previously you would have had to actually understand the internals which no one would have ever done. It would have been very difficult to access and everyone would have just kind of ignored it. And there's lots of other ready settings with Docker and there's lots of other ways that you can secure and protect against your system in case anyone does access that process. And so you can set the memory usage and so you can say that this process should only ever have two gigabytes of memory on the host. And so if there's a memory leak or if someone's trying to launch something potentially malicious, it will limit it and so it can't take over again exhaust all the resources. You can do the same for CPU, you can do the same for the IO disk rights and so you can start delivering a quality of service to your application. And whether just beyond security, you can say that this process will always need this much of memory available, this much of disk IO in order to deliver the performance and the expectations from a user's point of view. And Docker makes that really simple by just again setting a few properties. But once we've actually gone through, a very common and very popular one is privilege escalation. So we've got our container running, we set all the right properties, it's running as a non-root user, but obviously people want to have root access so they can do potentially more interesting things. And so there's some interesting hacks and some interesting exploits floating around. This one in particular was released with an exploit which took advantage of Ambutu 16.04, which was the one which was released a couple of months ago. And it took advantage of BPF, which is kind of like something to do with file descriptors. I mean, it's basically... There was a certain system core which if you hooked in at exactly the right point with exactly the right process, you could have some additional things running as root and then you could core things like setUID and setGUID on a process. You can remap these from being the low, privileged 1,000 user to being zero. If that all works successfully, any additional processes launched would be root and as such you can launch bash and you would be a root user. And you can kind of see things as this happening. So this is the piece of concept which is floating around. You run it, it outputs some nice messages and at the end of it, at the beginning you was Ambutu, which was the non-root, non-sudo user, at the end of it you are root without ever handling the pseudo password, which is kind of quite dangerous and scary because there are no restrictions and once they are root they can do whatever they would like. Again, Docker thought about this and so even though the latest version of Linux kernel which was shipped in Ambutu 16.04 was vulnerable to this attack, Docker have defeated it in there to protect against it and when you start launching a container, you can set a property saying no new privileges and what this is basically saying is given the user which the process is running at when it was started, no matter what happens it cannot request additional privileges, it cannot access additional and it cannot go from being non-root to being root and so even though that kernel exploit exists and that's something which could be exploited if they had access to the container, they still wouldn't be able to get root because Docker is providing an additional level of security. And so this is where it starts getting interesting because you're no longer talking about Docker security, you're talking about Linux kernel security and you're talking about what exploit potentially exists and that as the Unicernel talk this morning was discussing, the Linux kernel is pretty big. There's a lot of really interesting code and a lot of really interesting code which people may or may not have looked at in a while. You only have to look at the open SSL to realise there are certain bugs in there which are quite embarrassing which people have kind of like missed over the years. And so set component of Parma, I'm not going to do that in case I drop it, are two additional ways that you can start locking down processes and start adding additional security in Linux. So when a process is running and when you run a process in Linux, under-covered is doing a huge amount of system calls and this is a trace of just changing the file permissions and with a process in the steps which is actually happening under the covers. And you'll see certain things like if checking that the file exists, it's opening the file and then right at the bottom is doing the actual to mud core to change the file permissions. And so with Docker and with SEPComp we can actually start locking down and start being selective about which system calls the containers can and cannot call. And so this is one of the really key security aspects of Docker because it just fundamentally blocks potentially dangerous, potentially malicious calls from happening. And if you can't call it, you can't attack it and that's itch, you are somewhat more secure. And the way that we can access this from a developer's point, from an implementer's point of view in Docker is through JSON file. There's a default JSON file which you can adapt and you can modify and it's just basically a list of all of the system calls in Linux of which there's around 331-ish and you can just say whether you want them to be allowed or whether you want them to be restricted. By default, you can restrict it and then when if someone tried to do that system call they'll get an operation not permitted and that's itch, the call will fail and so you can add that additional level of security. This is really interesting when you start and look in at how people are exploiting certain parts of applications. This is another exploit against the Linux kernel. What this was doing was attacking certain performance counters in the Linux and was taking advantage of certain different hooks and certain ways that you could call them in a certain order. It's quite complex but the result is it pins a CPU to 100%. If you look and you do a system trace about what's being called, you can see that there's thousands of calls just to these two system calls alone. One of them which ends eventually in 100% CPU and it's locking the CPU. If you run that scale, you could take over the entire box and it would be DDoS and again, you'd have to do a hard rebate. By using system sysc comp, you could take this modify our DT system call which generally most applications never call and there's arguments to say that it should be blocked by default in Docker anyway add it to your set comp profile and even if that application did get hacked and they wanted to exploit via this vulnerability, Docker would say that operation is not permitted and that's such that the exploit wouldn't be able to run and against it securing you even though your underlying system is vulnerable, Docker is adding that additional layer to making sure that it's secure. And this is all great. This sounds great, this sounds perfect but what happens when it all goes fundamentally wrong? So this is what happened to me. So it starts off by enjoying a allegedly nice Saturday afternoon watching some tennis on Wimbledon, watching Wimbledon on the TV and you get this really cool, interesting email from your hosting provider going, there's some malicious or there's some odd activity on your network. You may want to investigate it. It was raised a little bit more strictly than that but that was kind of like the gist. So you go into the control panel and you have a look and you look at the networking and you kind of see that there's 568 megabyte bits per second and it's kind of like, oh my God. Like I know my blog's popular and everything but that's just like, that's unheard of. I don't even know how to generate that much traffic in the first place so at that point you kind of realise something horribly, horribly must have gone wrong. But then you kind of realise like everything I'm running is on containers and so like they couldn't have done that much damage in theory and so there's actually tools within Docker to help you understand what has happened if an intruder just break through. And so the third thing is all of the commands or all of the standard out and standard error is being logged. It's being logged by default onto disk but you can also tell it to log into an out stack or Splunk or other log centralisation systems. And so you can look at the system and it will tell you what happened. In this place, in this case, the attackers kind of exploited the fact that I had forgot to turn off an elastic search instance which had kind of opened up to the public without a user name and password which was pretty embarrassing, but it happens. And so I looked at the log files and you could see what they were actually trying to run and they took advantage of a feature within elastic search which allowed you to make a send Java code which would execute via the API. But obviously, like the elastic search team had thought about this and it was that they had blacklisted certain parts of the Java API so you couldn't do anything malicious. But on the flip side, hackers, there's always two ways to do something and if you only restrict one of them, they'll just find and use the other one which is what they did here. And that's why they're doing things like new input stream, getWinTime, exec. And then they're using that to download additional binaries, execute additional commands onto the server. And to be fair to them, they were very polite, they did try and clean up after themselves and delete the files which they had left over, which was nice, but it was still potentially problematic. Based on the way Docker works, when you start a container, it stops from an image and this is a pre-known pre-built image. And then any changes which get made when that container is running is basically an additional layer on top of that image which is running for that one container. And so what you can do is compare and you can diff between the current live state of the container and what the original image was in the third place. So this is actually really good if you think your system is immutable and nothing ever changes. Like you can actually go in and verify that your application isn't writing additional data or changing configuration files when it shouldn't be because it's all immutable infrastructure. But in terms of security and in terms of figuring out what attackers had done, I could actually see and I could compare and contrast. And you'll see that they had changed certain key properties of Linux. So the net stat, the PS, the SS, this is probably trying to hide the fact that they're doing something malicious. So when you would normally be trying to debug it and you think what process is running, this would have hidden that for you that you couldn't tell. You can see that they've added some SE Linux security properties, probably trying to override something which should have been there in the first place but I didn't implement it. But they've added those in the first place. And then they've added some additional files like DDoS 99 and DDoS 8 which you imagine is where all of the traffic was coming from in the first place. But you can now compare and contrast and you can look at what the hackers did. And because it's running inside a container, I know that if I remove that container and if I delete it, all of the files and all of the potential exploits will be removed with it. I don't have to worry that they dropped a root kit and they're kind of sniffing all of my network packets in the background without me being aware of it because the container, by default, is locking it down in it, offering the security levels which you need. In reality, if I had listened to my own talk, I would have been running the container as read-only mode. A read-only container is basically it allows you to write data to any mapped volumes. So anywhere where you see the explicitly said this container was allowed to write to. In the case of elastic search, that's slash data, which is where it's doing all of the indexes and all of the log files. But everything else it will get due to the read-only file system, you're not allowed to write. So when we think back here, there wouldn't have been allowed to change the bin files because it's read-only and it wouldn't have been allowed to add the additional DDoS binaries because, again, it's read-only. So, yes, they would have got access to elastic search, but that wasn't an empty cluster anyway and that's not what they were looking for. So just by adding this one flag, I would have still been hacked but they wouldn't have been able to launch and do anything malicious on top of it. You can then start combining this. You can say that there's no new privileges so they can't escalate and then you can use AppArma. AppArma is basically a way to define a profile of what a normal running application looks like and you can say that normally elastic search is allowed to write to these directories, it's allowed to access these network options and it's allowed to work in this particular way. But what it's not allowed to do is write to these particular directories and call things like WGet because, honestly, why does WGet even exist inside elastic search container and I still haven't figured that one out? But, again, this is like things which we have to deal with and things which we're having to deploy along with our applications until we get to the point where we can do unicernals. And so AppArma is allowing us to define this normal looking application and anything outside the bounds of that is considered malicious and not allowed to happen. If you actually think of this in a non-database terms and you refer to think about how hackers are trying to raise awareness and raise what they're doing, the amount of defasements happen because the hacker can go in and write to the disk and change what files are there and replace it. If the file system was read-only in the first place, they could attempt to exploit it, they could attempt to try and access all of these WordPress vulnerabilities, but they wouldn't be able to actually write the changes. And so your website which you deploy in should never, ever change with a read-only container it never gonna change, even if someone does exploit a part of your system and you can't get it embarrassment and defasement. So if you think about that, is Docker secure? It's still to be confirmed, but Docker is doing all the right things. Docker has got security by default turned on and they're adding all of the right features in all of the right places to ensure that your systems are as secure as it potentially can be. If you think of the elastic search that would have taken over the complete box and just by running it inside a container means that it doesn't. But Docker is only as secure as your best practises are and if you have got weak security practises, then you are gonna get hacked, i.e. me opening up elastic search to the entire world. And there's also certain things which you need to be careful of. So if Docker has lots of official images on its hub, but it's also got lots of community-driven images on its hub too. And so if someone offers you, like me, the promise of cute little kittens if you give them access to the Docker file, the Docker Thucket file, you need to be a little bit cautious. Because once I have access to the Docker file, I can use that to launch additional containers. And these containers could be privileged and they could have all the access which I need and it could have a completely the same access that would if I had complete root access to your entire host. And so you need to be aware that certain things are special. And while it's really useful for containers to have access to this Docker API, if you don't trust it and you don't trust that image which you're running in production, you don't know potentially what's happening inside of it. And so you just need to be aware that that exists. You also need to be aware that when you're opening Docker to the outside world, Docker has got a very unique, very interesting signature when you try and send a HTTP request to the API. By default, it would turn a 404 and then it sets in the interest in headers. So the content type is always no sniff and the content length is always 19. We can use this. There is a beautiful search engine which is going around the internet and aggregating all of the open ports and their responses of every single IP address on the internet and then making it available for you to search. And so you can use this combination of knowing that Docker has this unique signature to find what could potentially be other Docker containers and other Docker demons. Obviously, I've never accessed one of these IP addresses because that would be breaking the Computer Miss Youth Act but you can assume that they may be Docker demons and you'll see things like content type of no sniff, content length of 19, and it returns a 404. If that's a very good bet that this is an open Docker demon, at which point you can do whatever you would like. You can run privilege containers, you can run DDoS attacks at the container and you can do whatever you'd like on the host which is kind of scary. But you can just actually just search for Docker and you can see what people are running and seeing what's actually out there. And most of the time it's with people with registries and if you're actually drilling it, you can use the information and you can use a host header to find out additional bits. So it's an old version of the Docker registry which potentially has got interesting security things which have been fixed in the future versions but they haven't upgraded. Or the running old OpenSSL which has got security bugs and you could use those bugs to exploit it, to gain access and potentially again to get access. And you may be thinking like, no one's actually going to do that. That would be ridiculous. No one's actually looking for these things. Well, for me you should say that. So if you look at the top, the top is from the search engine and it's got certain interesting properties. The main one being that the default response says it's a private Docker registry but that's always already kind of like alarm bowser ringing and the domain is docker.vineapp.com. Again alarm bowser ringing. Vine being the called video clip start-up which Twitter just recently purchased. And so this person went on to the website and it was still up, it was still accessible and they could access it. They could then use the Docker API to sniff around and have a look at what images were available and so they could then go ahead, download the image from the Open publicly available Docker registry and get access to the complete Vine source code because someone had forgot to close the ports. Oops. That's kind of just a little bit embarrassing. So thankfully Docker, this was reported. The registry was taken offline within about five minutes and the person behind it who found this was rewarded $10,000 for reporting it and being the honorable good citizen and actually telling Docker not just taking advantage and hacking it which would have easily been done. There's a great tool and a great website called Docker Bench. Docker Bench basically taken all the things which I talked about today plus 100 other security loopholes and other security advice and it's written it and it's built it as a container. The container will audit check all of your settings, all of your running containers and make sure that they're running in the most accordance to the best practices and it's also a really in-depth long report which is very insightful, very interesting to explain why these settings need to be there in the first place and what potentially happens when it doesn't. And so you should run this on your system just to make sure that you haven't done anything accidentally by mistake. And if Twitter didn't do this beforehand then they would have saved themselves $10,000 and I would have one less demo in my presentation. And kind of we need to remember the fact that by default most of the things which we run at least what I do and Linux systems were running at root without any set comp, without any restrictions and without any security in mind because it's so difficult. It's so difficult to actually understand how all of these moving parts work and by having that layer making it accessible through Docker at least we're starting to get to a secure by default design because without it it kind of just looks a little bit like this. And like if anyone gets in they will just take over the entire box and there will be nothing you can do about it. And like if you think that I don't need Docker I run virtual machines and virtual machines are secure you are somewhat misinformed. This year at the start of the year KVM had an outbound memory access which would allow a virtual machine guest to break out and gain access to the host which is embarrassing. There's not a proof of concept I don't believe but it was a theoretical risk. What there was a proof of concept for was Xenom which happened a couple of years ago and this kind of took advantage of an attack vector in the legacy floppy disk driver. And again like why did my virtual machine hypervider have a floppy disk driver? It's just insane. And then this is what caused Amazon to have to do rolling reboots across the entire EC2 because code running on an EC2 could use this attack vector and break out onto the host EC2 machines which would have been bad for everyone involved. And so you need to do this security in depth. And so with this in mind how secure are containers? So Docker has strong defaults. At the moment there are no known exploits which I can report that Docker has got advantage of and no one is talking about it but obviously everyone is trying and everyone is trying to poke in really interesting ways and it's only a matter of time. So we need to have the strong protections in place and be aware of the potential risks and so we can help mitigate them through things like PID limits and no new privileges and set comp. And we have to remember that really honestly nothing is secure. It's all about calculated risks and calculating how important is what we're doing and how secure does it need to be. Containers are one option, VMs are one option. Can the mining on will potentially make more secure sister but also it may not. You need to calculate this for yourself. From what I'm seeing and what I imagine the attack vectors won't be against Docker. It will just be taking advantage of exploits in a Linux kernel. It will go down deeper and then it will kind of come back up and pop out the other side and have complete access. And so this is why I really like set comp as an approach because you can start locking it down. But as I made a takeaway, to start looking at this of your own system and start figuring out how secure your infrastructure is, running Docker Bench is a great start. It will give you an oversight into what's happening on your system. It will tell you potential risks and it will tell you how to fix them. And it's all for free, it just wins out the container, wins you're finished, you delete the container and there's no harm done. With that, thank you very much. So we now have a 30 minute break but before we do that I'm going to ask Ben a quick question because I think security is becoming a bigger concern for everyone all the time and I don't know if there's any particular security specialist in the room but I feel like we all need to learn a little bit more. So where would you start? Learning security. Yeah, being a better developer, having a bit more knowledge. Cachicoda obviously. Cachicoda will have security training workshops up very soon. Oh, I didn't even do that but in terms of being secure, it's very hard, security is a hard problem but looking at how exploits are happening, how people are being hacked is a great example of what not to do and so that's a very good way of learning by other people's bad practices to start figuring out where there's potential weaknesses and potential risks in what you're developing on a day-to-day basis. That's where I learn most of my stuff is by looking at exploits because then you find other ways of thinking and that's the hard bit. You need to think differently and the best way to do that is by copying other people in many ways. All right, thank you so much. So now we have a 30-minute break. Thank you again, Ben. Round of applause.