 So, once again, my name is Salman Vasid, and my colleague will talk about container security platforms, container security and how that has evolved from last year and where it is going next. So container security is a hot topic, and we have talked about container security in OpenStack Summit last year and also in Meetups during this year. There is plenty more information about container security available today than it was year and year before. In this talk, we thought we would take a different approach, perhaps a more developer focused approach on how developers can make use of security that is available through containers that may not be available, maybe difficult to configure when developers run their applications directly on the host. So the talk outline is that we will talk about, obviously, what is containers, very quick overview, what are the threats, what is the evolution, and then focus on running an application and Apache web server on a host versus running the same Apache web server on a container on a host and compare, you know, what are the trade-offs and which one is better. We will talk about Docker versus Alexi versus Stockhead in the context of the report published by NCC and we will also discuss the ongoing work in container security. Before that, let me ask the audience this question, right? So when you are deploying an application in a cloud, you can either deploy that application on a host so you can run Apache web server directly on the host or you can run that Apache web server or any other application inside a container on a host. Do you think, please raise your hand if you think that running applications on the host is more secure than running applications inside the container on a host. Okay. Please raise your hand if you think that there is no difference in terms of security if you run application inside a host versus you run the application inside a container on a host, okay? And please raise your hand if you think that running applications inside on a host itself is actually less secure than running applications inside a container on a host. Okay. So that's good. So we think that running applications inside a container on a host is actually more secure than running it directly on the host itself. And this is primarily because with containers are lightweight sort of components that allow us running an application and they allow deep visibility and more importantly developer focused and usable security. Something that is difficult to do with virtual machines. So before we talk in detail, let's, you know, talk about what a container is and what are some of the threads and Phil is going to talk you through the next couple of slides. Sure. Thanks, Solomon. So this may not be new information to many people but I find, as Solomon said, as we've talked about containers in various venues and events over the past couple of years that a lot of times there's this sort of jumping right into the details of using Docker or Rocket or LXC or whatever container engine you're interested in but a lot of people haven't sort of stepped back to understand what does it mean. I know that there have been other talks in the same room over the past few days that have given you a sense of what a container is but just to revisit that to make sure we're all on the same page because these are critical concepts to even understand why containers are interesting or why people are using this kind of containment to run applications. So there's really two core features within Linux that are relied on that most of the container engines that we use are exploiting to create something that we're calling a container. It's a different world than a virtual machine. In the hypervisor world it's very clear what we mean when we run a virtual machine. There's all the sense of booting a system. There's all this virtualized infrastructure, devices, BIOS. The container is not as clear because it's really built on a set of disparate concepts in the Linux kernel. Those two main ones being namespaces and C groups. Let's talk about namespaces for a minute. There are six namespaces today in the Linux kernel. They did not all appear at the same time. They've appeared over a number of years. The mountain namespace being one of the original ones. But as the kernel has matured and these namespaces have been added, these are ways that you isolate a process in Linux from the rest of the processes on that system. For example, in the mountain namespace, if I start my process and ask the Linux kernel to give me a new mountain namespace, I now see none of the mounts of the original host system or any other contained process on Linux. My file system view can be different and obviously that's used heavily in the container world to basically make it appear that I have my own sort of file overlay inside my container. Again, the rest of these, the PID namespace, I only see the processes that are created from my initial PID 1, so to speak, in my container. IPC, I can isolate in a process communication. The network namespace, so my network devices and routes can be isolated from everyone else, so it looks like I have my own network. UTS just holds the host name and domain name, which is important for obviously applications that rely on the sense of they have their own host name and domain. The username space is really one of the, I guess, most recent of the six. The work has been done and even continues to mature, but most of that only has been around in the Linux kernel and in your distro of choice maybe has only appeared in the last few versions, and so username spaces says, I've isolated all these other things. How about even the user ID and group ID range is shifted from the host such that my process appears it's running as a certain UID, but that shifted such that if I were able to break out or see other file system mounts or processes, I would be fully unprivileged from the rest of the system. You combine these six, and I don't know if you understand the graphic, but if you ever bought a flat pack box from IKEA, so you fold this up and you now have a physical box, and so these namespaces in essence create this isolated box around your process and close it from the rest of the system. C groups, the picture I like to use kind of are the elements that allow my box to shrink or grow as necessary because C groups is about resource limiting, so it's great that I've been able to isolate my process from everything else on the system, but I also don't want that process to be able to use all the CPU or all the memory or all the IO bandwidth, and so C groups is another feature of the Linux kernel that has been maturing over time. We'll talk in a few minutes about new versions of that that have appeared in the Linux kernel with more capabilities, but you can basically think of C groups again as this control mechanism. I've both isolated my process, but now I also want to limit its capabilities for how much of the host system it can use, and so when we combine these two concepts, this is really what we, most of us should think of when we call something a container, it's namespaced, isolated process that also could be using C groups to control its use of the host resources. So this is part of a talk that I've been giving of various meetups. One of the things that people want to know when they come to containers, and as Solomon said just a few minutes ago, security has been both a hot topic, but also a fast-moving area such that a year ago, things that we talked about in Tokyo have changed drastically even in this 12-month cycle of both improvements in the Linux kernel, improvements in engines like Docker or Rocket or LXC, but here are the things that people are curious about when they come to containers, and first on the left, we'll think about our use model because that may affect how we consider what we need to care about with security. So in a single-tenant trusted code world, there may be less concerns about malicious attack vectors because my containers are all running on the same host, no one else's containers are running there. The code is all from my organization. I've vetted how that code gets into my system. Moving from that to a multi-tenant but sort of organizationally trusted model, it's all on my intranet. These are all various tenants of my organization. Now I have to worry a little more because I've got resource use from different organizations. They want to make sure that their resource use is validated. So if they ask for .5 CPUs, that there's not another set of containers that's going to use up the CPU and sort of the noisy neighbor problem. And then multi-tenant untrusted code, a public cloud model where I am sharing host and allowing containers from different tenants. This is all, obviously you're going to have a lot of the similar risks that you would have to an internet-facing application with this added concern about malicious neighbors. And so IBM has been operating for a few years a public cloud container service where Solman and other folks from our research team have been looking at all these issues and problems. And some of the capabilities and enhancements in Docker in the Linux kernel have come out of this work. So quickly some of the threats, you can read them here. I won't read them all for you. But obviously we are going to be concerned with how we protect our host and C groups are going to help us with that and other features that we'll talk about later. We're also going to be concerned with a container's access to information on the host that they shouldn't have access to, newer features like set comp, app armor, LSMs will obviously enable us to protect containers from accessing that. Containers have the shared kernel model. So all the containers on a single host are sharing the same kernel. So we don't want underlying kernel information or even the ability for a container to modify information about the kernel. Most of the container systems that you're going to use have an API. Many people having considered what it means to expose that to a large number of users. So how you provide that access, how you limit it is important. And then similarly to anything else that we're going to package up and virtualize, we have to consider what's inside this code base to have libraries, to have other binaries, other code that has exploits, that has latent, unpatched code. And so the container communities have done a lot of work in the last year on that that we'll look at. And then container provenance, how can I validate, certify that when I ask to run application A, it's really built from the code base from my source code repository combined with some base image that I've vetted. Is it what I expect? And are there features of container runtimes that help with that? And so these are all the things that people have been asking about considering. And we've seen a lot of activity in the community around that. So at this point, as Salman said, we could go through sort of a laundry list of talking about these features and what's new. And we'll definitely itemize that toward the end. But we thought it'd be great to really do this comparison. What does it look like for an operator to deploy Apache onto a host versus containerize Apache and deploy it onto a engine like the Docker engine? There's a lot of information out there from, for example, the Center for Internet Security that provide details and best practices to an operator and administrator on how I should configure my OS. So if you're using REL7, you can go download their report. And you can climb through 304 pages of recommendations up into 296 pages. You can download their Apache guide and follow all the best practices across those 163 pages. Docker also has a CIS benchmark document, which has now been updated for Docker 112. Again, 189 pages there. The OWASP rules, again, following these practices you would go through. And as you set up your Apache server, you would make sure that you've configured it according to the best practices for security. As you can see, this is, as we say, hard and tedious work. Will containers solve all these challenges? No, but as we walk through this, hopefully, you'll see where containerization and the improvements to container engines have provided a lot of shortcuts or provided you with capabilities that decrease the need for a lot of manual effort. So as we said, we're going to look at Apache based on these security best practices. It's not that containers take away the need for hardening the application, making sure that it's secure. But as you do this and duplicate this over multiple hosts, obviously, the work, if it is manual, becomes duplicated or needed to be automated through some other system. Containers were positing, make it easy to add the security to containerize these best practices and make it usable for both your developer and operator admin. So let's start. Apache web server, obviously very popular. Deploying it in a container, we believe, as we walk through this, can be more secure in a containerized environment. So let's look at the steps involved to do that. Obviously, we're going to have to configure the host file system. We're going to have to configure the partitions. So as we walk through this, just to give you some clarity, there's a lot of content on these charts. We'll make them available later so you can dig into them in more detail. But on the left, you're going to see the steps for a Docker host. On the right, you'll see, again, it's not VM or bare metal. It's just whatever you perceive as a system. Obviously, there are, in some cases, going to be some more steps because in the case of containerization, we now need a Docker engine or your container engine of choice. In this case, you've got Docker, which now has container D and run C. So these things will have to be installed and configured properly. Again, on your host, you're going to have to configure that file system and configure the partitions and have something like SSH available to do this work. So step two, this is one of the first places that we'll talk about specific container features. Actually, from IBM, we contributed the username, space support in the Docker engine that appeared in Docker 1.10. And so one of the things that you might have to make a choice about with Apache is Apache on a host is going to run as a non-root user, importantly, because you don't want an escape in your web server to be able to now have malicious privilege on your host. With user namespace support in the Docker engine, we could use the capability in Docker to package a container and run it as a non-root user. With username spaces, that ID space has already shifted, such that Apache will perceive that it's root inside the container, but it won't be root on the host. So that's obviously a capability that gives us a step that we don't have to worry about in the containerized environment that we'd have to worry about on the host and gives us a whole deprived model of this application, even if there was an escape to root from the other ID with username spaces, we wouldn't have to necessarily be concerned about that inside the container. Salman, you want to run through some of the next steps? OK. So we have configured separate partitions for Apache. We're installing Apache on a host, and we have configured separate partitions for Docker. So separate partitions for Docker and separate partitions for Apache. And then we have configured a separate user for Apache and configured the container engine, in this case, Docker, to start containers as non-root. Then we want to set up, as an operator and developer, we want to set up various monitoring tools. We want to set up logging so that logs can be collected from the containers running, from the Apache running inside the container, or if we are running Apache directly on the host. We want to configure audit demons so that system call activity can be tracked. We want to configure monitoring. The key point here is that if you were running Apache directly on the host, and we will be configuring all the monitoring, that will be, in some sense, specific to Apache, or the logging that will be specific to Apache, or will be specific to an application. But when you're running applications inside a host, all this stuff is configured for a container, and containers just have to write their logs. And they can be collected by the logging demons as well as the audit demons that track system call activities. So in other words, if your host gets compromised, when you're running Apache directly on the host, then all your other information about where your logging servers are, maybe keys, also get compromised. But if your Apache container gets compromised, the attack surface is reduced. When configuring, so step four is that one has to configure these various network services, like NTP, firewalls, DNS. One key difference between running Apache inside a container and running Apache directly on the host is IP forwarding. The center of internet security guidelines require that IP forwarding be disabled on a host, and that is easy to do. But when you're running a container engine, such as Docker or Rocket, the IP forwarding on the host has to be enabled in order for the network to work with the default networking. However, one can disable the IP forwarding inside the container itself by either configuring the container engines, Docker engines, or bypassing appropriate parameters to disable the IP forwarding. So step five is which one has to do both on the containerized host as well as the host running Apache. One has to set up software updates or malware agents. And that's a step that's common to both. That has to be done regardless. So Phil talked about user namespaces and how we should run an application as non-root. So that's part of the so-called discretionary access control in Linux. Besides that, it's a prudent thing to configure mandatory access control in Linux when you're running various applications. So the center of internet security profile for Apache requires that when we are running Apache, we should configure the appropriate app armor profile for the Apache web server. Configuring that is difficult. One has to run the profile and see whether the profile is going to work, because sometimes it can interfere with the application. But the container runtimes, on the other hand, provide a default mandatory access profile that can be used with all the containers. And that profile can potentially be tuned for a specific application. So what does these profiles protect against? So as an example, the sensitive information related to kernel is typically present in the proc file system or on the proc syskernel. And the app armor profile of container engines, such as Docker, would prevent any read or write accesses to the proc syskernel, which means that those parameters, which are very sensitive to kernel, cannot be modified. So Phil talked about C groups and how C groups can control the limits available to a container. So imagine if Apache was running on a host, and suddenly Apache started misbehaving. It could start maybe creating too many processes, sending a lot of network traffic out, maybe going, consuming a lot of CPU. And while it's possible to limit the resource consumption of Apache using facilities such as SystemD or Upstart, that work is error-prone and tedious. One can see here, I listed some properties, such as one can limit memory limits using SystemD. So that can be done, but that's difficult. Container runtimes, on the other hand, make provide a nice and developer-friendly interface to use these control groups to limit the resources that are available to a running container or the Apache web server running inside a container. So they can limit CPU, memory, IO bandwidth, process forks, and they can tag network traffic with certain class priorities, which can then be shaped by Linux tools such as DC. So we're running the Apache web server as a non-root. We have running it on different partitions, and we have taken various other measures to isolate the Apache web server running inside a container from the host. But what else can we do? There are a couple of other things that can be done, both for Apache running on the host and for Apache running inside a container. This thing called Linux capability. So in the old days, there used to be this Linux root, which was all powerful root. The capabilities of the root have been divided into some fine-grained capabilities called Linux capabilities. And one can configure the Apache web server running as non-root to only use some of those Linux capabilities. And that can be done, but that's, again, manual and tedious. Whereas if you're running Apache directly on a container, most container run times will only use a subset of the Linux capabilities that are available. So applications, as Phil mentioned, there is a single kernel that multiple containers or applications running on a host interact with. And whether applications are running inside a container or whether they are running directly on the host, they will interact with the kernel through system calls. There are 313-plus system calls out there. And some of the system calls are legacy calls that may not be inactive views. And a number of times, the security bugs that are found in kernel are related to those system calls. So one can configure the Apache running on the host, which is already the privilege to further restrict the system calls that can be invoked. Or one can use the container engines, such as Docker or LXC or Rocket, which would already limit some of the system calls that an application can invoke. So that's the isolating host from the application, whether the application is running on the host or directly or inside a container on the host. But one has to configure the application in a secure way also. So in this case, the Apache itself has to be configured to make sure that it is configured with the right SSL protocols that does not have the unnecessary modules, and so on and so forth. And these recommendations apply both to container running, both to Apache running inside a container and to Apache web server running on the host. Although I will show shortly that it's much easier to provide deeper visibility into the application vulnerabilities and configurations that make these containers much better in terms of security. So once we have done all of these steps, a host is isolated from a misbehaving container. So in the event of a compromise, the host can still function. In all the data on the host, all your key passwords that maybe your Jenkins had installed and configured for various things are still protected from a misbehaving container. So it's a good thing to run applications inside containers on the host. And the same mechanisms that we have used to isolate these containers from the host can be used to isolate containers from one another. So if you were to summarize this and talk about the defense in depth, so as you can see, these capabilities that I've talked about in steps 1 to 10 allow a defense in depth. And the key takeaway from this slide is that if you are using running applications inside a Docker container or inside a Alexi container or a Rocket container or whatever your favorite container platform is, they make it more easy to use to consume the Linux capabilities to Linux features that isolate hosts from the running containers. Whereas if you're running the applications directly on the host itself, all these configurations have to be done manually by the developer, not on just one machine, but on all the other machines that the application is set up. And these configurations then become per application, which adds to the amount of work that has to be done. So we have observed from production that application misconfigurations are among the top causes for container compromise. No longer in containers, you're not supposed to run SSH. So probably developers are running their applications. And these applications could be misconfigured. And so wouldn't it be nice if the container platforms, which allow developers to run containers, provide some mechanism to check for vulnerable packages for applications, or check for application configurations to make sure they're running with the right TLS or right ciphers, et cetera? So there are many solutions out there in the market providing vulnerable packages, IBM's vulnerability advisor, which is linked with the IBM Container Service, provides such mechanisms to detect vulnerable packages and also application misconfigurations. So what are some of the new security issues that arise when running applications inside containers? So Phil is going to talk about that and take you to the rest of the slides. All right, thanks, Solomon. So I think we have just five minutes left, so I'm going to go a little quickly through this portion to see if we can save a couple of minutes for questions. Obviously, there are some considerations that you have to make decisions about as you move from maybe a traditional model to containers. One is this whole concept that many container engines run. Your application is root, so the Docker file and other models of creating containers can provide ways to work around that. User namespaces is, we believe, a key way to truly isolate and deprivilege containers. There are other considerations here, like the IP forwarding that Solomon mentioned. The whole concept of who are you going to give API access to your container runtime? Because obviously, that API is powerful, and if it's not limited through capabilities such as authorization, authentication, you can basically provide root access simply by providing full access to the API. So there are those considerations. But what we've seen and what we believe is that the evolution, even in the last 12 to 18 months, features and capabilities in this slide is focusing on Docker. Many of these features underlie or actually overlay on Linux kernel improvements, such as some of the PIDs limit C-group limitations. So I can now protect against fork bombs. That came in kernel 4.3, C-groups V2 that Solomon mentioned earlier. No new privileges, which, again, is a kernel feature now exposed in both RunC and Docker. And then there are other capabilities that, depending on the engine that you're using, have added things like content trust or content addressability, so validating that the layers of an image are exactly what you expected. Quotas, so again, handling the fact that containers are basically file systems that you're downloading, and you're now allowing a container to write into that file system how to limit that. And so on. One great resource, if you're actually looking to compare how these different engines are applying and trying to come up with sane defaults, you can go download the NCC Groups Understanding and Hardening Linux Containers, which was updated again earlier this year by Aaron Grattifiori, the author. He walks through, and again, there are just hundreds of pages here, but it's extremely well researched and detailed. This is just kind of an overview chart that toward the end he shows what he's finding today, as far as the defaults in these different engines. And I think one of the things that Solomon alluded to, you can always write your own SetCom profile. You can always write your own app armor profile and set up a container deem in the way you want. But it's actually great for these projects to try to come up with sane defaults, knowing that many users won't have time or capability or understanding to create those custom profiles. So what's still happening? There's still lots of Linux kernel activity around the container space. There are still parts of the Linux kernel that aren't necessarily namespaced. And so if you get on the containers Linux mailing list, there's discussions every day about different pieces of the system. Should they be namespaced? Should they be included in some of these isolation mechanisms? And so many times, it's good to remember that container security work really equals Linux security work, Linux kernel security work. And there's a lot of work going on with hypervisor isolation. Maybe you've heard of hyper.sh or other Intel Clear containers. IBM research has been doing some work in this area as well. If, again, we're running out of time. In fact, we're basically out of time. But Run C is now a pluggable run time under the Docker engine. So if you implement the open container initiative interface and use a shim hypervisor, today you could run Docker and have lightweight hypervisor isolation as well as the kernel isolation. So with that, we're done. Maybe we have time for a quick question before we go. Anybody have a burning question over here? If you want to come to the mic quickly. And then we'll probably dismiss after that. Salman and I will be up front if you want to discuss more. Do you have any numbers for cost for isolation for each and every container? Because you have so many security features. You have a CPU cost for this, for container. Yeah, I haven't seen any concrete measurement of this. Most people are finding that compared to many other virtualization technologies that it's a very small cost. But that's worthwhile, maybe for a future bit of research. One thing I should mention is that if you were in the VM world and you were running these applications inside the VM on a hypervisor, you have to set up all these security agents per VM. So now instead of running one security agent or one patching agent per VM, you just have one agent per host. So you can increase the density of applications you can run on a host by just having one copy of all the various security agents that had to be done. But quantifying that cost, I think still needs to be done. All right, well, thank you very much for your time. Thanks, Salman.