 Thank you for having me here at the scale. And my name is Chris Vantyne, Chief Technologist for the Western Region at Red Hat. I've been with Red Hat for about 10 years working with strategic partners and customers who are adopting Red Hat's emerging technologies. Red Hat has certainly moved well beyond Linux in the middleware, software-defined storage, virtualization, container and cloud management. And certainly one of the hot topics with our customers is around containers. And everyone's talking about Docker or Linux containers. Customers are asking us questions. Are containers ready for the enterprise? What about security implications of deploying containers in the enterprise? And what are the tools that I could actually use to help me manage security compliance, checking for security vulnerabilities? And so today we'll be talking a little bit about OpenScap. But before we dive into the technology, we do want to dive into, you know, why Linux containers in the enterprise? What's really accelerating the adoption? What are Linux containers from a technical perspective, as well as container security? And then go through some example use cases and the different tool sets that are available with OpenScap, which is an upstream open source project. You know, does anyone recognize what this is a picture of? That's correct. It's the Tesla assembly line. And the reason I show this is that, you know, when was the last time cars actually improved after you left the dealer a lot? That's exactly what happens with Tesla. And software is a huge enabler of making that happen. You read in the papers how they release a software update to improve the 0-60 acceleration time, update navigation systems. They also are able to do vehicle recalls, if it's a software issue, over the wire without having the owner bring in the vehicle into a dealership. And so that's saving tremendous amount of money and totally transforming the automobile industry. And so software is a huge part of that. And this need for accelerating software application delivery from development into production is being driven by business need. And so IT has really got to respond to this. And Gartner calls this the era of the Mode 1 and Mode 2. Mode 1 being your traditional applications, maybe it's an Oracle database, really relied on the underlying infrastructure. It wasn't updated very frequently, maybe 12, 18 months. You see that on the left side. And there's a shift in really the how, what, and where of application development process, application architecture, and the underlying infrastructure. And containers are a big part of that because they're enabling the business to accelerate application delivery and improve overall quality. And so there's a key problem facing IT today. We talk a lot of enterprises and they have on the left, you have the business really driving the need to move faster like Tesla and deliver updates to their customers. And they're putting a lot of pressure on the developers to write that code. And the problem with the developers, they want to have the access to the latest technology, the latest tool sets. But on the right side, in the enterprise, as you have operations in security and compliance, they're trying to maintain uptime, they're trying to maintain security, and it's causing friction between the two sides of the house. And so how do we eliminate that friction? And containers are a big part of that. So containers, the promise is that they're going to improve application delivery and enable the developers to have access to the tools they want, and be able to bring in the dependencies for those applications and allow them to control that. Yet at the host level, allow operations to have control and lock down the systems in terms of a security perspective. And so this is the end state of where you have the developers doing what they need to do and delivering to the business in a timely manner. And operations having the ability to have their environment secure and meet the security and compliance standards without having this friction between the two groups. And so that's the promise of containers. And containers, really the key thing there is that it allows you to package once and deploy anywhere. It can take a container and deploy it on physical, virtual, private, public cloud. And it provides a lightweight isolation of the process, the network, as well as the storage. And containers have been around for a while. So Layer, Zones, Linux has been in the kernel. Google did a lot of early work. But what is around, until recently, the past couple of years when Docker came along and kind of standardized, provided a standard API, an image format, and also a runtime, but also a delivery and sharing model. And so those have a lot of security implications as well. And one of the things about adopting and accelerating technology quickly is coming together as an industry around standards. And so that's where the Open Container Initiative, which is a part of the Linux Foundation. It's really a consortium of all these different companies who came together and said, hey, we need to standardize in terms of the Linux container technology in terms of the packaging format as well as the runtime. And so they took Docker's standards and made those the standard of this consortium. And so as a result, we can move forward and continue to innovate at a much faster pace. And so when you take a look at containers from a security perspective, really three things. One is you're building the containers. As a developer, you have a Docker file. From a best practice, we want to keep those Docker files in a source code repository. So you have versioning and control. Secondly, you ship those containers, images, into a shared repository, whether it's a public or private. A private repository provides a lot more control. Also, you know exactly what's going in there and who put it in there versus a public registry. And then lastly, you're going to be running these images on your physical, virtual, cloud systems. And then there's security implications for having many containers running on the same host, which we'll get into. And so from a technical perspective, what's the difference in terms of a container architecture versus a traditional system? On the traditional system, you have an application, maybe application A and application B, and they're sharing all the different libraries on the host operating system, as well as in a virtual machine. Each virtual machine has its own dedicated kernel. Now take a look at the container side. When you build the container, the application pulls in its dependencies into that container package. So you can have different versions of libraries all dedicated to a particular container and have those run side by side. But a big difference here is that these containers share the exact same kernel. The container doesn't have its own dedicated kernel. So very different than a virtual machine. And so think about from a security perspective, now a security vulnerability in a container environment can not only impact container A, but also can impact container B and the host as well. And so patching becomes that much more important. And the kernel is that much more important as well to the overall environment, all the applications. It's not isolated. And so what is the underlying technologies that really bring containers to the real world? And there are several key technologies. First off is control groups. This provides resource management. Google did a lot of early work around this. It provides the ability to limit the consumption of CPU, memory, disk and network. So providing quality of service, which is really important, especially in a public cloud where you have different customers running on containers on the same system. And you want to be able to provide guarantees in terms of quality of service in terms of CPU availability, memory, etc. Secondly, namespaces, they provide process isolation. So this is a namespace such as a user namespace. The user ID in a container of let's say root is different than the user at the host level. And so this is an area that's being developed in the upstream community right now and is technology preview in terms of user namespace. And we'll talk a little bit about other namespaces in a minute. SE Linux provides additional security. So if you're here for Dan's talk prior to this, Mr. SE Linux over at Red Hat, he talks about how SE Linux provides mandatory access control and allows Red Hat to deliver containers in a much more secure manner because it basically enables the container to be locked down so that if there's a security breach in the container, it doesn't negatively impact other containers or the host running on that system by limiting the files that can be accessed by that container user. Also, another key part of security around containers are the images themselves. So Docker container images have different layers. So you typically would start out with a base layer. You want to make sure that that base layer such as rel7, sent off, etc. You want to be sure where are you pulling those images from? Is it a reliable and a trustworthy source? And then in the container image as you write or add applications or make changes, those layers are layered on top separately from that base image. So let's talk about container security. Does anybody recognize some of these brands? Well, here are some of the most recognized brands in America and what do they have in common? Is that in 2014 they experienced some of the largest data breaches. They represent over a billion data records that were hacked in 2014. And so what were the top reasons for these data loss, right? Malicious outside attacks, accidental loss, but also malicious insider attacks. And so we did a poll, a forester poll to know whether the top challenges for enterprises adopting containers. And certainly here you see security is the number one concern. So as you deploy into enterprise, you need to definitely have a sound security strategy policy as well as tooling to implement that. And so over the years I've worked with many customers when I first got to Red Hat, I'd asked them about their security practices. And the common response was, you know, we have a firewall. That is our security process and tool. But as we saw, malicious attacks on the inside are also a key concern. And Dan speaks about SE Linux and containers. And containers do not contain. And that's a key thing to understand. And they don't contain because not all resources are namespaced in Linux. So you have the kernel key ring which is shared amongst the containers, the kernel itself, and all the modules, any device, the system time, and then Red Hat Enterprise Linux user namespaces as well. It's in tech preview of turning those on. By default it's disabled, but you can enable it by specifying the kernel boot option that's listed here. And so what does this mean is that, you know, today the root user in a container is the same as the root at the host level. So if you get out of the container you then have root access to the entire system and all the containers running on that system. In Red Hat Enterprise Linux 7.2 it's tech preview. So in Red Hat Enterprise Linux 7.2 and sent off, it's in technology preview at the kernel level. It hasn't been enabled for Docker yet to take advantage of that. So from a Docker perspective, or other distributions where it has been enabled. But Red Hat is focused on enterprise so we want to make sure that this new technology is hardened and ready for prime time in terms of enterprise deployments. So it's an active area of work upstream. And certainly, you know, there's parts of it that have been backported into REL. And if you specify this boot option you can try it out in the REL 7.2 which is the current release. So what are some of the security risks with containers? You know, kernel exploits. So if you exploit the kernel you didn't have access to all the different kernels potentially. Denial of service attacks. Container breakouts. Poisoned images. We'll talk about that in a minute. Also compromised secrets because being able to specify those in the Docker file. And you know, why does the container image matter? So we did an analysis of the container images on the public repository Docker Hub where you can go and pull down images. Many of the developers probably in your enterprise are doing that. And you know, 36% of those images had a high priority security issue such as Heartbleed or Shell Shock. 28% had a medium priority. So 64% of the images publicly available, that probably a lot of the developers are pulling down into the enterprise, don't have security issues. And that's just at the time of pulling them down. Just think about what happens to that image over time. And let's take a look at that. So here is an example of different container images and the dependencies that they pull in into that container image. So for example, we have a C application on the far left that pulled in glibc and bash, Java bringing in jre, noJS application, and then a Perl PHP application. You can see the different components it pulls in to the container image. Well these little triangles show you the number in there of how many times a security update has been released for that component since RHEL 7.0 GA, PHP over 20 notifications and security updates. So the key thing here again is at the point of time you pull it down it might be completely secure from a vulnerability perspective, but six months down the road you may have 5, 10 different vulnerabilities that need to be patched. So what's your process to update these containers over time? And here shows you in terms of how many vulnerabilities per package last in 2014 for the different components and the kernel number one in terms of having security issues. Also Java, another victim of security issues as well. So the kernel is so important in a container environment. You really need to have a security process and tooling to make sure that constantly update it as well as the programming applications. And so this brings us to OpenSCAP, right? OpenSCAP upstream project to address compliance and vulnerabilities. How many have heard of NIS? So NIS they provide a government database of CVEs basically releasing to the public updates about security vulnerabilities to different software packages. They also have checklists to help you define policy for your systems to conform with security standards by the government agencies. So CVE what it looks like is it has the impact, important, critical, etc. the release date and in Red Hat we provide the associated bugzilla and then also the details around the security vulnerability. So as you're maintaining your environment you have a good idea of why I would actually want to address this issue and how critical is it to my environment. And then CVEs is a checklist database of different policies that are recommended to keep your environment secure or your container secure. So in this example there's a check to make sure how long is your minimum password setting on your Linux host. And so OpenSCAP consists of really three things, content, scan, and reports. Content, Red Hat is providing security guides for our distribution. Other distributions are also providing guides as well. The CVEs are driving the checklist for compliance. And then the CVE database is really driving the vulnerability notifications as well. And then there's the tooling. So OpenSCAP will go into the different tools in a minute but it's a set of different tools and technologies. Formin is one of the tools that it integrates with which is an upstream provisioning project for hosts, virtual machines, etc. And then the scan outputs different reports about your system or set of systems in terms of is my container, what vulnerabilities does it have, what packages need updating. And then also from a policy compliance perspective what checklist failed in the scan in terms of maybe a minimum password, what services are enabled that should be disabled, etc. And so the OpenSCAP tool set really consists of currently five different key things. One is the OpenSCAP base project. This is the CLI in library. And we'll go through some of the use cases. OpenSCAP daemon, this is a service that runs in the background. It can automate the running of scans. SCAP Workbench is a way to actually edit and define and customize policies for OpenSCAP. SCAP daemon is a middleware that can be integrated and today integrates with Formin, provides a database so that you can have a history of your scan information. And then OSCAP the Anaconda add-on is a plug-in essentially for the installer, the Anaconda Linux installer, so that when you install a system it can run the scan, the policy check at system creation. So you start on a good foot and we'll get an example of that as well. So let's take a look at OpenSCAP base and some of the use cases. So first use cases, you want to be able to scan for compliance. Things you might be wanting to do, our password quality requirements set, our obsolete services enabled like Telnet, you want to make sure that's off. Is OpenSSH properly configured? Is Flash Temp on a separate partition? And so to do this you would install the OSCAP utilities and there are a variety of command line options to customize the output of where it's going. And then an example in real time of running the OSCAP tool shows you each individual check and whether it passed or failed. So this is running on a host system. And then as an output of that scan you can then go into a web browser and view the evaluation report. It has some information about the particular host. But then you can see an overall compliance and scoring for this system. So here you can see 34 out of 68 checks passed. That's a lot of failures. It also shows you the criticality, 3 high, 16 medium. And then I can drill down and see where those issues are. And then line by line I can see the individual checks that are in the policy. Again this policy is customizable. And the nice thing is that you can actually drill in into each check. You get the details about the check. There's also a remediation script that can be applied to the system to bring that container, that virtual machine, that host into compliance with that policy. So you can press the button and it will run this script on the host. Or you can just copy that and customize it a little bit and run it as well. Reflexible. Another use case for OpenSCAP is the scan for known vulnerabilities. These are out of date RPM packages. For instance, what ones need updating was the criticality of these vulnerabilities. What's the vulnerability? And what CVs have not been addressed on this particular host? And so I can go ahead and pull down the security content. I can then run the vulnerability scan here with the OSCAP using the Oval option now. And then you can actually view the report. And in real time you'll see the output on the command line of every CV check and whether it's false or positive. And then once it's done, I can go ahead and view the report in the web browser. And so you get some overall information on the particular host. And then line by line I can see where the CVs failed and I need to actually patch these systems to resolve that issue. So those were examples of running on a host or a virtual machine. But you also can run these checks and these scans on containers. So you want to know is the Docker image compliant? Is the image patched? Is the Docker container compliant? So the image being at rest, the container being running. So you can do it for either use case. The first thing you want to do is install what's called the OSCAP Docker utility. So these are all examples on CentOS or Red Hat Enterprise Linux. So I did a yum install. I do install Docker here. And what I'm going to do is fire up a Docker image as well, rel6.2. So I'm running 6.2 on a rel7 host. And I can do offline and online. So the first example, I'm going to check a Docker image that's just on my file system. It's not running as a container yet. And I can do a compliance scan or a vulnerability scan. So there are a lot of ways you can automate this or run it at scale, maybe against your repository. And then also maybe you have a production environment running containers. You can do a scan on those containers that are running as well without bringing them down. So you can do a vulnerability scan or a compliance scan. So the slides will be available if you want to look at the details in terms of the command line options. And so then there's also the OSCAP daemon. So this is a service that runs in the background. You can evaluate machines, containers according to the schedule provided, so like a Cron schedule. You may want to continually evaluate a container or machine for a specific policy. Today it's available on Fedora. It will be available on other distributions as it matures. I mentioned the OpenScap workbench. So this is a GUI tool. Easy way for average user to really define policies, customize them for a single machine or a remote machine. And you can actually do remediation through this tool as well. It's available on a lot of different operating systems, Fedora, REL, CentOS, Scientific Linux, Debian, Ubuntu, and even a Windows and OSX version as well. And then here's the OSCAP Anaconda add-on. So just showing you an example. This is the CentOS or REL installer. So as a part of that process you'll be presented with the option of selecting a profile from a security perspective that you want to apply to this box. And so as a part of the install process it will remediate any issues in the base OS through the installation process. So let's take an example of the value this is providing. On the left side is doing a base install on REL and then running this particular policy check. And it was only at 64% compliance score. This is with selecting the policy and having it applied during install process and then running the scan after. And it was roughly 99% compliant. So from the get-go your systems are secure and not exposed to any vulnerabilities right away. So good practice for enterprise deployments. And I mentioned SCAP Demali. This is the middleware. And it today has been integrated with Foreman which is the provisioning solution for bare metal, virtual machines, etc. And so this provides a dashboard to manage not just one system but hundreds or thousands of systems. So you get a summation and a database. So you have an audit trail and records of these scans over time. And so this is just an example showing you the different hosts. You can see the pass failed and then you can actually drill in and view the report. So upstream project, open source, Foreman. And so what are some of the container best practices? So first thing is only run container images from trusted parties. You are deploying in the enterprise. You want to know where are your developers pulling these images from? What's inside these images? Who built these images? Also running containers as root, not a good idea. So drop privileges as soon as possible in your containers especially until user namespaces is fully implemented and supported. Host operating system matters. The kernel is critical to a container environment even more so than before in a virtual environment. So you want to make sure you have a solid kernel. You want to make sure that you are updating that kernel and you have a reliable source and provider of security updates for that kernel. Again, apply kernel security fixes because of that on a regular basis. How many of you disabled SE Linux today? In a container world, not a great idea because if you hack the kernel, you get access to the host and all the containers. So by having SE Linux policies, it will restrict where they can get to. And that example we had earlier, over time container images will have more security issues than when they were initially pulled down from the repository. So you want to have an ongoing process to scan and remediate any issues in the containers in terms of security vulnerabilities or policy failures. Additional resources, in terms of best practices where it has a security guide you can download on our website. In terms of hardening SE Linux, Dan had an earlier talk and a lot of great tips and overview of SE Linux. He also has a coloring book to really simplify and make it easy to understand SE Linux. Audit logs, syslogs, systemd, journald. Identity management, so having user management, security. Security blog, we publish a blog with different security tips on Linux, but also containers, Docker, etc. A great place and resource to check out. The three pigs coloring book that Dan Walsh at Red Hat who leads our Docker engineering team in SE Linux has one on containers as well as SE Linux. And the container one talks about the three little pigs story and how that relates to containers versus virtual machines versus hosts. Really showing you that a container kernel, there's a single kernel that impacts all the containers whereas a physical machine has its own dedicated kernel as well as a virtual machine. So really important to understand the implications of that. And that's all I had today so thank you very much for attending today's session. Happy to answer your questions about OpenSCAP Project. I encourage you to check out the OpenSCAP website as well. You can get a lot more detailed information. I think there's some demonstrations, tooling, and then also community as well to get help or even contribute to the upstream projects. So thank you very much. The policies are defined by the government and then we are a distributor for our distribution of these. So there's different, there's like PCI compliance, there's standard REL security guide, etc. So we've also customized these a little bit but the original source of this content is this. And the workbench allows you to customize them and remove certain checks that may not be applicable to your environment. Without actually modifying the files, it's actually just kind of, it's a layering technology. So yeah, not edit the file. So it's tech preview. And they need to enable it with Docker. Typically tech previews, I don't think we've committed on a date or a release yet. Kind of get customer feedback and see how it goes. But typically, certainly in the REL 7 timeframe, I would anticipate. Foreman can do that for instance. So it has the ability to go into all the systems like an agent base, satellite server for Red Hat. Allows you to do that. Oh, it's the latest version. It has SCAP integration. Any other questions? Great. Well, thank you again.