 For those who weren't here for part one of Chris's presentation, this is Chris Fentine. And he's a chief technologist at Red Hat, and he's going to be talking about a DevOps state of mind continuous security with Kubernetes. So with that, give a round of applause for Chris. Thank you. All right, welcome to Continuous Security with Kubernetes. This builds upon the previous talk in Focuses on Security. And we'll talk about how that can be enabled with DevOps and Kubernetes as well. So first off, in the industry, there are a lot of security breaches going on in the newspaper. This slide represents a billion data breaches back in 2014 to some of the marquee brand names in America. So significant impact causing a loss in value to those businesses. And when you take a look under the covers, what are the drivers or what is causing these security breaches? First off, employees not taking proper security measures was number one. Number two was due to outside breaches, three unpatched or unpatchable. But commonly, we find that in an organization, it's those handful of servers, those legacy ones that are very old version that are forgotten about in their automated processes, and they aren't getting patched, or they don't have patches available. And then lastly, internal attacks by the enterprise, or the own employees. And so as I talk with customers, when I first started out at Red Hat about 13 years ago, I talked with operations about their security best practices. And they typically would say, oh, we rely on the firewall to provide that protection of our applications. And certainly with hybrid cloud, public, or private cloud, moving to a software-defined network, relying on that firewall, just simply you cannot contain your security breaches because you don't own that particular physical infrastructure anymore. And the second point is that the discussion around security is no longer just an operations issue of ownership. It needs to broaden out across the organization. And so that leads to DevSecOps. DevSecOps is about inserting security into DevOps. It means having a security first mindset as a part of your organization, across your development, your testing, and your operations, across the business as well. Also, integrating security in an automated way in your process, your CICD pipeline, how do you incorporate security so it doesn't become a burden to the developers? It's automated. And then leveraging automation tools to enable a secure infrastructure from end to end. And so the advantages of DevSecOps, one is it reduced the risk by catching these issues early in the process, lowering the cost of resolving them, also speeding up the delivery of your applications in a secure manner, and being able to react quickly to any security issues in your production environment. And we do this through security automation, process optimization, as well as continuous security improvement. And we'll talk how you can do this with containers and Kubernetes. And so when it comes to securing containers, here are some of the key areas to think about in terms of security. One is around the images, the builds, the registry, your CICD pipeline, as well as the container host. And so let's drill down on these. First off, container image security. So we learned in the previous session about the separation of your code, your configuration, and your data. This is improving your security because you're not storing passwords into your container images. You're keeping that separate in a Kubernetes secret, for instance. You're also isolating the data, centralizing it in a data service or persistent storage. And with the container image, you're reducing the footprint of your application by just pulling in the immediate dependencies rather than a full-blown operating system as in a virtual machine. Secondly, the shift to containers has a big impact in terms of what's being delivered through the pipeline. In a traditional non-container environment, the developers are traditionally just delivering, for example, in the Java world, a jar file to their tests and then to production. Well, as you move to containers, that container image not only contains the application, but it also contains the runtime for the middleware. It also contains the OS dependencies. Now who owns the security of those layers? Who is responsible for maintaining security patches at the runtime in the OS dependency area? We'll talk about that. And this is important because as you build out these container images, there are potentially security vulnerabilities to these components that are being pulled in at the runtime or the OS layer. For example, from left to right, we have a C, a Java, a Node.js, and a Perl PHP application. The orange boxes indicate the components that were pulled in and have a known security vulnerability. And that second column for the Java app, the JRE has 66 security issues, notifications since the major release of RHEL 7. And it's just growing and growing. So as a developer, I'm pulling down these images. I need to have a process to scan my image, not just when I pull it down, but over time as well. Another best practice is to sign these images. In a production environment, I want to make sure that my images come from a trusted source, a known source. And only those images are started. So in Kubernetes, you could actually check to see if that image is signed from a trusted source before it started in the cluster. Also, I can building out my container images. So we talked about the different layers in the container image. What about ownership? Well, the nice thing is with Docker is you have build file. That build file can be used for each layer. In today's world, I'm using different technologies, kickstart file for the OS. I'm delivering the middleware and a zip file. And I'm delivering the application in a jar. Well, now in a container world, I can define all three of those in the Docker file. Now I'm talking the same language, the developers, the middleware, and the operations folks. And that image can be built upon build images for the OS, for the middleware. And I can layer on top my application. And then the respective teams of middleware and the core build team or the operations, they own their layers. They own the responsibility for updating that layer for security fixes, bug fixes. I, as a developer, are not taking on added responsibility from a security perspective, because I'm just building upon what my middleware and operations team provide. Some best practices when it comes to builds. Treat that Docker file as a blueprint, version controller, put it in Git. Don't log into your container instance. Make changes and save it and call that your image. I have no idea what those changes were made down the road. Also, be explicit in the version numbering. So it's reproducible. You get the same version every time. Also, in terms of where you store these images, this is probably the first step in an enterprise is they want a registry in which they can store their enterprise images. And the reason they want to do this is because in the public registry, 64% when we did a scan have a higher medium security issue. So if your developers are pulling from public repos, you need to be aware there are probably a lot of security issues in those images. And so that first step of building out a private registry within your enterprise is typically the first thing I see operations team to try to insert themselves in terms of the conversation of containers in their environment. It's usually the developers are already experimenting and trying out. How about integrating security with continuous integration? So we talked about the CI CD pipeline, continuous integration, build, and deployment. From a containerite perspective, this is about the developer checking in their source and to get. That then builds some RPMs. Those RPMs are then used to build a base container image, for instance. You then put that image into a registry. I then deploy that out to my production environment and pull that from the centralized registry. So how do I integrate security? What are some steps? First off, operations and developers defining those build files, storing it in Git and version control. I also want my container images to be reproducible. Every time I build a container image, I want the same output. What steps could I take to improve that likelihood? One is I can leverage what's called a build image. The build image has all the tooling to build that particular type of application. And then rather than store my application in the build image, it creates a target image with just the built application and its immediate dependencies. It doesn't have all the build tools. This reduces the attack vector. It also provides reproducibility by version controlling my build images. So I can go back in time and use that same build image to produce my container image. And then storing that container image in the image registry and deploying that image from dev test into production without rebuilding. Another best practice here is the Jenkins CI pipeline. Dashboard is adding a security phase into my CI process. I could leverage this to ensure my container images meet my security policy. Don't have any known vulnerabilities as well by using a scanning tool. If I have a private registry, I may also want to have a trigger every time I push a new image, it automatically scans that image that's being pushed to the registry. And so one such tool is OpenScap. It's upstream open source tool. There was a session yesterday, it kind of did a deep dive. But it basically provides the ability to do an automated scan for known vulnerabilities from an RPM perspective. And it also allows you to find a security policy, such as like minimum password length. And it checks that container image or instance. It can be either at rest or running. And it will generate a report, an audit trail, in terms of known vulnerabilities, how you could address that by installing newer versions. And then secondly, it shows you where the security policy triggered a failure. And it generates a script. Now it comes in Ansible as well, that you can run against that image to rectify that particular security violation. Also, continuous delivery with containers. One of the changes as you move to a container model is you're not pushing the jar file from Dev into production, you're pushing the container image. And you're not rebuilding it in QA, you're not rebuilding it in production. You're taking what the developers built and put through testing, and then move it into staging and production, as is. And so what does this mean if I have a security issue out in production? How do I address that? Well, I don't go directly and patch my container image. I don't SSH into the container instance, apply an update, and move forward to the next 100 different instances. Instead, you go back to step one, back to development. And I build version N plus 1 with that security fix. And then I push it through my automated pipeline. Why do organizations not do that today? They allow containers. It takes six to nine months to deploy that new major version. Now with this automation pipeline, it can be done in minutes or hours. That's not a big tax to pay when I improve the overall quality and security of my environment by pushing it from Dev into production as is. And so in order to do that in a matter of minutes or hours, I need some automated deployment strategies to deploy microservices that are containerized at scale. And I don't want my developers writing scripts to do this. I want to be able to leverage something out of the box so that I can free up my developers to focus on business differentiating logic. And so in terms of Kubernetes, there are a variety of deployment strategies we're going to talk about that recreate rolling updates and blue-green in this talk. And so first off, one of my options when it comes to deploying that new security update is to do a recreate. What does that mean? That means I have version one out in my cluster. There's a known security vulnerability. I'm going to address that in version 1.2 of my application. I put that through CI testing. And then with a recreate, I take some downtime by taking down version one and then rolling out and pushing out version 1.2, so recreating that cluster with a new version. The advantage here is that it's simple. It's clean. I don't need backward compatibility from an API perspective or the database schema. So if this is not a mission-critical application that can allow for downtime, this is a simple and clean approach to do anything. It also reduces my development from a backward compatibility perspective. The next option is rolling updates. Rolling updates is, again, I have version one out there. And there's a security issue. I want to put that in 1.2, put it through testing. But this time, I'm going to incrementally roll out this new version into my infrastructure. I can also leverage Kubernetes health checks so that it's not automatically added into the load balancer until it successfully passes that check, whether it's TCP, HTTP, or some script I wrote. So once it does pass the check, that service instance will be discovered and added into the rotation. And I can gradually roll it out, 50% and then 100%. If there was an issue, I could stop and roll back anywhere along the way. The advantages to this is that it's really provides me zero downtime. It also reduced the risk if there is an error, because I'm not totally committing to that new version until I gain confidence. And then I could easily roll back at any stage in this process. The con is that I require backward compatibility from a data perspective or the APIs. So it provides a little bit more work for my developers. It's good for horizontally-scaled applications or microservices. Another option is blue-green deployments. So with Kubernetes, I can do blue-green deployments. This is where you have version one out in production. This is the blue environment. I have a security issue. And instead of rolling update or recreating, I'm going to create a second environment, my green environment. And it's easy to do in Kubernetes because I can create these mirrors logical instances of my production environment. It is very powerful. I'm empowering my developers to have a production-like environment in dev, in test, in production. It's a mirror. I no longer have one organization I talk to. They have a whole standby data center. Just for the instance that they have an excessive capacity need or there's a drastic failure in their primary data centers, they also use that for change control to try and experiment out. They have to open a ticket, wait for the SLA to lapse, and then they get that change made in that standby data center. Now I can have an n number of instances of production. I can experiment and try things out. And so that's the green environment, a mirror. And so in this case, I can then flip the traffic after the testing completes by redirecting the route of that service over to version 1.2. Now if there was an issue, I could revert back to my blue environment. But the problem here is that it's low-risk. I have a completely separate environment. It's a mirror of my current production environment. There's also no downtime. And then you get that production-like testing because it is an exact clone of my current environment except the version of my application. And of course I can do a rollback as well. Some of the cons resource overhead, I now have double the capacity requirement. And then also I need to synchronize the data from a data store as well because that database. And then this is useful if you have a self-contained microservice with its own data. We talked about deployment strategies. Another aspect is host security. Containers are Linux. And Linux provides several capabilities to ensure you have a secure environment. First off, you have C groups, which provides quality of service so you don't have that noisy neighbor problem of two projects sharing the same infrastructure. Namespaces provides that logical separation, for instance, of users and groups from a container in the host and other containers. So root is not root on the host. So if there's a security vulnerability, I don't have to worry about taking over the whole system. And then SE Linux provides mandatory access control. So even root processes have to explicitly state what objects they have access to, such as files or directories. And then Set Comp, that allows me to apply filters so a process has a limited set of capabilities with the kernel. And then I can also leverage read-only mounts. So I mount the file systems as read-only and limit the potential negative impact of writing. So first off, here we talked about C groups. This shows you I can manage CPU memory network and storage by defining guarantees to my process. And so the kernel will ensure that they get at least that number of resources and don't have a noisy name or problem. Also, from a namespace perspective, two important ones are process ID, the network as well. So you can have multiple networks on the same physical infrastructure for multiple applications. So I can have isolation. Also SE Linux mandatory access controls. For instance, with a normal root process, they can have access to the entire system. With SE Linux enabled, it's explicitly stated what they have access to. And in Set Comp, these are the Linux kernel privileges. You can turn off a variety of privileges to kind of limit your exposure to any security vulnerabilities or compromises. And then also, from a read-only perspective, as you mount, you can limit what they can do so you can't write to slash proc, for instance, and make system configuration changes. In terms of best practices around host security, don't run as root, limit SSH access, use the namespaces for the reasons mentioned, also define resource quotas, enable logging, apply security errata to the host as well as the container, and then leverage Set Comp filters. Kubernetes also provides security capabilities integrated in a few other areas. One is network isolation, storage, API, monitoring, and federated clusters. From a networking perspective, you can define network policy to limit traffic between two containers or two pods. So you can lock it down kind of like IPsec or a firewall. And then from a storage perspective, you can restrict access to the persistent storage through authentication and authorization within Kubernetes, integrates, for instance, into LDAP or other OAuth providers. Also, you can limit access to the API through username, password, credentials, different service accounts as well so that you can limit the exposure of who can actually provision, deploy applications into the overall Kubernetes environment. And then deploying, monitoring, and logging. Ability to have things like Prometheus so that you have real-time monitoring and feedback loop to your DevOps team. And then also having auditing, logging capability that is then centralized so that you can search and view logs from a security perspective as you troubleshoot and look for a root cause as well as alert. And then federated clusters. So the ability to have multiple clusters. Maybe you have different security levels for different clusters and then you can deploy applications based on what their security level requirements are as well. So how do you know you're making good progress in terms of DevSecOps? Here's some of the metrics that you may wanna track. One is a compliance score in terms of security policy, vulnerability checks. Also, what is the deployment frequency? Is that improving from a trend perspective? Also, what is the lead time when you know and discover a security vulnerability? How long does it take to generate a fix? And how often is that service going down? And when there is an issue and there will be an issue, how long does it take to recover? That should be improving as you make this switch over to DevSecOps. And then of course, from a business perspective continuity, what is the service availability? All right, so thank you very much, Chris Vantyne. You can reach me on my email as well as on Twitter here as well. So thank you so much for attending today's session.