 Hello, and welcome to this track on Container Security 101. I hope you're having a good time at the summit so far. My name is Sandram. I'm a cloud architect and technology advisor currently working with a couple of companies, both startups. More than half of my 15-year experience has been on various cloud technologies. And lately, I've been focusing on security, especially assessing the security posture of infrastructure, application, data, implementing recommendations, and remediation. This talk will provide a quick overview of cloud-native security, but focus mostly on container security, covering the challenges and risks that come with using containers. I will then point out some common threats in scope and highlight some of the container vulnerabilities that are out there. The talk will finally introduce various tools, mostly open source, that can be used to provide a layer of protection for your containers. Cloud-native computing is the hottest trend in the cloud ecosystem and is increasingly becoming the new norm for application development and deployment. And at the core of cloud-native computing infrastructure are containers. With the increasing adoption of containers, along with the scale of deployments that we see today, security becomes an important aspect of the adoption strategy. One of the best practices around security, even before cloud became popular, was the layered approach to security, otherwise called as defense and depth approach. The four Cs of cloud-native security are based around that. These four layers of cloud-native security are code, container, cluster, and cloud. Cloud-native security cannot be managed by just container security alone. So you should ensure that other layers are secure too, such as following secure coding practices, checking for code vulnerabilities during development, maintaining security for containers, clusters, such as Kubernetes, Docker, Swarm, or any other platform you use. Similarly, securing your hosting platform, which can be a cloud provider, for example, is also equally important. In this talk, we will be looking at container security specifically. But remember that the other layers are also equally important, and you should follow the layered security approach around all of them. Before we look into container security, let's quickly see what containers are, especially for those among us that are new to this term and will then look at what the container adoption market looks like today. The definition from Docker states that a container is a standard unit of software that packages code and all its dependencies, so the application runs quickly and reliably from one computing environment to another. Now, the idea of containerization, or otherwise called as process isolation, is not new and goes back a few decades. However, the way it is used today is definitely a game changer. The problem statement that we're trying to solve with containers is reliable movement of code between different compute storage network and security environments, while keeping it lightweight, fast, and modular. And we do this by packaging the code and its environment, which includes dependencies, libraries, binaries, configuration files, all into a single unit minus the operating system. So containers involve virtualizing the operating system versus virtualization in general, virtualizing the hardware. And containers are becoming increasingly popular. There's a growing community, and the enterprise adoption of containers have spiraled over the years. The rise of container-native applications, container-native infrastructure, DevOps, have further fueled the adoption of containers. So containers solve problems around portability, image size, agility, delivery, modernization, et cetera. If you're talking about containers, there is a very likely chance the word Kubernetes will pop up. So what is Kubernetes? It is an open-source software to orchestrate containers at scale. Securing Kubernetes falls under the cluster security layer of the cloud-native security and deserves its own focus. It's a huge topic. Certain security vulnerabilities we will talk about later, as well as recommendations on implementing container security may fall under the realm of Kubernetes too. Coming to the container adoption trends, container adoption is spiraling. This can be seen everywhere from tech blogs to media to talks where presenters describe their experience around containers. All of this gives you a sense of how popular containers have been, both among developers as well as business. This can also be seen in various studies and reports done from time to time. So we will take a look at a few reports in the next couple of slides to get an idea on what the container adoption trend looks like. This report, 2018 report by 451 research, states that containers will be a $4.3 billion market by 2022. And container adoption is mostly driven by its maturity over time, the growing community of developers, resources that are available, new enterprise products that are built around containers, and a lot of advocacy by tech giants and high-growth startups that eventually build a sense of trust and confidence for others to adopt it. Containers are transforming businesses, improving agility, reducing cost, risk associated with a faster release. And hence, an increasing number of companies are adopting a container-first strategy. Similarly, this 2019 report or survey by Aqua Security and Portworx shows that container adoption in production continues to grow. And this includes large enterprises as well. 87% of the respondents use containers today, and out of them, more than 89% use it in production in some form or other. Similarly, Gartner also predicts that by 2022, more than 75% of global organizations will be running containerized applications in production, which is a significant increase from what it is today. There are many such adoption surveys that show the same trend. This is because of the many benefits mentioned earlier, and this is only going to grow, and some experts say this is just the beginning of the popularity of containers for applications, or application containerization trends. On the same lines, this CNCF survey of 2019 shows that the use of containers in production has increased significantly. 84% of the respondents are using containers in production, and this is up more than 15% from 2018, right? If you look at the first survey of 2016, where it was 23%, this is an impressive jump. This is a result of organizations having more trust in containers and using them more and more in user-facing applications. We also have a similar report from Forrester. This study conducted by Forrester Research on behalf of Capital One shows the rising popularity of containers, and the priority for businesses and IT leaders is mostly to become more and more cloud-native and implement a container-first approach in their business. So this growing adoption of containers, especially technologies such as Docker, Kubernetes, OpenShift is great, and the benefits like faster deployment, better scalability, portability has obviously fueled the growth of cloud-native technologies. But this also means security, monitoring, management of containers is going to be a matter of concern. It's going to continue to be a matter of concern. We'll again look at a few surveys, and we will see that container adoption is not without its challenges. This survey by QuoteWorks and EcoSecurity on over 500 IT professionals across a variety of industries and company sizes shows that security at 51% remains the top challenge for IT professionals. Similarly, the CNCF survey of 2019 that we saw earlier shows that security remains the second biggest challenge in container adoption. Monitoring is also on the list at fifth. Why did we see that containers are inherently secure due to their architecture, isolation, for example? How and where we use them today introduces a lot of security concerns at our operating system level, at the container engine level, application level, and data level. So as containers are deployed by more and more companies and they are used for sensitive workloads, they are used for critical workloads, used for production workloads, they become hot targets for hackers. Having said this, development environments are also hot targets for hackers. Some may say that container security may not be that much of a huge concern at the moment, but it will increase as more and more bad actors or exploiters learn container-focused exploits. Security is a growing priority in container adoption, and if it's done well, they could help react to security issues faster. When it comes to security for containers, we will continue to follow a zero trust model, and we will see that in the next set of slides. We would always want to verify every application that runs in our containers, every container image that we use, every host that we plan to run containers on. So zero trust model applies to container security as well. In fact, it applies to everything around cybersecurity. Let's now take a look at container threats. With containers being used everywhere today, such as mobile applications, IoT, machine learning, data science, and even databases, it is important to know what types of container threats are possible. There has been an increase in the attacks on container environments such as Docker and Kubernetes lately. And one of the top container threats that you will see most people talking about will be container malware. Container malware is a gaining popularity. An example of this threat is the KinSync malware, which is provisioned in a container environment, mostly because of a misconfigured Docker API. The malware container then works as a crypto miner moving laterally and spreading the malware to other containers and hosts. So basically these containers that get created within your environment are used for crypto mining. Another interesting threat is the crypto jacket. This involves fraudulent cryptocurrency mining, as it is a lucrative area. This is also a type of malware, and it will use computing capacity of infected machines to mine virtual coins, such as Ethereum and Monero. The cryptocurrency is mined from these containers, are then sent to wallets controlled by attackers. Another container threat is dirty cow container exploit. In this threat, attackers exploit the kernel bug to override a so-called set UID program in the system, which allows the user to temporarily elevate the privilege in order to perform a certain task. By replacing the set UID program, basically the attacker gains root access privilege when the program is executed and is able to do anything. This threat is not removed if the compromised container is removed. Hence resulting in a situation where the container immutability is broken. An example of such a set UID program that could be compromised in a Linux system is the EAS SWD. Another one on my list is escape vulnerability. This involves Run-C, which is a runtime that ties container engines to Linux kernel. In this threat, an attacker with root-level access inside a malicious container may gain root-level access to the host by performing certain tasks, such as running privileged container, executing privileged escalation bugs, or misconfiguring the container, etc. Docker engine was affected by this and has been patched since. This is just an example that I wanted to point out. This list is not exhaustive. There are many more container threats out there. I just wanted to bring to your attention some of them that are very important and you should be cautious about them. Here's a list of container vulnerabilities that you should be aware of. We will talk about only a few of them in the interest of time and the level of content. Container images are on the top. Images or container images are building blocks of containers. They are layered in architecture and are made up of other images, such as base images, which could be published by a different author. Hence, they are a common source for injecting vulnerabilities into your system, into your applications. Further, it's a common development approach to reuse images, especially in open source developments, and hence these images may already have vulnerabilities in them. Another vulnerability area that you should be aware of is container root account. Many images provide root access within the container that too, without a password. This is a type of misconfiguration that can expose containers to vulnerabilities both within the container and in some cases to the host as well. An example of this is the official Alpine Linux Docker image shipping with a blank root password. This was found by Cisco Talos researchers in 2019 and while this may not be a direct threat and only systems configured a certain way may be vulnerable, this definitely is a concern and there are many such images out there that come with blank passwords. Access control is another type of vulnerability where vulnerabilities are introduced through less secure container engine or insecure orchestrator APIs Docker API, misconfigured APIs, et cetera. This vulnerability allows attackers who get into your systems to provision containers misconfigure them or take control of your container environment. In an example, incident involving Tesla, hackers gained access to unprotected Cuban disk console and in a classic example of lateral movement they found Tesla's AWS S3 access credentials and gained access to some telemetry data. They also performed crypto mining and according to the report some sophisticated evasion measures were employed by the attackers in the attack. The next is privilege escalation which involves exploiting vulnerabilities during container runtime. The runcy exploit that I was talking about earlier is an example of this. Networking is also an area or a threat vector. It's also an area where one could find vulnerabilities, one could inject vulnerabilities. Within a container environment, especially one with orchestrators, containers talk to each other over an overlay network so a compromised container can spread laterally to other containers and a similar situation may arise if there are internet exposed containers as well leading to vulnerabilities around distributed denial of service or DDoS, SQL injection, cross-site scripting, etc. Insecure or less secure configurations also result in vulnerabilities being exploited to steal data or consume computer resources and this is one of the most common vulnerabilities out there. You will find many attackers or exploiters actually looking for consuming computer resources or stealing your data. Here are some examples of container vulnerabilities with their CVE numbers. I will not go through each one of them in detail but just wanted to quickly run by a slide highlighting a few. The CVE-2019-5736 vulnerability, for example, allows a malicious container to override the host runc binary and gain root-level code execution of the host. This vulnerability has been patched since it has been found out. This is a vulnerability that would allow root-level code execution, container escape, and access to the host file system. Similarly, another CVE from 2019-5021 is an issue related to the use of blank passwords for root accounts in the Alpine Linux Docker images, the one that I was referring to earlier. We will take a look at some reports again just to reinforce or stress on the fact that security is actually something that is a concern for enterprises, for businesses, for developers. This particular security survey by PortWox and Aquasecurity shows that when it comes to container security, the top challenges include vulnerability management and runtime protection. So how should you be protecting containers? We will broadly look at two things here, container image security and container runtime security. One of the most important elements in container security is securing the container images. And container images use commonly available open source Linux distributions as the base, hence bringing in many vulnerable system libraries along with them. So a few questions you should ask yourself in this step are some of them on the slide. For example, are the container images free from vulnerabilities? You should also include base images as part of this question. So to counter this, you should use latest images. You should run some image vulnerability scam. And like I said, include base images as well. Prefer certified images and do not pull images without validating authenticity. Zero trust model. Fix vulnerabilities before using them in production to run your code. Do you trust the images? Sign and verify images and use only signed images, preferably pushed by the publisher itself. And look at the history to see if anything has been modified. Similarly, you would be using some container repository. Is your image repository or registry secure? Compromised registries are also a threat. They can be used to tamper images. Secure your registry. Use private registry wherever possible. Another question that you should be asking yourself is are you following the least privileged model? A container is executed as a root user if you don't provide a user and could be a security issue. So see if you really want your container to run as a root user. When that namespace is then mapped to the root user in the running container, it means that there could be a potential for the container to have root access on the host as well. So this is an important question that you should be asking yourselves as well when it comes to container image security. This report by Snipe, a company that provides products for detecting, fixing and monitoring vulnerabilities, shows that the top 10 most popular Docker images each contain at least 30 vulnerabilities. Similarly, there was a blog post on Kenna Security, a company that specializes in using data science to measure, prioritize and predict cyber risk. And their report states or their blog states that nearly 20% of the 1,000 most popular Docker containers have no root password. While the vulnerability is environment dependent, and just because there is no root password doesn't mean it's alarming. This is something that you should definitely not ignore given your use cases and giving your workflows. So don't take security lightly. And if you have no need for your containers to be without a root password and for security in terms of a username and a password. If you don't really have a need for it, do not run containers as root as well. There are certain use cases for root privileges in containers such as Docker and Docker or containers that require direct hardware access. So in those cases, you may look at running containers as root, but otherwise you typically don't need containers to run as the user root. So the things you should be looking at when it comes to container runtime security, and of course this is not the complete list, could be knowing your standard application behavior and monitoring to detect any deviations from the standard behavior. You could use an intrusion prevention system here or an IPS system. Secure your container API access, orchestrators, service mesh. These technologies pose additional complexity and bring in new attack vectors. Again, monitor and secure them. Follow best practices and secure container engine platforms. For example, disable capabilities such as setGID and setEID to prevent processes from changing their GID, which can result in escalation privilege. Use a separate user instead of a privileged user for Docker commands. Use C groups and namespaces. They also help in avoiding privileged attacks. Secure your communication between client and server using certificates, TLS. Don't forget network security. As I mentioned earlier, containers can talk to each other by default. Isolate them using network namespace. Use network security policies, which is, for example, by default available in orchestrator softwares like Kubernetes. You can also implement a forced tunnel approach to direct the traffic through an IPS IPS system. Use multi-stage builds and create runtime container images that are smaller in footprint and contain only the required modules and dependencies. So this will reduce your attack surface area as well. Use container-specific operating systems such as CoreOS, Red Hat Atomic, RancherOS, there are many out there. Keep your host OS up-to-date and apply the latest security patches frequently. Unauthorized access to your container runtime should always be prevented. You can use solutions such as SE Linux, App Armor, container security policies or work in bringing or creating your own container security profiles and implement them in your container runtime. Container runtime security is very, very important. Anyone gaining access to your container runtime or finding out a threat, vector, or an attack surface to compromise your container runtime can compromise your applications, data, or the environment or the complete environment itself. Let's now take a look at some tools that are available to help you secure, protect your containers. Many of them are open source and these open source ones also have a paid version. When it comes to container image security always give a thought when selecting a tool or a software to the following. What kind of scanning ability does it have? Does it provide real-time scanning? Does it provide real-time monitoring? How frequently are these tools or software updated? Can you automate through DevOps? Do they provide policy support? These things will help you implement container image security with ease. Clare by CoreOS, for example, is an open source project for static analysis of vulnerabilities in application containers. Anchor Engine is also an open source tool that provides deep image inspection and vulnerability scanning. It provides a CVE-based vulnerability reporting and is DevOps compatible and supports policies as well. OpenScap is also an open source project that provides various benchmarking guides and configuration baselights. You can use them in your container deployment or you can use them in your container image security areas. For container runtime we can also find a lot of tools that are out there. There are many companies that are also building security tools for containers nowadays. It's an emerging area too. If you're using Docker itself, for example, provides in-built protection such as control groups and namespace capabilities, but you can always use tools to provide and increase runtime security. Look for capabilities such as real-time monitoring, access control, network monitoring, forensics, compliance and audit in these tools. Cystic Inspect is a powerful open source tool for container troubleshooting and security investigation. It provides a deep system-level capturing and real-time analysis capabilities. Falco is also an open source cloud-native runtime security project and created by Cystic and is the de facto threat detection engine on Kubernetes. Docker image security scanning and runtime monitoring and scanning should be a core part of your Docker security strategy or in general container security strategy. Although image scanning won't protect you from all possible security vulnerabilities, it's a primary means of defense against security flaws or insecure code within container images. Hence, it is a foundational part of overall container security. Some registries like Docker registries for example, the Docker hub provides in-built scanning but if you implement your own container registries or repositories make sure you implement one of these container security tools. Docker Bench for security very popular. It's a tool to check for the security posture of your default Docker installation. It implements common best practices for deploying containers in production and is based on the CIS Docker Benchmark currently at version 120. Use Docker container trust for signing images. This will encrypt them and avoid man-in-the-middle attacks when moving containers around. You could also use notary for image signing. There are network security policies or there are tools that provide network security policies like Celium and Calico. So if you use them they come with network security policies that you can leverage. So to summarize our container security best practices use latest images always scan the images for vulnerabilities use images from trusted repositories always enforce a least privileged access whether it is APIs or the container or the Docker host itself always ensure that you are following the least privileged access and to take care of container runtime security monitor monitor user sessions monitor network, monitor application behavior, monitor privileged access, monitor privileged containers, monitoring will help you understand if there is a deviation from your standard behavior. I want to end this talk by stating that container security must be embedded into the container lifecycle and should not be an afterthought. Hence take a look at how you can use DevOps practices for this and remember to shift left when it comes to security and implement a zero trust security approach. Thank you very much for listening to this track and I welcome your feedback and any questions you may have. Thank you.