 Okay, so thank you everyone and welcome to this session. I'm going to talk about one stop talk for Kubernetes security. And who am I? My name is Nishant Kumar and I'm a senior software engineer at Microsoft. And I work for Azure Operator Distributed Services as a developer. And you can reach out to me on LinkedIn or Twitter. So talking about the agenda, it was a really big challenge to put all this information inside one talk as you might all know that Kubernetes security can be a conference in itself. But my main objective behind putting this talk was to help all the beginners or someone who want to learn about Kubernetes security but does not know where to start and what are the different areas they need to understand. So hopefully at the end of this talk you'll have enough information to start your Kubernetes journey in more depth. I'll start off by covering the overview of cloud native security. But you'll not dive in too deep as this information is accessible on the Kubernetes official documentation. I'll then talk about different areas like policy as code, Kubernetes secrets, threat detection, et cetera, that falls under the overall scope of security. And during this talk I'll also cover few specific tools that are used to, that are used to address few security concerns but that doesn't mean that those tools are the only one for those particular use cases. The general best practice is to have a list of requirements and then explore which tool can solve your purpose. So I think let's begin. So the four Cs of cloud native security. When you look for Kubernetes security in the official documentation, this is the first picture that shows up and it really sums up the scope for your cloud native security. So you can think about security in four different layers. The four Cs of cloud native security that is your cluster or your data center, the containers, clusters, containers, and code. And each of the layer of the security model builds upon the outermost layer. So you cannot get away by just implementing security at one particular layer. So the main takeaway from this slide is that each of these layers that you see here is equally important and you need to implement the best practices. And this talk would primarily be focused on the cluster and container areas where Kubernetes orchestration plays a major role. So these are some of the important things to consider in your infrastructure cloud security. So all access to the Kubernetes control plane should not be allowed publicly on the internet and should be controlled by network access control list. And nodes should be configured only to accept connections, again, by the network access control list from the control plane on the specific ports. And again, if possible, these nodes should not be accessible on the public internet. And it is always best to provide the cluster with cloud product access that follows the principle of least privilege for the resources that it needs to administer. And coming to HCD, as you all might be aware, it is the data store of Kubernetes and it stores the entire state of your Kubernetes cluster. So it's really important to limit the access of HCD and it should be accessible only on the control plane and over TLS. And another best practice is to encrypt HCD at rest as well. And again, there are official documentation available on how to achieve it. And then there are two areas of concern for securing Kubernetes. One is your cluster components that are configurable and other is securing the applications which run in the cluster. So the three most important and the basic things when securing your cluster component like the Kubernetes API scheduler and Kubelet is API authentication, authorization, and TLS for API traffic. So talking about TLS, Kubernetes expects that all API communication in the cluster is encrypted by default with TLS and most of the installation methods will allow the necessary certificates to be created and distributed with the cluster components. And API authentication, you should choose an authentication mechanism for the API servers to use that matches the common access patterns when you install a cluster. So for instance, a small single user cluster may wish to use a simple certificate or static bearer token approach and larger clusters may wish to integrate an existing OIDC or LDAP server that allows users to be subdivided into groups. And once authenticated, every API call is also expected to pass an authorization check. So Kubernetes ships an RBAC component that matches an incoming user or group to a set of permission bundled into roles. So as with authentication, simple and broad roles may be appropriate for smaller clusters, but as more users interact with a cluster, it may become necessary to separate teams into separate namespaces with more limited roles. And again, these are some of the important things to consider within cluster security for application, container, and code. Again, this is not a complete list, but these are some of the high priority items that you need to consider when implementing security. For example, encrypting secret at rest or enabling port security admission, enabling the network policies, running tools for checking container vulnerability, image signing and information, disalign privileged users, static code analysis, dynamic probing attacks. So these are some of the high priority items that needs to be considered when you're implementing security. So moving on to another section, which is policy as code, and what is policy as code? It is writing code in a high level language to manage and automate policies, which means you can store these policies in your version control system like Git. You can use these policies in audit or enforcement mode to monitor existing workloads, services for misconfiguration or prevent the misconfiguration applying in the cluster. So if you're looking to design these policies, you should broadly categorize them into three categories. One is the standard policies, best practices across the cluster in your organization. A few examples could be requiring resources to specify the resource limits or preventing workloads from running as root and then organizational policies, which are specific to your organization. For example, enforcing a private image repository list to pull, and then last, the environmental policies. So for example, your production environment will require more stricter policies. And some of the examples while implementing policy as code is like disalign latest image tags, restricting image repositories, checking for deprecated APIs, and host path volumes within the pod must be forbidden. So pod security admission, it falls under the same umbrella of policy as code, and it provides us with a mechanism to secure our pods. And pod security admission is a successor to pod security policy, which was deprecated in Kubernetes in 1.21. And pod security admission places requirement on pod security context based on levels defined by pod security standards. And pod security standards is again a separate thing which defines three different policies to broadly cover the security spectrum, and these standards let you define how you want to restrict the behavior of pods in a clear, consistent fashion. And these three different levels are privileged, that is purposely open and entirely unrestricted. Baseline, which is aimed at ease of adoption for common containerized workloads while preventing non-privileged escalation, and restricted, which is aimed at enforcing current pod hardening best practices at the expense of some compatibility. And then these policies can be applied in two different ways, either via the namespace label or the admission configuration resource. So using namespace label allows for granular per namespace policy selection, whereas admission configuration allows cluster level defaulting along with exemptions. So again, you can go to the official Kubernetes document which has more details as to what are the different things that are allowed within each of these levels which are found under the pod security standards. So again, here's an example for enabling pod security admission at the namespace level. So this manifest defines a namespace, my baseline namespace, and it blocks any pods that don't satisfy the baseline policy requirements, generates a user-facing warning and adds an audit annotation to any created pod that does not meet the restricted policy requirements and pins the version of the baseline and restricted policies to 1.23. And here is an example for enabling pod security admission at the cluster level. So as of v1.22, Kubernetes provides a built-in admission controller to enforce the pod security standard, and you can configure this admission controller to set cluster-wide defaults and exemptions. So now moving on to another area which is policy engine. So while pod security admission that we were talking earlier in Kubernetes is a set of mechanisms for ensuring validating controls for pods and their attributes, but as the name would imply, it only operates on pods and nothing else, and contrasts with the policy engine such as Gatekeeper and Keveno, the capabilities are far more broad and it's applicable to more than just the pods. So having a policy engine for Kubernetes can be thought of as a way to more holistically control the Kubernetes environment and not just a single domain. So OPPA Gatekeeper has a constraint framework and a constraint is a declaration that is author wants a system to meet a given set of requirements and each constraint is written with Rego, which is a declarative query language used by OPPA. For example, on the left side is a constraint template CRD that requires certain labels to be present on an arbitrary object and once a constraint template has been deployed in the cluster, an admin can now create individual constraint CRDs as defined by the constraint template. So for example, here's a constraint CRD that requires the label HR to be present on all the namespaces. And another popular policy engine tool is Keveno, which is specifically designed for Kubernetes that is managed as Kubernetes resource and no new language is required to write these policies. So in the earlier example, we saw a constraint template, then a constraint CR with Rego to be learned, which can be a bit complex. And in this case, the same policy is defined in a single file and is more Kubernetes native. So yeah, it really depends on the use case that you want to do. If you have more complex policy requirement and you want a general policy framework, then you can use OPA. But if you're just dealing with Kubernetes resources, it's worth trying Keveno. And a network policy is another aspect. So if you want to control traffic at the IP address or port level, then you might consider using Kubernetes network policies for particular applications in your cluster. And by default, all the connections are allowed. So each of the pod can talk to everyone if there are no network policies. And here's an example of Kubernetes network policy versus Calico. So the Kubernetes network policy API provides a standard way for users to define network policy for controlling network traffic. However, Kubernetes has no built-in capability to enforce the network policy. So to enforce network policy, you need a networking solution. For example, Calico. And Calico is a network policy and extension of Kubernetes network policy that has more capabilities. So on the left-hand side, you have a Kubernetes policy which allows egress traffic from pods in the same namespace. And on the right-hand side is a Calico policy which provides capability to control traffic to or from endpoint in a namespace. And now talking about Kubernetes secrets. So the default way of storing secret as base64 encoding is not secure. So we need tools to manage our secrets in a more automated and secure manner. So here are some of the open source solutions like Bitnami seal secrets, Hashikop vault, or external secrets operator and Helm secrets, and there are a few others as well. So we need these kind of external tools to manage your secrets. But I'm going to talk about one tool which is Bitnami seal secrets. And a special thing about this is that you can encrypt your secret and store it in your Git repositories along with your manifest files. So seal secret is a one-way encrypted secret that can be created by anyone but can only be decrypted by the controller running in the target cluster. So the seal secret is then safe to share publicly and upload to the Git repositories. And once the seal secret is uploaded to a target Kubernetes cluster, the controller will decrypt it and recover the original secret. So, and it consists of two parts, a cluster side operator or controller. So that is the one which is running inside the cluster and is used for decrypting. And a client-side Kube seal, which is used to encrypt your Kubernetes secrets. So on the right-hand side, you see a normal Kubernetes secret on the right-top-hand side. And then you use the Kube seal command line tool and then convert it into a sealed secret. So, and then it's, you can store it in your Git repositories with a manifest file. So now let's talk about the Salsa framework which is also known as the supply chain levels for software architects. And it is a security framework, a checklist of standards and controls to prevent tampering, improve integrity and secure packages and infrastructure in your projects, businesses or enterprises. So one of the biggest challenges for software developers is the need to make informed choices about the external software and products that they're using within their own system. So evaluating whether a given system is appropriately secured can be challenging, especially if it's externally owned by a third party. And the supply chain has been in scrutiny over the years with attacks on the software system. So in collaboration with OpenSSF, Google had proposed the new Salsa framework which formalizes criteria around software supply chain integrity to help the industry and open source ecosystem to secure the software development lifecycle. So it has different levels up to level four and each level deals with specific aspects of security. So this is still a very new project and I'll ask you to try keeping an eye on it for future enhancements. So SIGSTORE is another project which was started to improve the supply chain technology for anyone using open source projects. And within SIGSTORE you have different sub-project which focuses on specific aspects of supply chain security. So these sub-projects are cosine, full CO, breaker, and open ID connect. And even Kubernetes had adopted SIGSTORE in its 1.24 release. So again, a very interesting project and keep an eye on improving and trying to improve the supply chain security. So within SIGSTORE I'll talk about one tool which is a cosine. And it is a tool to simplify the signing and verification of the container images. So most containers available today are vulnerable to supply chain attacks because they can be published with nothing more than a simple API key. So if that key leaks, it's easy for an attacker to publish a legitimate looking container that actually contains vulnerabilities. So one of the best ways to protect users from these kind of attack is by signing the image at creation time so that developers can verify that the code they received is the code that the maintainer had authored. And then you can also combine different tools to verify the complete flow of your container image signing process. So with cosine, you can sign an image and make this signature available to anyone with pull access so it can be verified. Then there's a project called Harbor which can prevent images from being pulled unless they have some signature. And then you can again use Kiverno to ensure at runtime that it is a signature that you want. So you can use these tools to automate your whole process. Now moving on to another area within the security umbrella is threat detection. So far what we have covered were the preventive measures which included proper access control, authentication and authorization in place. But no matter what level of protection a system may have, there's no full proof silver bullet security solution. So a defense in depth strategy should be deployed so when each layer fails, it fails to a known state and sounds an alarm. So the most important element of this strategy is timely detection and notification of a compromise when it happens. And thus threat detection falls under the detective measures and FALCO is a project which can be used for such cases. Taking a look at the FALCO architecture, it can detect an alert on any behavior that involves making Linux system calls. FALCO alerts are triggered based on specific system calls, arguments and properties of the calling process and it operates at the user space and kernel space. So the system calls are interpreted by the FALCO kernel module and then the system calls, the system calls are then analyzed using the libraries that are present in the user space and the events are then filtered using a rules engine where FALCO rules are configured. So suspicious events are then allotted to outputs that are configured as syslog, files, standard output and others. Some of the examples that FALCO can detect is a shell running inside a container or a pod in Kubernetes and if a container is running in a privileged mode or an unexpected read of a sensitive file such as HC Shadow. Now talking about the Kubernetes security guidance frameworks. So since Kubernetes follows a loosely coupled architecture, securing the ecosystem involves a cross combination of best practices, tools and processes. It is also recommended to consider frameworks that issue specific guidelines for easing the complexity of administering the security and compliance of a Kubernetes ecosystem. So to help with these various institutions of a standardized frameworks and guidance for administering security in a complex dynamic Kubernetes ecosystem. So these are some of the frameworks, the CIS benchmark, Mitra attack, PCI DSS, NISF framework and the very recent NSA CISC Kubernetes hardening guide. And these are the comparisons between the different security frameworks and it says when we should use them. And also at the bottom you can see, again, you can use the tools which have been developed keeping these frameworks in mind. So if you explore these frameworks closely, you'd find each of them have a huge list of security best practices that they recommend. But it can be quite time consuming if you look at the complete list, but instead that you can leverage open source projects such as mentioned in the tool section which have been developed keeping these frameworks in mind. So these are some of the open source tools, the Kubernetes vulnerability scanning tools and some of them have been developed keeping the mind, keeping in mind the frameworks that we were talking about earlier. So a few of these tools are cube bench, cube hunter, cubescape, cube audit, cube scan and crane. And I'll be talking just about one tool and that's cubescape. So cubescape scans your Kubernetes cluster, your static YAML files and help charts, detects misconfigurations based on multiple frameworks such as NSACIs and the Mitra tag that we saw earlier. It calculates risk scores and shows risk trends over time and it really has a good super friendly UI. So you don't need to see the complete list of the policy violations in your console and then you can move directly to the UI and see what are the different violations that happen. It also has capabilities for image scanning and RBAC violations and it can also assist in remediation and it can tell you what steps you need to take to get away with those policy violations. And lastly, one of the best ways to develop software, deploy them and manage them in a dynamic environment is to adapt the DevSecOps model and DevSecOps stands for development, security and operations. It's an approach which integrates security as a shared responsibility throughout the entire IT lifecycle so in the past the role of security was isolated to a specific team in the final stage of development but and that wasn't problematic when development cycles were lasted even months or years but now it develops and agile methodologies. I mean we need to take care that security is taken into consideration from the start. So DevSecOps means thinking about application and infrastructure security from the start and it also means automating some security gates to keep the DevOps workflow from slowing down and some of the benefits are faster delivery, improved security posture and reduced cost. And finally I think without these information available in the internet the stock wouldn't have been possible so thank you for all this blog post which is out there and thank you, thank you for coming to the stock and yep, hopefully you liked it, thank you.