 Thank you everybody for joining us today, and welcome to today's CNCF webinar on Mitigating Kubernetes attacks. I'm Michelle McLean Vice President of Marketing here at Stack Rocks, and I'm joined by my colleague and good friend Wei Lin Dang for he's our head of strategy at Stack Rocks, and we're looking forward to a great discussion around Kubernetes attacks. I'm going to kick off a couple of housekeeping items just to get things going so everybody's clear because we're CNCF is recording this for later playback. Your lines are muted, but we very much want to hear from you. Wei and I would like to do questions as we go actually. I think it's a lot more interactive and a lot more fun to take those when you have them. So please use the Q&A box at the bottom of your screen to drop in questions there, and we'll get to as many as we can as we go along. There's also a chat box. You can use that if you're having any technical difficulties, but please put your questions in the Q&A box. This is an official webinar of the CNCF, and as such it's subject to the CNCF code of conduct. So we ask that you not add anything in the chat or type in any questions that would be in violation of that code of conduct. At the end of the day, simply put just be respectful of your fellow participants and the presenters and everything will be great. So we are recording this webinar, and the recording and the slides will be posted later today on the CNCF webinar page, which you'll find at cncf.io slash webinars. So with that, let me give a hearty welcome again to my friend Wei. As we kick off our discussion on Kubernetes attacks, what they can look like, the form that they're taking, and steps that we can take to mitigate those attacks. Wei, can you take us through what we'll be talking about today? Thanks, Michelle. And good afternoon, everyone. Thanks for joining us. You know, I'm glad to be able to talk today about the Kubernetes attack matrix. And, oops, there we go. And, you know, more specifically, I'm going to be going over something called, you know, the Kubernetes attack matrix and how you can use it to think through effectively securing your Kubernetes clusters from the broad range of threats that you or your organization might encounter. And so here's what we have in store for today's session. I'll start by covering some examples of Kubernetes attacks that have been observed in the wild. And those include, you know, attacks from threat research that the Stack Rocks team has previously done. I'll, you know, then briefly touch on what makes Kubernetes threat vectors different from those that you may have had to protect against in other infrastructure environments, prior to cloud native environments. And then we'll turn our attention to the Kubernetes attack matrix itself. And we'll cover, you know, what it is and what it means for Kubernetes users. And the goal that I hope we can really accomplish today is to distill some main themes based on our analysis of the attack matrix and I'm going to be aiming to help you come away this afternoon with a few key takeaways and practical recommendations that you can immediately use to increase the security of your Kubernetes environments. One set of questions we often get asked about is, you know, what types of Kubernetes attacks exist. People are interested in what sorts of attacks, you know, other organizations have experienced or dealt with and, you know, what can we collectively as a community learn from them. And it's useful to start there because Kubernetes attacks and the ways in which cloud native software can be vulnerable and potentially exploited really falls within I think an emerging threat landscape and it's a landscape that we're all still trying to understand better. And but given that, you know, there have been a number of attacks and threat vectors that have been at least publicly disclosed and this is a representative set on this slide and we can see it spans, you know, different types of attack vectors, you know, they include campaigns to abuse resources by mining cryptocurrency at companies like Tesla. There's also been disclosures by Microsoft Azure regarding large scale campaigns for cryptocurrency mining being threat vectors such as, you know, the discovery of potential exploits via things like server side request forgery vulnerabilities at places like Shopify. And then there are, you know, different types of, you know, exposures, you know, via things like backdoor images from Docker hub or other public registries that can be accomplished or utilize for malicious activity, as well as, you know, critical vulnerabilities that have been disclosed, you know, in the Kugmei system itself. And initially, you know, at Stackrocks, you know, our team has also done quite a bit of research, you know, that has ultimately corroborated the notion that, you know, containerized applications are subject to various threats with some important qualifications and so, you know, to better understand some of these Kubernetes attacks, you know, we have done some work to set up a honeypot environment using a number of large scale Google Kubernetes engine clusters. You know, these clusters are running all different types of containerized apps. You know, these might be web servers or databases or developer services built using common images that are available from places like Docker hub. And, you know, these images had a number of, you know, different known vulnerabilities. And we also deployed them with, you know, deliberately with weak configuration so example might be, you know, we left authentication disabled. And what we found from running this environment for about five months exposed to the internet is that, you know, many of the types of attacker actions that we observe were not too dissimilar from those that can also impact non containerized applications and I think this makes a sort of sense, you know, since an attacker, you know, at the outset wouldn't necessarily know that a particular application is running in a container or not. And, you know, these types of sort of attacker actions that we observed are, you know, very common techniques such as injection attempts or intrusion attempts or attempted downloads and commands that you often sought to gain, you know, maybe additional targeting data such as you know, information about the operating system or, you know, potentially launch some form of command and control traffic. So way just to point of curiosity here in terms of these findings versus some of the some of the headlines you shared on the previous slide. How does how do they compare like what what surfaced in the stack rocks research that's either similar to or different from some of those headlines of known attacks on our known vulnerabilities in Kubernetes? Sure, Michelle. I mean, I think it's a good question. I think in terms of some of the attacks that we saw in the headlines or what have been was been publicly disclosed. I think a lot of those were focused on, you know, specific, you know, attacks that leverage or utilized, you know, parts or specific components within Kubernetes, you know, might have been leveraging, you know, specific Kubernetes vulnerability might have been leveraging a specific, you know, component like the Kubernetes dashboard. And what our research, you know, how our research of compliments that is that is, you know, sort of the observation that there are a lot of attacks that are still relevant to container environments that are not, but are not specific to Kubernetes. And so I think what you end up with is that, you know, there's always a potential that an attacker may seek to do both, you know, they may seek to, for instance, gain, you know, access or, you know, intrusion, you know, via, you know, something that's maybe not specific to containers, but then they might leverage, you know, as part of that or part of their overall approach of methodology, they might leverage, you know, specific aspects and are likely to leverage specific aspects of Kubernetes itself, you know, to go further. And so, you know, I think as the combination of both what's been publicly disclosed in those headlines as well as our own research suggests that you should be aware of sort of like these new specific attack vectors for Kubernetes but you also need to keep in mind that there are set of attack vectors that are more generally applicable when you're thinking about how to protect your Kubernetes environments. Yeah, that makes sense. So you need to have kind of the protections in place you would have for any infrastructure, but then be aware of the, the new attack surface that's kind of exposed by or introduced by Kubernetes. That's right. And, you know, I think that that takes us into, you know, thinking about, you know, what's different about Kubernetes threats which is, you know, with some universe of known attacks, you know, which may not necessarily differ from traditional threats in terms of attacker objectives, you know, sort of question arises, which is, you know, why do we need to think about Kubernetes attacks differently and, you know, first I think it's best not to look at those headlines or those specific attacks, you know, individually, you know, it's better to think about how to protect against them in a sort of one-off or ad hoc manner. You know, it's much more effective to approach security more systematically and, you know, think about sort of threat vectors in a more comprehensive way and, you know, as we get into it, I think that's exactly what the attack matrix enables. You know, second, while, you know, attacker objectives and techniques, you know, may be familiar, the way those are executed may actually look quite different in Kubernetes because there are several important differences that Kubernetes introduces. It is a new attack surface. It has new components, for instance, you know, if you're looking at the control plane, for instance, you know, which makes global decisions regarding a cluster's operations, or whether you're looking at worker nodes, you know, each of those different parts of a cluster has, you know, introduces new components that are, you know, could be, you know, the form part of an overall attack surface within Kubernetes. You know, for example, you know, if you look at compromise of control plane components, you know, that could easily result in complete compromise of entire clusters. Versus, you know, if you're looking at worker node components, for instance, that could result in direct compromise of the containerized application components, you know, themselves, namely, namely pods. You know, the other aspect is, you know, about these environments is that, you know, pods that are running your applications, you know, they're ephemeral, they're distributed, you know, especially if you're using microservices architectures. And then additionally, you know, Kubernetes itself is complex, and this can often give rise to new possibilities for operator error, especially as you know, organizations are learning Kubernetes and getting started with it. And then, you know, finally, because these environments are, you know, highly orchestrated and automated, which has a number of benefits, I think that the trade offs or sort of downside impact of that is that, you know, things like any misconfigurations or unintentional exposures can actually be propagated much more quickly, you know, throughout the entire environment. So we do have actually our first question that's come in way. And I think it was spawned by your discussion around how threats can look different in this landscape so somebody's asking we have multiple. We have we. Oh, okay. Okay, so there's two questions here. One is around, we use a managed service, do we need to think about Kubernetes attacks any differently on a managed service versus if we were running Kubernetes ourselves. Let's start with that one. Sure. You know, that's, that's a question we get quite often, which is, you know, if you're using a managed service, for example, via a cloud provider offering, you know, for example, an EKS or GK or AKS, among others, you know how to think about security and you know, I think a starting point for something about that is that if you're using one of those services then you know the shared responsibility model that applies to that, you know, those providers, you know, comes into consideration and so in those services, you know the provider is going to the control plane for you, the nodes on which they run, and that, you know, certainly helps, you know, with aspects of security, for instance, you know, upgrading versions of Kubernetes applying any relevant security patches, also hardening, you know, parts of the stack, you know, that are relevant to those control plane components, but there are still aspects of security you'll need to think about, you know, for instance, some, with some providers, you know, the API server might expose a public IP address by default, or you might need to think about, you know, segmentation or other ways of locking down access, you know, for instance, you want to think about authentication and authorization which is left up to administrator and user configuration and so while, you know, manage, you know, services and cloud provider offerings and certainly, you know, go a long way to help, you know, with aspects of security, I don't think it changes the way you should think about security overall, it simply shifts so responsibility potentially to someone else for handling aspects of your overall security approach. And the other one is somewhat related and you touched on it a little bit, but this person asks, we have multiple managed vendors. Is there a checklist we can use for security to check on their offerings? There are good starting points, you know, out there, many that have been published, I would say by the broader ecosystem of, you know, folks that are stakeholders and interested in furthering Kubernetes security. You know, some of them are actually listed on this slide but, you know, it's a starting points for how to think about Kubernetes security. You can look at references such as, you know, the guide that the NIST, the National Institute of Standards and Technology put out. You can look at, you know, benchmarks put out for Kubernetes by, you know, Center for Internet Security or CIS. You know, there's also a wide range of resources that, you know, folks like Stack Rocks have put out there in terms of best practices and, you know, capabilities that you should look for when it comes to implementing Kubernetes security. And then finally, you know, the community itself has undertaken and continues to further, you know, understanding of, you know, things to keep in mind when, you know, implementing Kubernetes security. And this ranges from things like, you know, the security audit all the way to ongoing work by, you know, CNCF and Kubernetes sakes that are focused on security and so on. And so I think there's a wide range of resources. I don't think there's any single one size fits all or a single definitive checklist or framework to refer to. But I think that this is also because ultimately a lot of it comes down to you and your particular organization's needs. Yeah, that makes sense. But those are some great starting points. And the person must have been seeing ahead because it perfectly predicted the slide. So tell us a little bit about the Kubernetes attack matrix itself. Sure. So, you know, with all the context that we've covered so far in mind, you know, the, the Kubernetes attack matrix exists to help us think about threats and also how to mitigate them in a, you know, I would say systematic and comprehensive way. You know, again, you know, you don't want to be thinking about threat vectors or how to guard against them and sort of ad hoc or one off manner. You really do want to think about it more holistically. And so with regards to the attack matrix, my focus today is going to be on distilling some main themes from it and the techniques that are actually described within it. And, you know, for more detail, you know, we and others have put out resources. We have a blog series, you know, on on this topic, which covers the details of, you know, each technique and the relevant mitigations and more detail. I'm not going to go, you know, to that level, necessarily today. But, you know, the, but please refer to those as they're useful. But the communities attack matrix was published a few months ago by Microsoft Azure. And I viewed as a great milestone for the community ecosystem and it really builds on all the fantastic work that some of the various groups and organizations which I mentioned just a bit ago have undertaken to date. You know, including, you know, the inaugural, you know, CUBE security audit and it was part of that audit a threat model was also published and it covers the major Kubernetes components. Again, there's work by, you know, relevant sakes and other organizations that have published these widely followed guidelines and benchmarks. The attack matrix is derived from applying and extending principles from the well known attack framework published by organization called MITRE. And, you know, to sort of provide a comparison on the right here is the MITRE attack matrix for enterprise and they're actually multiple matrices that exist. And this helps provide an idea of what the attack Kubernetes attack matrix is meant to be, which is it's really meant to be a knowledge base of adversarial tactics and techniques. Based on real world observations and it provides a way to classify, you know, different methodologies that attackers might utilize, you know, based on the particular objective they serve. And so by sort of taking some of those principles or this overall framework that's laid out by MITRE attack. You know what the Microsoft Azure team did was extend those to Kubernetes. And if we look at the matrix itself, the matrix consists of nine separate tactics that encompass 40 different techniques. And these tactics, these are objectives such as, you know, execution and persistence privilege escalation lateral movement. You know, these should sound familiar they're not not new, but they are really about the why, you know, an attacker might undertake particular action they really focus on objectives and you know these are not necessarily going to change. Just because you're in your Kubernetes or a cloud native environment these these objectives are general across different types of infrastructure environments. But when you get to the level of techniques which are really about the how of what an attacker might do. These are mostly dependent on the components and resources that are specific to Kubernetes itself. The techniques, you know, are pretty similar to each other, you know, or they're also relevant to multiple different tactics, you know, like they're, there may be a, you know, a technique that, you know, is actually applicable to, you know, one or more or more different objectives. And addressing a technique can also require several mitigations, you know, it's not to say that there's a one to one mapping or a single mitigation for every single technique. You know, some of these techniques actually spun, you know, span multiple potentially multiple aspects of the attack surface and such so it may require, you know, multiple different mitigations to be implemented to effectively and completely protect against against them. You know, next I have, you know, one good example of this which is if you look at, you know, technique 1.4 and technique 2.4, you know, they're outlining the matrix either actually pretty similar. There's a reference, they reference an application vulnerability, for example, a remote code execution vulnerability, as well as your application exploit. You know, some of the mitigations that you might actually look to implement to protect yourself against these techniques, you know, these best practices will entail some of the following, you know, preventing vulnerabilities from being introduced into Kubernetes clusters in the first place. You can do this by taking steps like scanning your container images, enforcing policies at deploy time, you know, via capabilities like admission control in Kubernetes. In that you could seek to restrict external network traffic. You could do that via things like network policies in Kubernetes, and then apply permissions on deployments for that application. And again, you can do that via a capability in Kubernetes via role based access control. And then, you know, separately, you also want to ensure that the pods for that application are locked down as much as is, you know, sensible or appropriate, you know, via some of the measures, you know, describe at the bottom, you know, which is restricting privileges, restricting, and so on. And so, you know, this is a good example of how, you know, a single technique that's listed in the community attack matrix may actually require, you know, a half a dozen or more, you know, different types of mitigations to be applied. And I think that there's some nice promise in what you're saying their way because when you at first blush if you look at the Kubernetes attack matrix and you see, oh my gosh, you know, nine different, you know, nine different techniques and in total 40 unique attacks like that can feel really overwhelming. But what I'm hearing you say here is that many of these have common elements. And so there's some steps that you can take you might have to take multiple steps. But those multiple steps that you take could mitigate multiple attacks in turn I think that's a bit reassuring. So before we get into those kind of fundamental steps of like how do people start thinking about protecting themselves. I'd love to just pause for a moment and just recognize that the MITRE model is really familiar to people on the security side, right, they they're used to thinking this way, and it's the whole mandate of like think like an attacker, and put, like you said, really systematic approaches in place to mitigate these things very broadly. A lot of folks obviously on the coup side are coming more from the DevOps world. And I'm curious, you know, this could be new language for them a new mindset for them. How do you think about the potential for the coup attack matrix or some of this framework in helping bridge this gap and helping these two worlds kind of work together a little more easily to be effective at protecting against these attacks. Yeah, I think the attack matrix provides, you know, very rich framework and a great starting point for effective collaboration between security and DevOps teams, you know, security to some extent you know is familiar with these types of objectives. You know, some aspects of the techniques, you know, maybe new and that's where, you know, I would say, you know, develop DevOps or, you know, operational teams that are, you know, running Kubernetes infrastructure, you know, are going to be familiar with these different components and some of them are relevant threat vectors that are associated with them. And I think, particularly when it comes to, you know, Kubernetes and cloud native technologies generally there there's certainly a huge opportunity to implement better security hygiene earlier, you know, in the life cycle of your applications, to apply these mitigations programmatically and automatically, you know, via, you know, things like everything from, you know, CIC pipelines to, you know, deployment tool chains, and so on. So there's actually an opportunity for, you know, DevOps teams and engineering and platform teams to drive more security, you know, throughout the life cycle of applications, you know, rather than thinking about it at a later stage. And at the same time, security can bring to bear, you know, their knowledge of, you know, how these techniques may fit together to, you know, within the broader set of, you know, attack or tactics, you know, to effectively think about what to prioritize first from a security standpoint, and so on. And so, again, the attack matrix provides this systematic categorization which allows your organizations to really focus in on, you know, where, you know, where they might be able to, you know, implement the highest value security controls. And then, and then go from there. Okay, that's, that's great. So let's dig in then on the steps that you've come up with to help people begin to protect themselves. Yeah, so, you know, it was like you were saying, you know, with with 40 individual techniques, you know, you might be thinking you might possibly have to do 40 different things, you know, for every technique in the attack matrix. And so if you're thinking that, you know, don't worry. Well, the attack matrix originally, you know, did not document any ways to mitigate against the various techniques. You know, our team at StackRox has done that hard work for you and outline best practices for protecting against each technique. And again, you can find those details, you know, in our blog series and white paper. But you know, more importantly, what I want to highlight today is that there are a few key steps you can take that will really give you relatively broad protection to start across the majority of techniques. You know, meaning that, you know, some mitigations, you know, are actually relevant to many of the 40 techniques throughout the matrix. And so it makes this, you know, makes sense to tackle these first. And, you know, our recommendation is that most techniques can be addressed by first using security features that are native to Kubernetes itself. And, you know, while others may require you to use things like cloud provider controls or third party tools, the majority of techniques outlined in the matrix can be addressed through, you know, native Kubernetes security capabilities. And today we're only going to focus on the techniques that can be at least partially mitigated using these native controls. You know, for the rest please refer to the resources we make available via our website. But, you know, our analysis across, you know, these techniques reveals that the best way to get started by applying these native Kubernetes controls, you can actually result in mitigating up to 35 of the 40 threats. And it doesn't have to be a ton of different controls that you apply either. If you use just three, you know, different native Kubernetes features, you will be well on your way to effective security against a broad range of attacks. Now, you know, one thing to keep in mind with this approach is that it also allows you and your organization to take advantage of the declarative and immutable aspects of cloud native infrastructure. So, as I was referencing earlier, it means that by applying these controls, you can proactively lock down your environment, you can do this programmatically you can do it in an, you know, a declarative API driven way, you know, to prevent attacks rather than having to rely on detecting and reacting to security instance after they've occurred. Another thing that we recommend in terms of using a native Kubernetes control is to configure Kubernetes role based access control or our back. And, as you can see as we've highlighted here this helps to at least, you know, partially mitigate nearly half of the techniques in the attack matrix. And some of the things to keep in mind when using our back. So the cluster admin role effectively grants unlimited access to a Kubernetes cluster and the users and groups that are granted this role should be highly restricted by administrators because any account compromises or errors can end up impacting the entire cluster. You know, second best practice or consideration that, you know, we suggest as part of using our back is you'll want to adopt a model of granting only the necessarily privileges to not just users but also service accounts as well. And then, you know, finally, there are, you know, a number of different, you know, features as part of communities are back. That have been incorporated over time role role aggregations actually a good example of this but you know some of these features can be used to simplify privilege grants, for example, could allow new privileges to be combined into existing roles. And these types of features if, if, if using a more sort of, you know, ad hoc manner they can actually result in, you know, in changes to the intended use of a role and then additionally role definitions. You know, sometimes they have overlap with each other and this can make it much harder to, you know, grant or revoke access to users or service accounts, because it becomes complex very quickly. You know, one of our recommendations is to start with the baseline, you know, in terms of different set role definitions and then you know incrementally add from there as necessary, you know, rather than sort of, you know, granting privileges or permissions to different users or service accounts like, you know, as those come up, I think it's much better to sort of outline your approach to RBAC overall, and that will help you down the road when when you actually start to scale and have a larger number of users and applications that are running within your cluster. And so, you know, this example, you know, is one that goes to show that you're using RBAC can be quite complex, and you do need tools to help monitor your RBAC settings, you know, that there are different ones available out there, for example, you know, just to highlight stack rocks, you know, is one of them that automatically monitors RBAC privileges on role bindings for users and service accounts and, you know, this can help identify whether, you know, any elevated privileges have been granted in an unintended way. Okay, before we jump there, we've got a one a couple questions here that have come in. It's a little related not entirely this person's asking about the Kubernetes dashboard. And that if we're looking at the components that can be a point of exposure that that's a well documented one do you have any recommendations for enabling it for supporting and monitoring of Kubernetes and how to protect it. Yeah, it's an interesting question I think I think almost a question that comes to mind or that would pose back is, you know, what would the purpose or why would you want to enable it be. And the reason I say that is, you know, the Kubernetes dashboard has been shown to be your component that has been, you know, leverage for different types of attacks. And as a result, you know, most major Kubernetes platforms or offerings, you know, do not deploy the dashboard or disable it by default. You know, even within stack procs, we have a built in policy that alerts when the dashboard has been deployed. So, so I would say, you know, you should really strongly consider only enabling and using it with caution. And if there's actually good reason to, and if you do, you know, ensure that you utilize, you know, appropriate network and access control settings, you know, some of the examples where the dashboard has been compromised, you know, involved cases where it was exposed specifically to the internet as well as it was granted elevated privileges which then allowed, you know, people to take, you know, certain types of actions that led to, you know, compromising aspects of the cluster. Okay, so if possible just avoid for that open door in the first place. And there's another question here you touched on cluster admin credentials do you have any suggestions for integrating Kubernetes authentication with LDAP or active directory authentication. Yeah, I would say a couple considerations or ways to approach it or break it down, you know, is to, you know, think about authentication authorization. And certainly, you know, authentication I think is going to be dependent on tying into and based on the particular type of platform, you're running and so, you know, if it's, for instance, you're running, you know, on prem and, you know, you need, you know, LDAP or authentication, you know, then, you're going to want to continue to, or if that's what you use for identity identity management today then you're going to tie into that. But if you're on a cloud provider, for instance, then you're, you know, likely going to want to, you know, hook into and utilize, you know, their, I am offerings, you know, for authentication, but then once you've handled authentication you're going to want to think about authorization and that's where you know Kubernetes RBAC comes into play and setting appropriate permissions on both your, you know, users that are operating within your clusters as well as, you know, the applications that are running and, you know, the privileges that they can take on the API server and other aspects of the environment. Awesome. Okay, that's really helpful advice. All right, let's get into the next step folks can take to use native coupon controls. Sure, the, you know, the seconds that we recommend is to apply a capability built to Kubernetes called, you know, Kubernetes network policies. And, you know, this can help mitigate several techniques in the attack matrix as well. Your network policies enable you to restrict both ingress and egress traffic to and from your applications, including pod to pod traffic so you can get to that level of granularity for network segmentation. So you'll need to use a network plugin that actually enforces network policies. You know, even though Kubernetes always supports operations on the network policy resource, simply creating the resource without a plugin that supports your network policy API will actually have no effect. So if you think that if no network policies are applied to a pod then you have no network traffic, you know, to or from, you know, is permitted but in fact, you know, in Kubernetes the opposite is true. If no network policies are applied to a pod then all network connections to and from it are permitted, which means, you know, by default, you know, all, you know, all network traffic is allowed and, you know, there's a good reason for this to even though it's advantageous from a security perspective which is it's really designed to enable operational convenience for users and so on. And so, you know, the best way to get started, which we recommend is to configure a default deny all network policy and you know this will help isolate all pods by default. Once this is in place you can add network policies to further enable either internet access or additional pod to pod traffic as is needed or warranted. And, you know, again network policy management as you scale your cluster scale the number of pods and applications that are running within it can get quite complex and so you know you want to look for tools to effectively allow you to you know, think about how you can more, how you can automate and manage these types of policies at scale and you know stack rocks as one of those tools helps with this type of configuration. There are a lot of aspects of visualizing to, you know, automatically generating and applying your network policies but, you know, this is, you know, it's a network policies and some of the other network capabilities within Kubernetes are an area of continued investment for the community and we look forward to seeing where all that continues to lead. At the end of the day this is really about kind of reducing the blast radius right and just like if you if somebody gets in what can they do and trying to scope that down. There's a question here about the the network side. Is there any difference between understanding the networking or applying networking policies on managed services where you don't have access to the masternodes. No, I mean I think in terms of, you know, even though the cloud provider for instance would be managing those components or say like the API server, you know, in that scenario, you know, there's not, that's not to say that there are, you know, any additional network restrictions in terms of what you're able to achieve. Again, you may want to look at what the default settings for some of those scenarios are for example things might be open when you don't expect them to you know, the API might have a public IP address, for instance, and so on and it may fall on you to actually configure you know and restrict network traffic appropriately so you know it's worth keeping that in mind but there would not be any sort of restrictions in terms of what you're able to achieve, you know, from a network perspective you know between your pods and applications and their need to talk to you know the controlling components and so on. That's everything for now way. Cool, and you know so we're going to the third step and you know what what we would recommend is to, you know, once you've thought about using, you know, native controls like RBAC and network policies, you know we then recommend that you harden pod configurations You know this can actually be done using several different in several different ways and using some options in terms of what communities enables. It's also I would highlight sort of an area of ongoing discussion and innovation within the communities community. The starting point for thinking about hardening pods is concept called security context and security context allow you to configure settings that restrict pods from doing things for example like running privileged or ensuring that the containers root file system is mounted as read only. You can set these settings at both the pod level, you know, or container level if you wish. And then can also enforce policies on the conditions that pods must meet in order to run within the cluster. This can be done using native capabilities like pod security policies, which utilizes an admission controller, or it can actually be done with using other ecosystem So the open policy agent or OPA gatekeeper and others if you look at some of the, you know, provide platforms out there like Azure community service they've taken steps to move towards implementing gatekeeper for enforcing policies, rather than utilizing pod security policies, you know, and then also, you know, keep in mind that since most pods are created indirectly as part of a, you know, Kubernetes controller, you know, which might be, you know, deployment or, you know, Damon said or things like that. The best way to authorize policies is to grant access to the pods service account directly. And so, you know, I mentioned some of the tools out there that can be utilized to help you enforce policies and ultimately harden and secure your pod configurations. You know, among those tools, as an example, you know, stack rocks, you know, will utilize, you know, dynamic admission control to enforce some of these policies as well. You know, running as non root user, you know, having a read only file system and so on. Alright, we've got one quick question. I think it was triggered by your earlier comment. What would be best practice for pod security policies to prevent running privilege pods. So there, you know, you don't want to look at things like, you know, pod specification and again like is one of the great things about Kubernetes and, you know, being cloud native is your declare declaratively, you know, configuring, you know, how a pod should run and the conditions that it should meet. So utilizing that pod spec to say, you know, certain criteria should be met, you know, for it to be, you know, deployed into the in running within the cluster and then while it's running a certain set of security configurations or settings that are applied. And then, you know, beyond all of these sort of, you know, configurations and preventative measures, you know, being important capability for, you know, any aspect of Kubernetes security is still being able to, you know, monitor, you know, actively what's happening at runtime, you know, when a pod is running and identifying either configurations or activity that is unexpected. And then that gives you a basis for, you know, further, you know, investigation and follow up. Okay, we have a couple questions, but I want you to go ahead and ensure these key takeaways and then just invite the audience. If you have any more questions, we're going to have a few minutes here after Wei wraps up that we can tackle those questions. So go ahead and send them over and Wei, help us with some quick takeaways. Sure. So, you know, we've now covered the major aspects of the Kubernetes attack matrix and how you can potentially put it to use. You know, the attack matrix gives you a systematic basis for thinking about how to protect against a broad range of potential attacks, you know, made up of, you know, one or multiple types of adversarial techniques. So native controls, native controls and Kubernetes can help you at least partially mitigate most of these. So, you know, recommend focusing on three to start, you know, these are configuring Kubernetes RBAC, and then security context and or capabilities like pod security policies to help harden your, your, you know, individual application components. And, you know, finally, while all these measures, it will certainly strengthen your security posture. When it comes to your communities clusters, no methods are foolproof. And you should always be prepared for the unanticipated and this means you're ensuring you still have runtime monitoring in place to detect your different types of activity that are indicative of attacks when they arise. And with that, you know, appreciate you listening to sort of our overview today of Kubernetes attack matrix and we'd love to open it up for questions. Awesome. All right, thank you for the great information way super helpful. And I think it's a really good point there's a lot we can do given the kind of infrastructure we're running into be much more proactive about security in terms of implementing a build and deploy time and getting those controls built into how we configure everything but still complementing that with the runtime securities is still really important. Okay, so there's a couple of other questions that have come in. We're talking about protecting Kubernetes. How important is it to keep upgrading Kubernetes and staying current people recommend staying current but sometimes cloud providers are a couple of revs back. What does that affect your security and when you upgrade. How does it affect running deployments. Yeah, so I think there are a couple considerations there, you know, with some trade offs to help you make the right decisions and I think that cloud providers give you several options, you know, to allow you to make the best choice for your organization based on those trade options. You know upgrading is certainly quite important from a security perspective you get the benefit of all the latest security fixes, even additional new security features you can take advantage of. You know, we have seen, you know, as a community, you know publicly disclose vulnerabilities that impact, you know, critical components like the Kubernetes API server and so on. And so those are, you know, best addressed by being able to upgrade on a relatively, frequent basis or at least staying close to the latest version of Kubernetes that is available. But I think the trade off is, you know, our operational considerations such as you know stability, you know, and ensuring that you know, especially if you're running production like not potentially not wanting to upgrade too frequently to minimize any disruption to your applications. And so the way I think that the cloud providers and other platforms overall have handled it to reflect this is that if you look at particular providers that they'll give you the choice of different either versions when you deploy clusters as well as different channels right if you look at GKE they're different channels that reflect, you know, the most stable, you know, versus something that potentially is less stable but you know has some of those benefits of being on a later version. And so I think that it probably depends at that point on what type of environment you're running, you know, how critical are the applications and workloads, you know, within that particular cluster and so on. And so I do think that, you know, you have choices when it comes to what version to be running on and so it's not to say that, you know, upgrade at all costs, but upgrading can certainly help with a lot of these vulnerabilities and Kubernetes that we've seen. Yeah, that makes sense. And just for people's reference, we've done a really comprehensive look at all three of the managed Kubernetes services and how they compare how they stack up what tradeoffs they've chosen to make in those managed services so you can check out the Stack Rocks blog for that just detailed review of AKS versus EKS versus GKE. There's another question in here. Is it possible to use namespaces as a hard security boundary? I think the way to look at boundaries, you know, across your Kubernetes environment are, you know, they're different delineations or distinctions, you know. And I don't know what whoever asked the question had in mind when they referenced hard, but I think you can look at physical boundaries, and then you can also look at virtual boundaries and I think both are important in applying in the context of Kubernetes. When we look at physical boundaries, you might consider, you know, clusters or nodes as, you know, important boundaries for potential isolation, depending on what types of containerized workloads you're running. I think beyond that namespaces does form an important form of virtual boundary for your application components that may be running across nodes. And so, you know, as a best practice, for instance, you may want to scope, you know, an individual application or deployment, for instance, to an individual namespace to gain the benefit of that, you know, greater isolation and that doesn't just mean type, you know, in terms of management, I think it means that, you know, you can scope controls, some of the controls we even talked about today, you know, to particular namespaces to think about, you know, sort of your overall security approach and how you want boundaries, you know, between workloads to look, you know, across your environment in a more consistent way. So I would say namespaces is considered a critical boundary, you know, within the Kubernetes system. And, you know, you should certainly consider applying it and leveraging it to your benefit, you know, to isolate and segregate, you know, different types of workloads based on the criteria you deem most important. You know, might be business criticality might be the data, you know, type of data that it's processing or interacting with and so on. Okay, that's really helpful. You talk about the importance of runtime monitoring. Do you have a system that you would recommend? I think there are, you know, the somewhat depends on what type of runtime monitoring you want to do we always recommend, you know, being able to ensure that you can monitor runtime activity at a system level for, you know, every single container within the environment and the reason for that is, you know, something malicious is going to execute it's going to execute within your at the level of an individual container and so this is why, you know, among tools you know stack rocks provides those capabilities. But again, it may come down to your individual needs. But, you know, as an example, you know, we will monitor at runtime, you know, all pertinent system level activity for every container and then help identify based on other contexts within Kubernetes, you know, what might warrant further investigation or what might be indicative of an actual attack and so on. Yeah, having the opportunity to compare it against baselines things like that looking what what's different can stand out better with this infrastructure. We didn't talk about secrets but this this person's asking. We talked about using native Kubernetes capabilities. What do you see as the pros and cons of using Kubernetes native secret resources versus something like a hashi court vault to distribute secrets. That's a good question and you know I think this is actually one that highlights, you know, in area security where, you know, you have choice, you know, and the ecosystem has, you know, provided you know various choices to you and you can choose, you know, for instance, secrets that are natively built into Kubernetes versus third party tools like vault and again I think it comes down somewhat to trade offs. I think the benefit of going with Kubernetes secrets is that, you know, it's completely built in it's much easier you don't have to worry or think about as much from an operational perspective, managing a separate solution or you know how to, you know, securely, you know, for instance maybe distribute those secrets and so on. I also think, you know, the communities team itself or a broader community has taken steps to provide greater security for secrets, for example, by adding encryption, you know, with an LCD of secrets at rest and so on. So I think there are benefits those are some of the benefits for going with built in secrets and Kubernetes. So I think if you're looking at something like hash code vault or their other secret management solutions cloud providers have them available as well. You know, some of those considerations are you might be looking at how you can have a single solution for secrets within Kubernetes but also secrets or other sensitive data credentials and other types of infrastructure environments. That may make more sense. And in that case, you know, you have to weigh the implication of some additional operational overhead or complexity. But I think if you're focused on Kubernetes, Kubernetes secrets is a great option that you should turn to first. Okay, I think we have time for just maybe one more question here. There's one come in that's come in. Well, you get a big thanks way so somebody said it's been super helpful session he appreciates your advice. This one is the stack rocks provide PCI compliance or other Fed ramp mandated compliance. We do. You know, compliance has is among is one of the several areas that we focus on with the stack rocks platform. We provide compliance for PCI as well as number of other industry standards or known benchmarks, for instance, so this is PCI HIPAA, as well as, you know, adhering to enabling compliance with best practices outlined by, you know, NIST SP that I referenced earlier, CIS benchmarks and so on. And we do work with a number of organizations that you know have PCI workloads running in Kubernetes for which that type of compliance is required. And, you know, aspects of PCI or there's some overlap with other types of standards such as Fed ramp, for instance, and we also have a number of federal agencies we work with, you know, to that, for which, you know, those standards are applicable. And so, you know, something that we look forward to continuing to innovate and lead in, as we have so far and that's because we see, you know, many different types of sensitive or compliance, or workloads that need have compliance needs, getting to getting deployed into Kubernetes in production. All right, that's super helpful. And, you know, we invite you to check out the Stack Rocks blog. Wei is one of our major contributors on that blog. He did the nine part series that looked at this Kube attack matrix in lots and lots of detail and went through for each attack. What's the mitigation you could take? Is it something that you could do native to Kube? Is it something you could turn on in your cloud? And so you can get more information there. We invite you to just take a look at the other technical content there. And of course, if anybody wants to take a look at the Stack Rocks software in action, we would welcome that opportunity with a demo for you. And with that, I think we're really at time. You guys have asked some great questions. Thank you for bringing them along the way. It makes the presentation a lot more fun. Wei, thank you so much for sharing your great insights on how to mitigate these attacks. I really appreciate your time today. Thanks everyone. Yeah. And the webinar recording and slides will be on the CNCF site later today. And we and all of CNCF and look forward to seeing you again online in a future CNCF webinar and we hope everybody has a great day. Thanks so much.