 Okay, let's get started. Thank you very much everyone for joining us today. Welcome to today's CNCF webinar, how to run Kubernetes securely and efficiently. I'm Jerry Fallon and I'll be moderating today's webinar. We'd like to welcome our presenters today, Joe Pelletier, VP of Products at Fairwinds, and Robert Brennan, Director of Open Source at Fairwinds. Just a few housekeeping items before we get started. During the webinar, we're not able to talk as an attendee. There's a Q&A box at the bottom of your screen. Please feel free to drop in your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of the Code of Conduct. Be respectful of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. And with that, I'll hand it off to our presenters for today's webinar. Take it away. Thanks a lot. Hello, everyone. Thanks for joining today's webinar. My name is Joe Pelletier. I head up the product side of things at Fairwinds. And we're really excited today to give you an overview of how to run Kubernetes securely and efficiently. We will spend a little bit of time today to talk about some of the reasons why organizations choose Kubernetes as well as some of the challenges they experience and some of the technical implications for the security and efficiency angle here. We'll also share a little bit more about configuration validation solutions and how they can help you with this challenge. So let's take a quick step back and just look at containers. And we all know that containers have really changed how software is developed. And at its core, container technology allows applications to be packaged and distributed across a number of different platforms. And a lot of this software packaging is shifting left into the development process. Developers are taking a lot more responsibility for containerizing their applications. And as they embrace container orchestration platforms like Kubernetes, they may also be exposed to the configuration of how those containers run in production. The second major trend that we're seeing is that container adoption is growing quickly. And according to Gartner, by 2023, 70% of enterprises will be running at least two or more containerized applications. Finally, Kubernetes has become the de facto standard for container orchestration and it's entering the mainstream. And in many cases, new projects or new development teams might be embracing Kubernetes as their platform of choice for running containers in production. And some of the reasons why organizations choose containers in Kubernetes is really at its core to enable teams to ship faster. At the end of the day, there has to be a business ROI. And that ROI is often measured in terms of code deployment frequency, the mean time to repair for applying security patches or updating a bad image, as well as the ability to provision and create new resources. And so when you combine containers in Kubernetes technology with cloud and infrastructure as a service, you can find that teams are able to get to production with their application a lot quicker, giving these businesses an order of magnitude and improvement in terms of their development process. But despite a lot of these innovations with containers and CUBE and a lot of the great business benefits that can come with it, there are still challenges. And there was actually a CNCF survey from 2019 that really helped unpack what some of those key challenges might be in using and deploying containers. And what they discovered was there's actually, the first challenges around that cultural shift, moving from containers to Kubernetes is really a change for a lot of development teams. And with that change comes new best practices, new concepts to learn and really new tooling. And so we'll talk about that today at a core. The second is really the security angle. With all that change, there's new security paradigms that need to be considered. And that also leads into the third challenge which is complexity. And complexity from our experience at Farowinds goes beyond just the technical complexity of containers and CUBE, but it's also the organizational complexity. And we'll speak to that in just a bit. So let's kind of poke at first the cultural shift and what's changing as part of the development process as development DevOps teams and security teams come together to embrace container and Kubernetes technologies. And depending on the organization, the development team might actually be the team responsible for building those container images. But we're also noticing that the responsibility around configuration, so the configuration for how those containers run into Kubernetes infrastructure, that is sometimes shared between development and DevOps. And finally, the DevOps teams and the SRE teams, they're responsible for maintaining the base core platform of CUBE. And that might mean selecting the right add-ons, configuring those add-ons and keeping all those configurations and software up to date. And finally, security. The security team definitely has a view and a set of requirements across the whole SDLC. And as it applies to containers and Kubernetes, there's new ramifications there as well. So here are some specific examples though of what needs to be considered from a technical perspective as organizations leverage containers in CUBE. At the container level, really understanding what your known vulnerability exposure might be and whether or not you're using trusted images becomes a key part of the changes that take place as you leverage container technology. And similarly on the configuration side, there's a number of different configurations that either can create security challenges or introduce reliability or efficiency challenges. On a security perspective, that you might over-permission a container. You might accidentally introduce privilege escalation or you might be missing readiness probes, liveness probes or the resource requests and limits that specify to Kubernetes how to scale up and down your app. And we're gonna cover some of these very specific examples later in the presentation to give you a sense of how these changes to the development process introduce new requirements that need to be considered. That other area that we talked about was really the complexity. And as organizations leverage containers in Kubernetes, different teams need to come up to speed on what this means for them. So with the developer, a lot of times the developers just really trying to focus on writing their application code and getting their applications shipped to production as quickly as possible. They're paid to execute on a roadmap and develop new features. And sometimes for the developer, the fastest way to get something shipped might be over-provisioning resources of saying, I just need to get this to run so I'm gonna allocate a lot of CPU and memory or it might be creating permissions that allow the application to run potentially insecurely. And it's just because they wanna get to production as quickly as possible. And a lot of that has to do with these new technical complexities that come into place with containers in Kubernetes. On the other end of the spectrum, you might have the SRE or the operations team and they're trying to ensure availability and reliability of the apps. And so as these teams receive applications and configuration from development, they're going to be looking for a specific set of best practices being followed. Like they wanna, for example, make sure that the health checks are implemented. So Kubernetes knows how to restart that application and scale that application accordingly. And finally, the security team and the leadership team, they wanna make sure everything that is running in production is secure. And that's where ensuring that you have your security requirements set as well as your checking for those things like known vulnerabilities and containers become really important for securing the infrastructure. So at this point, I'm gonna actually ask my colleague, Robert, who heads up our open source here at Fairwings to share a little bit more around the technical aspects of where security and efficiency challenges may emerge in cloud-based infrastructure. Thanks, Joe. So there's really three layers of this stack you wanna be thinking about as you develop a strategy around Kubernetes. At the lowest layer, you have the container level. This is where you wanna be thinking about the operating system you're running on top of and thinking about any CVEs that might exist inside of those container images, either at the operating system level or in the libraries that are being installed on top of that operating system. And you also wanna think about how those containers are being pulled as they get deployed into your cluster. One layer up from that, we have the deployment configuration. So this is basically how Kubernetes is running your container inside of the cluster. And here you wanna be thinking about how this configuration is set up. Are the containers being given excessive permissions to have too much access to the host? Are they being configured in an efficient way? Do they have health checks set? Are the memory and CPU limits and requests being set appropriately? All that kind of configuration. And then finally, at the highest level, you wanna think about the Kubernetes cluster itself. So here you're thinking about what add-ons are installed? Are those add-ons up to date? Is role-based access controlling in place? And is the principle of least privilege being applied? How are the cubelets being configured? Are my node sizes being chosen appropriately? Those sorts of things. So today we're really gonna focus on those first two layers. So namely the container layer and the Kubernetes configuration layer. And the difference here is that at the container layer, we're really thinking about the underlying operating system and the libraries that are being installed on top of it. Whereas at the Kubernetes configuration layer, we're thinking about how that container runs on a host node. What permissions does it have? So at the container level, really what we wanna be focused on are known vulnerabilities. So these might be vulnerabilities that affect the base operating system or might affect libraries that you've installed on top of that operating system. And these vulnerabilities could be exploited in really any number of different ways. They could allow the user to gain root access to the container itself, to the host that's running the container. They could allow the user to, the attacker to exfiltrate data, to trigger a denial of service. Really any kind of vulnerability could crop up in these libraries. One big issue that we see frequently is as teams will scan their containers once, say at build time or at deploy time, and think that they're safe as long as those scans passed. But as new CVEs get announced, your images that pass the first time could become vulnerable. So it's important to be continuously scanning these images over time. Another issue we see moves up to the Kubernetes configuration level, which is where you specify how those containers are going to be running inside your cluster. Kubernetes provides a lot of configuration that allows you to say give your containers a little bit more access to the host nodes. Typically your applications don't need this access, but if an application isn't running, often the easiest way to get it working is to just start adding new permissions here. Developers are often not certain exactly which permissions their application should need in order to run. If you have these permissions set, overly permissively, it could allow an attacker to gain access to basically spread their access if they attack this one application, spread that access throughout the cluster. So say if your container is running as root, that could allow the attacker to gain access to the host node. Another common misconfiguration issue is not setting health pros on your workloads. This is technically an optional part of the workload configuration, but it's highly recommended for all production deployments. This is really how Kubernetes can tell if your workload is healthy or not. It will repeatedly call the liveness and readiness pros to see whether your workload is responding appropriately. This is really important just to make sure that if your application does crash or become unresponsive, Kubernetes can tell that, kill it and start a new pod, and to make sure that as you release new deployments, Kubernetes can scale the old deployment down and scale the new one up appropriately so you don't experience any downtime during that process. Finally, another issue we frequently see is inappropriately set resource requests and limits. The big issue here is that it's hard to know ahead of time exactly how much CPU and memory your application needs in production in order to run appropriately. So often developers will either completely neglect to set these settings or they'll over provision. They'll ask for an excessive amount of CPU and memory, far more than they actually need just to know that their application isn't going to crash in production. And this can lead to huge cost overruns. You may be spending a lot more money than you actually need because developers have requested twice as much memory than they actually need. It can also lead to Kubernetes not being able to detect legitimate problems with your application. Say if there's a memory leak inside your application and you've got 10 times the memory you actually need requested, that memory leak is going to continue on for a lot longer without Kubernetes detecting it. So to recap, you can see security issues from both the container level where known CVEs are running inside your containers or at the deployment configuration level where a container has permissions to say run as root on the host. You can also see efficiency issues with missing health probes and with resource requests and limits not being set appropriately. That's great. Thanks, Robert, for sharing some of those technical details. It sounds like a lot of these issues come into play once organizations leverage Kubernetes and containers at scale and then need to have a way to monitor these configurations and stay ahead of these problems before they occur. Absolutely, yeah. That's actually kind of gives way into a new category of open source and software tooling that we're seeing in the Kubernetes space called configuration validation. So a lot of times, when it comes to ensuring that the containers are being tested for known vulnerabilities and that the configuration is also being inspected for the things that Robert went through, you need to leverage these configuration validation solutions so that DevOps teams can maintain those consistent configurations for the containers, the deployments and the overall cluster infrastructure as more and more teams end up leveraging and deploying into Kube. A lot of these configuration validation solutions are providing organizations with the findings as well as the fixed recommendations usually in the context of a YAML snippet or YAML code as well as controls for policy-driven enforcement that allow you to prevent these problems from being introduced into the cluster. And so ultimately, the business value of these configuration validation solutions is a way to reduce risk, save money and just avoid wasting a lot of manual effort that the ops team would ultimately have to take on. So we wanted to kind of give some recommendations for any organization that's looking to implement configuration validation solutions, sort of what are the key steps that you'll need to take in order to move in this direction. And really the first step is sort of the discovery step where you need to get the data around where you have potential misconfigurations around security or efficiency or even reliability challenges. Fortunately, with the Kubernetes ecosystem, there's a prolific number of open source tools and options out there. And a lot of them are really great for getting a grip on specific challenges or specific issues. Best place to start is going out and researching to find the right tools for the right specific area, whether it's scanning your containers for known bones or checking your deployment configurations or inspecting your cluster. Once you find those tools, you have to script them so that they can run on a regular basis. And then you'll have to do some work around analyzing the data. So that might mean deduplicating findings, ensuring that they have the right severity on them so that you can prioritize which of the issues that you wanna fix first. And then make sure you have the reporting and the integrations into dashboards and into downstream systems like Slack that may allow you to notify the end users at the right time of when they might, there might be a configuration challenge. Once you get the data though, the next step is really to assign ownership, right? You wanna make sure that there are owners to these things and that you're getting the information to the right end user. For example, like a CDE might be something that a development team has to take on, whereas an out of date add-on in the Kubernetes infrastructure or a misconfigured deployment might be something that a DevOps team or an SRE team may have to take on. And so depending on what you're finding as issues, you're gonna have to assign it to the right owners. And ultimately, we talked about how containers in Kube are new. They're changing the development process. The best solutions will provide you with some amount of remediation guidance so that you know how to fix these issues as you go along. Now, what we just described was a sort of a multi-step journey so around how to configure, validate your Kubernetes configurations. And fortunately, there's three different ways or three different paths you can choose. We do see some organizations building their own tooling so that they're taking their best practices and forcing that as their own auditing standards. And that's definitely one path. The second one is leveraging some open source and we'll share with you today some recommended open source solutions. And the third is leveraging a purpose-built platform that focuses on configuration validation as its core feature set. So the first one is around building your own tools. Obviously, the advantages of this approach is that it's gonna be fully customized to your needs and it might give your team some room to even learn about Kubernetes in the process of building these tools so that you're defining the best practices that meet your organization specifically. But as with any effort like this, it's gonna require some amount of time outside of normal business as usual activities. You're gonna have to plan for this. You're gonna have to allocate some engineering resource and then you're going to have to maintain the tools that you do build. Some organizations go down this path and it works really well for them. But there is an overhead associated with that mostly around the long tail of maintaining these tools as the Kubernetes ecosystem evolves. The next one is really focused on leveraging open source tools. And as we mentioned earlier, the Kubernetes ecosystem has many, many different open source options. We have a link here to a great GitHub page that provides a list of awesome open source tools. And because it's open source, they're free. You can download and start using them right away. And many of the solutions too are very powerful. They're more than just small little tools. They are powerful solutions that enable you to inspect and audit your Kubernetes environment. The challenge though with some of these tools is that they are usually focused on just one doing one thing really well, which means that you might need to use multiple tools in order to get a well-rounded view at your security challenges or your efficiency challenges. There's definitely different, each tool comes with its own way of running it and using the data. So it can be hard to operationalize them when you have to consume the data in multiple formats, which means that you might also have some challenges in getting support if it's just an open source project and there's no vendor that offers support, it might be a challenge to get the help you need to interpret the data and use these tools at scale. Which really leads, those two different options really lead into a third option, which is leveraging a purpose-built platform. And Fairwinds does have software in this space called Fairwinds Insights. There is a use case around being able to aggregate the findings and leverage the tools that are best-of-breed in the Kubernetes ecosystem. So you have one platform that can audit your Kubernetes infrastructure for security, as well as efficiency and reliability challenges. The great thing about this is that it is ongoing support, it integrates the best-of-breed open source so that you still get to leverage those same great tools. But the value is really on adding, organizing the data, helping you prioritize it, and ultimately implement policies so that you can enforce the rules specific to your organization around what's allowed and what may not be allowed. Obviously the downside of solutions like this is it generally requires some amount of budget in order to pay for these tools. And some of the solutions may not be fully open, open source, they might contain some open source components and then have a certain amount of commercial components to it as well. So if you're interested in purpose-built platforms, the best ones that we've seen have the opportunity to prioritize findings from multiple open source tools. So having that framework where the vendor is adding more and more tools over time and making it easy for you to integrate data from third parties into a single pane of glass, that is one of the key components of a configuration validation solution. You'll also wanna be able to integrate the data into downstream systems that you use. So that might mean CICD systems, Slack or chat ops systems, as well as monitoring tools like Datadog or Grafana. The other key consideration here is run these configuration validation platforms. They really need to run in multiple locations. You wanna think about it as a way to check these configurations in the CIE process, as well as right before deployment so that you can have a validating webhook that allows you to validate whether or not the deployment's meeting your best practices, as well as in production. So you can understand what's already gotten into production that may or may not be in compliance with your standards. Finally, any of these solutions should also be able to address containers, configuration and cluster level type of risks. So here's a brief kind of illustration that shows the ideal integration points, being able to integrate into a CICD context. Some of these tools are great for running earlier in the process like scanning your containers for known vulnerabilities. That's something that can happen at the time of the build phase, as well as scanning your Kubernetes manifest or your YAML files, your Helm charts. That's also something that can operate on the file that comes from your Git repo. At the pre-deployment stages, as well as in the cluster, you might have an opportunity to do a lot more security scanning as well as resource efficiency and optimizations. So one example is Fairwinds has an open source tool that is called Goldilocks, which helps you with optimizing your workloads to be resource efficient. So that means we're going to give you recommendations for setting your CPU and memory requests and limits so that you can keep your workloads right sized and avoid them from being starved in production in the sense of some workloads not receiving enough compute or resources, as well as not spending too much money and over allocating those back compute. So finally, you know, let's recap on what the business benefits might be. You know, when you can configure, when you can validate your Kubernetes configurations, you're putting your business in a position where you can improve the overall security posture of your containers that are running, as well as the configuration that describes how those containers should run in Kubernetes. With configuration validation, you do have the opportunity to prioritize so that you're not just addressing all the problems. It will help you understand which are the most important. The second business benefit is going to be really focused on cost and efficiency. Just, you know, security is very important, but so is also making sure that your workloads are running efficiently and reliably so that you're not overspending and you're not also starving some of your most important applications. And finally, any of these platforms really need to demonstrate not only a financial ROI from a cost or security perspective, but also a time savings perspective. How can you ensure that your ops team is not manually inspecting these configurations on an ongoing basis? And how can you provide a platform that allows you to bridge that gap between developers, SRE teams, and security teams so that they all have a single pane of glass to collaborate from? So with that, we wanted to make sure that you were all aware of a great open source guide or a great guide that we have on thefairwings.com around helping you understand the best ways to manage Kubernetes configuration for security, efficiency, and reliability. Or if you're interested in trying out some of the tools that we had discussed today, feel free to check out fairwings.com for a free trial of our insights offering. And with that, I'll turn it over to Jerry for opening up any sort of Q&A that we may have at this stage. Okay, well thank you both for a wonderful presentation. We have plenty of time for a question, so if anyone has any questions, please feel free to leave them in the Q&A box and we'll get to as many as we can. So there's a question we have here. What is the best way to get started? Yeah, that's a great question and I'll go and take that first. So we talked about how configuration validation platforms, especially purpose-built ones like fairwings insights, provide a bunch of tools out of the gate. But another great way to get started is just by going back to the open source community, the Kubernetes open source community and looking at the individual tools that may exist. Give you an example, we at Fairwings have developed a open source solution called Fairwings Polaris, which you can run very quickly to understand where you might have certain configuration issues with the workloads running in your cluster. That's a solution that we see a lot of folks gravitating towards in order to get a quick view in terms of where their risks might be today. Similarly, there's also a variety of great container scanning open source out there. Trivif, which is maintained by Aqua Security is a fantastic container scanning open source tool that many organizations use to understand the risk of their containers. And finally, we mentioned about Fairwings Goldilocks and other open source offering that focuses on resource optimization. So there are great open source starting points. I think that is a way to get started if you're focused on a very particular problem. If you realize that you find value in all these tools and you wanna run them on a consistent basis, that's a great point to then graduate to a configuration validation platform like Fairwings Insights. Robert, I don't know if you have anything else you wanted to suggest though as well. No, I totally agree. I think in terms of getting started, the open source community is, that's where you're gonna get the quickest bang for your buck in terms of just being able to get a quick overview of what's going on inside your cluster and maybe where the biggest issues lie. And then as you mature along your journey, I think a larger platform like Fairwings Insights is very helpful. Do any of these open source tools confirm to security benchmark standards? Yeah, that's a great question. So there are some standards out there like a popular one is the CIS Kubernetes benchmark. That is a Kubernetes security benchmark that's published by the Computer Information Systems Consortium. I may have gotten that wrong, but it's a very common one for understanding from an automated perspective and a baseline perspective is my cluster aligned with security best practices. There are some great open source implementations for that benchmark. I believe we leverage one called CUBE Bench which is also I think managed by Aqua Security. It's an open source tool so anyone can use it. And it gives you a quick view in terms of where do I line up right now in terms of my cluster as it's designed and running? Am I violating any sort of obvious compliance in security issues? And so that's a great one to start with. We do hear about standards like PCI and HIPAA and SOC2 and depending on the specific requirement there, there are some Kubernetes aspects to that, but that's where you probably need to look at more commercial type offerings. Does Fairwinds Insight also take care of application level misconfigurations or vulnerabilities that could be present due to bad programming? Good question. I think that's, I see they're referring to injection as an example. So Fairwinds Insight does not focus on the actual application code. There are a variety of application security type solutions out in the market that are open source and some commercial, but we focus more at the container, the Kubernetes configuration layer and what's running in the cluster. So we're focusing kind of on the layer just below that. If you are interested in application security solutions, there are definitely some open source ones out there. And in general, those are great complimentary solutions to what we're doing here as well. When you think about the need to validate your Kube configurations and test your containers, that is something that you should do in conjunction with testing your application code as well. Any more questions at all? Are there other pen test tools for Kubernetes besides Aqua Security? Yeah, that's another great question. I think there's probably several dozen different auditing tools I would say in general for Kubernetes. We actually shared a link earlier in the presentation to a number of those tools. I'll actually just put it here on the screen for a second. There's great lists on GitHub that share lists of auditing tools. Some of them might be penetration testing specific as well. What Farowinds has found is that you actually wanna take kind of a comprehensive perspective. So while you do wanna do pen testing tools, you also wanna make sure you're implementing checks on your containers, on your configuration, as well as what's running in the cluster itself. And so that will give you more of a comprehensive view at the configuration and security posture of your infrastructure. But a great starting point might be going to this GitHub page and looking at some of the auditing tools that they have available. Okay, anyone else? If anyone does have questions, please leave them into the Q&A box. It'll be easier for us to spot if you can. Thank you. Anyone at all? All right, well, no one else has any other questions and we'll wrap this up a little early today. I wanted to thank both Robert and Joe for their presentation today. Thank you to everyone for joining us. Have a wonderful rest of your day. Thanks folks. Thank you.