 Okay, hi everyone. Happy Friday. I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, Zero Trust Services in Kubernetes. I'm Christy Tan, Marketing Communications Manager here at CNCF. I'll be moderating today's webinar. We would like to welcome our presenter today, Randy Abernathy, managing partner at RXM. A few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen. Please feel free to drop your questions in there and we'll get to as many as we can at the end of the presentation. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of all your fellow participants and presenter. Please also note that the recording and slides will be posted later today to the CNCF webinars page at cncf.io slash webinars. With that, I'll hand it over to Randy to kick off today's presentation. Take it away. All right. Thanks, Christy. Hey, good morning. Good afternoon. Good evening. As appropriate, everybody. This is the Zero Trust Services and Kubernetes talk. And the idea behind this webinar is to sort of cover a bunch of security concerns that face developers building services for modern cloud native types of environments. These days, developers are often faced DevOps teams in general are often faced with very sophisticated pipelines that move, you know, code pushes through continuous integration then on to continuous deployment and finally ultimately into a Kubernetes style environment. And in many of these scenarios, the pieces of the puzzle are broken apart into many components. And so sometimes it's hard to get a handle on the overarching security concerns and how all the different pieces work together. And so the idea behind this seminar is really to talk about Zero Trust, but also, you know, sort of combine that with least privilege perimeter lists and security in depth other types of concepts that are really all ultimately related in many ways from the viewpoint advantage of a service developer. And then to kind of show how each of the different stages of the pipeline have their own considerations and concerns security wise. But there are tons of security topics out there, of course, that we could talk about specific to administering clusters or administering containerization processes and hosting. But we're going to really focus on the, the, the vantage of the developer, the DevOps team, the, you know, service reliability engineer, whatever the case maybe the people concerned with building those services and trying to do their job to secure them as much as possible. So, in this particular context, we could sort of define Zero Trust as a model where components trust nothing. And every single thing that is done by a particular component needs to be authorized. So authentication and authorization, of course, being the component there. And so in cloud native environments, we have this sort of mental model of a perimeter list platform. Right. If you are running your software in a orchestrated cluster, where is that cluster. It's not really supposed to matter. Right. It could be on Amazon or Google cloud or Azure. It could be in your own bare metal data center. It could be in a multi tenant environment or a single tenant environment. But if we want our applications to be robust and to operate in any of these settings. We just sort of need to assume that there's no protective perimeter. Right. And really, when you think about security in depth, it's best to assume that anyway right so Zero Trust sort of promotes a lot of concepts that really have been around for quite a long time. And we can focus on them and that's always a good thing. And so everything has to be verified before access is granted. Right. And we can look at that in a lot of different ways. When we think about perimeter lists, right, that's that's the thinking of there is no safe perimeter, which means then of course you can't trust anybody because there's no perimeter you can't say that just because you're coming from host a that everything's okay. Or that you're coming from host see everything's okay or you're coming from the internet and things aren't okay. You shouldn't trust host a or see any more than you trust the internet, because you're running in a cloud that has multiple tenants and the administrators that Amazon aren't your employees and you don't know, you know, what they may be doing what they have access to their bad actors, you know, cropping up everywhere so we just sort of need to encrypt everything on the wire, we need to encrypt everything at rest, we need to think about off in all scenarios and you know that's just sort of perimeter list thinking least privilege falls into this as well. You want to give every component of your system the smallest set of privileges that you can possibly give it. And this goes back to zero trust right trusting nobody right except the specific authenticated authorized activities right and so not only do I make sure I know that that host B is host B, but I make sure that host B is only doing what host B is supposed to be able to do. You know, cranking it down to another level and security in depth means making sure that you're dealing with all of these concepts, every opportunity you get. So what we're going to see is this is going to apply to service engineering so the software development process is going to apply to containerization it's going to apply to the deployment of your applications in a cluster like a Kubernetes environment, and then at runtime, while the applications up and running in those environments as well. We're going to look at a lot of different things to sort of bolster this we're going to look at minimizing attack surfaces. We're going to talk about assuming that you'll be compromised. And if you take that stance assume that this piece of software will be compromised. What can you do such that when that, you know, fable day happens. The attacker is bereft of any actual real ability to privilege escalate or to break out or to attack other components within your system, right so assuming you'll be compromised and taking that advantage. And this is hard I think for a lot of engineers because we want to defend ourselves and make sure that we create this iron clad barrier around all of our software. But in the end, there's always tricks right and there are things outside your control that could occur, specifically, when you're in a containerized environment because you have the container platform itself. Whether that's Kubernetes or something else you have the containerization process, you have the supply chain now all the build tools and things like that so there's a there's a there's a lot of opportunity for for new attack vectors to be leveraged, and some of them are outside the control of the developer so if we assume that will be compromised, maybe we can reduce the impact of that compromise off all connections and traffic obviously provide software with the smallest set of permissions. Right so so give make sure your service has the just the permissions that it needs. And one of the things that I find traditional engineering sort of skips is really taking ownership of the runtime nature of the application even in some DevOps environments. If you ask a team hey, what are the exact capabilities and access permissions that your software needs, they find it hard to articulate and then when you flip it the other way around and say, who are the exact legal clients that should be accessing your service. Often, again, may be hard to articulate. And this is stuff you've got to articulate if you're going to create any kind of policy that's going to control this stuff. It sort of ties into resource consumption and things like that all of these new things that we need to think about when we're operating in a shared pool, we used to run on our own machine. We would think okay, this is my box I have the memory and the CPU it's got. When you run in a shared pool of resources like a container orchestration environment, you have to say how much memory how much CPU you'd like to have limits and requests and things like that. And it's the same for the security side of it no longer are we on host be and we trust host see and just don't worry about it because they're behind the perimeter. We have to think about a lot of other things and we have to articulate these things in order to manifest them in some sort of enforceable policy. Emitting security telemetry is another piece of the puzzle. If you are building services and if you assume that you're going to be compromised well one of the best things you can do when you're compromised is make sure that everybody knows it. And so, you know, obviously that there's there's there's lots of third, you know, party tools you can use for surveillance and and the the site of the attack is not always the best place to try reporting things. But there's a lot of things you can report that are security related in your telemetry specifically in login logging so that it has no impact on your application but that you give people who are monitoring the environment that extra leg up that they might need in order to deal with the with the an occurrence understanding and controlling new attack factors and containerized environments we're going to talk about that and then never packaging unrequired bits. We'll see a lot of containers out there in production environments that have all sorts of stuff inside the file system of the container that's not needed by the service. And there's a trade off here that we'll talk about but we'll also take a look at how to really strip it down to the minimum. We'll talk about never exposing unrequired functionality. And there are scenarios where people do this accidentally, and in the name of observability, often. And so we'll, we'll take a look at some of those concerns as well. So lots of things to think about. Look at securing small services micro services whatever you want to call them or any kind of a service actually that you're going to deploy into a orchestrated environment. We have these kind of five stages that you might want to think about. You could break it down other ways of course but this is not a bad checklist service design and construction. Right to the actual software engineering process right you have to build things the right way so that they can be deployed in these kinds of environments. Correctly so that they can be observed but also secured and then service packaging and container image design. This is another step where we're taking that service or putting it into an immutable containerized image and then deploying it and how can we make sure that that image is as secure as possible. Then pod specification. This is how we're going to tell the system to deploy these containers when they're run and the pod specification could be a template inside a deployment or it could be something that's, you know, being set up in a replica set by spinnaker or that pod specification is really the stock and trading Kubernetes and it's for the most part almost purely up to the team that builds the software to define that pod specification because it's the software engineering team that knows how that container needs to run. What are the resources it requires what are the environment variables and configurations that it can tolerate or should have. And then step four platform based pod policies so as an engineer working in a cloud native environment you're also going to have to face challenges maybe outside of things that you're familiar with when you're when you're going to a Kubernetes environment. We can have policies set up in namespaces that constrain what is doable by a pot. And so we'll look at that and talk about pod security policies and other things that fit into that category and finally networking and the services that we expose are going to have, you know, the traditional you know, IPs and ports and things but there are a lot of very robust techniques that you can make use of in a cloud native environment to augment the policies that you would normally implement, you know, in firewall, you know, traditional firewall rules to perform more granular controls over what can be done and where it can be done from and there's also new challenges of course in a dynamic environment where things are being created and destroyed by the orchestrator. Due to scaling concerns or just, you know, rollouts of new versions of things and that sort of stuff. So that's that's the list that we're going to go through. So we're going to hit each one of these. And, you know, I've got a short little demo for each of these set up that I'll do for you guys. And at the end of the day, there's, you know, there's a lot of time you could spend on each one of these. And I'm not at all trying to give you a comprehensive view of any of these things, but just trying to give you a little example of the the sorts of stuff and to make it you know give you some context around these topics. So let's start off with service design. So at the beginning you have to write some software. And there's a lot of things that are asked of developers who are building software for cloud native environments. You may be asked to emit metrics data that can be scraped by a system, some some some SAS system, but like, you know, data dog or or a cloud environment, or Prometheus or something like that. And then you may also be asked to emit certain kinds of logging information, that sort of stuff. And these are opportunities to report security events and security events aren't all bad right we're not saying, you know, Oh, this, this terrible thing that I, you know, denial of service attack occurred at such and such time. That's great stuff to report of course as well. But which might also want to think about is just providing a baseline of information that's detailed enough that gives people the opportunity to know when something changes. So you can do a lot of this stuff at startup and really avoid any, you know, burden on the application at runtime. For example, why not log the user ID and group ID that the service is using. A lot of times this is specified by the cluster not the developers right we we probably don't care what user ID or group ID. We're using when our services running and maybe the in the cluster environment that has some specific meanings and making sure that we're using the user ID and group ID that we're supposed to be using is something that can be audited of course, but it's also something that can be a matter of record and logs and gives us some traceability capabilities. As we'll discuss our sets of functions that can be invoked in the Linux kernel. That are grouped together by these capabilities like net admin, for example, allows you to bring up a down network interfaces things like that, logging the capabilities that are enabled. Other system call permissions logging volume mounts that are present inside the container logging the namespace IDs I know I know it's that that are present. So just capturing a lot of startup stuff. I've, I've built some, some pretty large trading systems at in the, you know, kind of finance space, and we've found this stuff to be really, really useful and we just sort of loaded all sorts of, you know, really useful security related types of information that described basically what every single, you know, thing was accessible to that that particular service, so that we could then, at a later time, you know, just out of it, make sure that nothing was changing or ask ourselves the question. A lot of times just going through the process of making sure that you're enumerating all of the resources you've got access to from a particular service helps you rethink, you know, do we really need access to that, you know, maybe we should drop that and figure out some way to limit that obviously logging connections in and out. And then another interesting thing is to think about some of the endpoints that you build into your services, just to make them work in a dynamic environment. For example, metrics endpoints health and readiness endpoints, all of those types of things are often exposed to provide observability functionality, but that's a discrete set of functions independent, typically of your applications behavior. So your application endpoints probably should not should be should be distinctly controllable right you should be able to say these people have access to the application endpoints and these resources are for things where people have access to the observability information. The observability information often will give you deeper insights into the application which may be very inappropriate for a lot of parties so there are some, you know, some some easy techniques we can use to handle that, and then obviously mtls everywhere. Now, there's an interesting a set of occurrences in cloud native environments that have identified a lot of these things I may actually go back a slide that have identified a lot of these things and said you know what, it's going to be a lot of work for us to make it so that developers log all of the connections in and out. It's going to be a lot of work for us to have all the developers create mtls and manage certificates and stuff like that on all other services. There's going to be a lot of work to do rate limiting and all these other things that that would you might need to do to sort of make our services more robust and in that vein, service meshes have popped up. And so we've got the ability with service meshes things like Istio and and linker D and so on. We have the ability to apply these meshes to our dynamically orchestrated environments and augment all of our concentric services with proxies that implement the, you know, many of the security and observability features that we're looking for. So for example, if you deploy this Dio, and you told this Dio to inject the on boy proxy into all the pods in a certain namespace, then the developers just are going to be able to deploy their applications and get mtls for free. They're going to get connection tracing in and out for free. And a lot of other things. And that is great in two ways. Number one, it means you don't have to do it. And number two, it means you can't not do it, because it's being enforced at the governance level right it's, it's applied to your service no matter what. And so this is a wonderful thing. But the downside is, you know, you're going to take a multi millisecond hit on your, you know, latency between services because you've got proxies on both ends. So it's not for everybody, but it's definitely an interesting thing to think about. So demo service design, I'm going to go ahead and just, you know, do a simple example to show you how this impacts a developer the types of things that you need to think about and do to, you know, sort of get us hip to some of the basics here. So again, you know, keep in mind, these are just super simple demos designed to, you know, give you some context here that type of work that's involved. So I've got a little bit of a crutch script here set up so that I don't have to make you guys suffer my typing. So I'm going to go over here and I'm just going to clone a repo that we've got set up with a sample application in it and I'll use this application for a few different demos here. Okay, so the next thing I'm going to do is build this thing. So I just cloned it. And now I'm going to build it. Let me list the directory here. You can see that I've got a simple Go program. We'll talk about this in just a second. And then we've got a Docker file that is set up to build this, this Go application into an image that we can run. So I'll, I'll walk through the Docker file in just a second. But I'm going to go ahead and just build it. And there we go. Okay, so the image is TL and it's built pre built it so that we go a little bit quicker. And now we're going to go ahead and run this guy. And so when we run this service, we can grab its IP address and hit it. But being just any user fix the container ID there just grab the container ID. Okay, so we're just we're just inspecting the service to get all of its metadata and looking for its IP address there it is. 1717 0.2. So we're, you know, we're working with our app. We've built it. We've containerized it. And now I'm going to curl it. But what I'm going to do is I'm going to curl the metrics endpoint in this service. So this guy's got to And we can see that I have no problem hitting the metrics endpoint. And the problem with this is that the application. The application features. Are also here. So for example, this guy reports the trash levels of trash cans in a given city and will tell you how full they are and help you figure out when to service them and that sort of thing is to hypothetical application. And so curling this guy. It's set up to be delayed on purpose and then to perform faster or slower based on memory that you've given it. But the idea here is that we say, okay, this trash level is at 15, but at the same exact IP and port. I'm exposing the metrics endpoint. Now. You may have some very sophisticated firewalling capabilities where you can say these paths are accessible and these aren't. That's fine. But the, the simpler thing to do would be just to simply say, hey, look, we probably don't want metrics on the same hosting. Port as the, you know, the, the main application functionality. And so to change this would be fairly straightforward. Right. We could just drop into the application. And this is just a simple go program. And we're using open metrics. Prometheus libraries for open metrics. To expose standard metrics. And then we just got some counts and things like that that are being added here. These are a bunch of just handler functions. But down here in the main area, of course, we can see that we're listening and serving that port and all of our routes, even metrics and the, the readiness performance. Probe that we've got set up and the health check that we've got set up. All of these guys are on the same port. And so creating any sort of easy and simple separation is perhaps a bit problematic. So what we could do is we could just break it down. And change the port for the metrics. Just set up a new router. Right. So I'm just going to create a second router. And, you know, if you're not familiar with go, that's fine. I mean, you get the idea, right? We're just creating a router and routers in many programming environments are just tools that allow you to automatically handle paths and hand that, that, that request off to a block of code. And we're using the Prometheus HTTP handler to handle that. But we're going to move that handler. We've created a new handler and a new port for that guy to listen on. And now we'll just drop in a new listener. To deal with port two. And to use the second router. And now we've got a discrete port for our metrics to operate on. So if we rebuild this application to a block of code and we're using the Prometheus HTTP handler to handle our metrics. But we're going to move that handler. We've created a new handler and a new port for that guy to listen on. And now we'll just drop in a new handler and to use the second router. And now we've got a discrete port for our metrics to operate on. So if we rebuild this application. Oops, didn't mean to do that. Let's go back here and build it. There we go. So we're going to rebuild the application has to recompile the change source code. So I didn't make any mistakes. It'll actually work. Okay, so if we run the new guy. This is a E 63 now. We go back up and try to curl E 63 watch it. Let's get his. Let's get his IP address. Okay, and so if we go back up and try to curl dot for that works. Right, we can still retrieve the application state. We can tell it works because it didn't give us an error. Just take them a second to respond. Heavy processing here to get the trash can level. Okay, so that works. And then if we try the metrics endpoint. No good, right, but if we change ports. So we're happy that the first curl failed because we were hitting it on 8080 and then I changed the port to 81 I think let's make sure 81. Okay, well, it looks good. Let's do a dump here. Some of the containers we've got have got Kubernetes running as well. Okay, well that looks like the guy so I'll, I'll, I'll debug the, the source there. But as you can see, we can no longer get to the metrics port on port 80, which was really the goal. Right. And then obviously making the metrics work again is important. But, but yeah, you know, just a simple, a simple code change like that. Gives us the ability to now easily distinguish between the operational aspects of the application and the, you know, the, the standard application components of the application. So a lot of times when we're building small services, you know, we need to think about the fact that we're going to operate in an observable environment and yet these observability features need to be segregated out of the standard, you know, application operations. So the next piece is container packaging. And in the container packaging side of things, we have concerns around container images images are the stock and trade in a containerized environment when we deploy an image into a cluster that image brings with it to basic things. Metadata. So a config.json more or less with some, you know, high level attributes of the image, things like the images name, the images, you know, metadata on the label front, you know, think key value pairs like licensing information, you know, what have you, what command to use to execute the program when the images executed without an optional command from the end user and that sort of stuff. Then the other piece of the puzzle, in addition to the metadata in the config.json is this basically what equates to a tar ball of the root file system of that container. And so the concept of building a bunch of stuff into the container image or using other images to create the image that you're going to run in production introduces the opportunity to include additional information in that container image that's that's that doesn't need to be in a production environment. And tools like introspection tools like net stat and PS are are really useful to attackers. If somebody breaks into your application and somehow gets the ability to exact another process, you want to limit the options that they've got you don't want a shell in a container image, unless there's a specific reason to have it, unless it unless you're using that shell to interpret the thing that's running in that image. And so the fewer tools we can provide in our running containers, the better off we are, and it is quite possible to create a container from quote unquote scratch, but we start with nothing. And we put explicitly what we want into that container image. Now, there are images available all over the place, many, many on Docker Hub, of course, that it is convenient to base other images on so developing a Java app on top of the publicly available Java image makes life easy because the Java image already has a lot of stuff in it and more and more images on Docker Hub are getting stripped down and there's a lot of stuff in there, but still, quite often they've got a lot of stuff in them that, you know, we don't necessarily need. And so, being able to build images that basically contain nothing is, is really going to be a big help in, in locking things down and and providing as few, you know, hackers to take advantage of as possible if they somehow are not a compromise your container. And so let's take a look at an example here. So I built this, this trash levels example a couple of times, and I'm just going to, I'm going to go ahead and list everything that here in the directory. And in the Docker file that builds this particular trash levels app, we've got two stages. The first stage is grabbing a public image go lang 113 and using that as the build tool. Now this right here might set off alarms, right in many environments, you're, you're going if you if you have serious security interest, you probably don't want to ever use a public image. Right, and you probably don't want to grab it in this way. There's no way when you look at this image to know what is what that image contains right there's no way to know the bits that are inside of that image. We can, we can make assumptions, but bad actors in various places could, you know, could, could subvert that image we can't control Docker Inc. We can't control people who can push to the official repo where this is coming from. So it's probably not something that should be trusted in the first place. And so you want to always make sure that your images are coming from a trusted place where they've been, you know, screened by your own internal security processes and bedded and that sort of thing. So if you've got standard based images that you're going to be working with developers are often going to have language specific based images Java, go, what have you Python, those should probably be built in house bedded and, you know, locked out. But we're going to go ahead and use this public image just to get started. And then we're going to copy our program into go source trash levels. We're then going to pull down the libraries that we need and we're going to build this guy, but we're going to statically link it. And when you statically link an application like this, it doesn't have any dependencies. Now, it's going to be a little bit trickier for a net or a Java application you're going to need to run time same thing with Python and so on but see C++ go and various other programming environments, give you the ability to compile to a static executable that has no dependencies. And even if you have dependencies, of course, you could include just the dependencies that you've got and avoid all the other stuff. And so our second stage in the build here says from scratch. And this in in a in a docker build and there are other tools you could use to build but I'm just using an example here. We're saying start this new container image with nothing. Right. And then we just got some key value data, which is just metadata that just goes into configuration. But then we go to the previous image, right, the build image. And we grab the executable from that build image, and we copy it in. And then we specify that we're going to run this program when somebody runs the image. And that's it. And so the build process that we ran just a minute ago is pretty straightforward right we just tell docker hey build the current file and then tags it whatever we want to call it. So if we want to build this guy again and retag it v2, it'll see that everything's already been built and use the cache but off we go. And so that gives us the ability to to now do this. Take a look at the actual image so this is the image that I just tagged right and it's 13 megs. This is the intermediate image. This is the build stage, and the build stage is 910 megs. And you can see the build stage is 910 megs, largely because the go environment is 803 megs. So we, we have 803 megs of Golang, then we've got all the library sources that we pulled down, and we have all the intermediate files for the build process and what have you. So the last thing that we want to ship off to production is a container image that has a complete build solution in it so that if if compromised an attacker can now write software and compile it and deploy it right from within our container so definitely definitely something that we want to strip out we want to have the minimum amount of code actually out there in production and so here's an interesting thing that we can do if we do a docker container ls and we take a look at the container that we just ran. Let's take a look at this guy right this guy's running that trash image and that's the same thing that we've been using so that's e 63. So if we docker exact a program into e 63 and let's for example list the root directory. We get this error can't execute LS so the LS executables not in there right how about PS Nope, how about let's run just shell so that we can jump inside that container and poke around nope right so you're going to be very very restricted as to what you can do in an image that is so constrained the other hand of this is that okay you know what if I need to debug things so this is going to make it a little tough to debug right you might want to run the image in the build container for experimentation and debugging and then you know run the production container in the production environment so you have to you know kind of deal with your CICD split there are we going to use the build image for you know for for say for example unit tests and then switch over to the production image for all the integration tests and you know non functional tests and things like that that's typically a you know a clean clean split but every environment's different every every you know team will will find their own boundaries. Another thing to think about is if we docker image LS help here you'll see that there's a digest switch and if you want to be very explicit about plural if you want to be very explicit about the images that you pull instead of specifying that you would like to grab a repo and then colon tag like colon latest colon be to something like that you could instead ask for the shahash and that means that you're going to get those bits period into story you will get those bits every time all the time because those are the only bits that docker will accept and so while you still have to download the image when the image is downloaded the the container manager is going to check the shahash against the hash that it was before and if the bits aren't the same somebody mutated that image and whether it was the people releasing v1 183 and they just decided to update it without updating the tag or if it was an adversary doesn't really matter right you don't want those bits. So this gives us another way to you know sort of improve the veracity of our applications content and you know so we have the ability to strip it down verify it things like that. So quick look at minimal containers. So the next step in the process is pod specification. And so when we're specifying pods we're saying hey we've got some containers that we've created and we want to run them and these are the the the characteristics with which we'd like to run them. And so things like what user that container is going to run under is something that you can specify in your pod. Specification and so assigning unprivileged users and groups is a good idea. Now it's interesting. We kind of go through some of these things here that are good practices but you may be forced to do them as well as we'll see in a bit. So assigning an unprivileged user group and there may be a selection of them to choose from with different you know capabilities so you may need to make the right choice there but definitely avoiding root at all costs if you're building an application service. You generally don't need any permissions of special you know capabilities or or features you're just going to be a running application a normal user is is mostly good if you're just simply listening on a network interface and handling service requests. So the developers that are building application level at you know components don't generally need a lot of the you know fancy features that provide you extra permissions just the basics are going to do so pick an unprivileged user and have at it. When people are building operational services and stateful services and things that you know are kind of part of the platform often they'll need to rev up their capabilities and that can you know be something you can also do with a fine tooth comb and just add the things that you need. So dropping capabilities dropping all capabilities avoiding privilege escalation. It just in Shaw hashes as we've mentioned in your pod spec pull the image by Shaw hash that's something to think about not everybody agrees with that because it makes pipelines sometimes a little bit brittle but there are some strong advantages there as well. And then we'll also talk about sidecars and init containers for just a minute. Sidecars are containers that run inside the pod alongside your pod. And so for example you might have a the envoy proxy proxy running inside your pod and that proxy is going to need some features perhaps that that your regular application might not need. Another thing that we can think about is a knit containers and it containers run before the main containers in the pod. And so when you're launching a pod in a in a Kubernetes environment. An init container can do things that that need to be done with elevated permissions and the main container then when it takes over can just once again be a plain vanilla unprivileged user with no special permissions. So you can you can take the pieces of of operation that you need to perform that require some bump and privilege and you can segregate them out right you can put them in other places and that way your main application once again can just be a plain unprivileged component. So the last thing you'd want to do is say well when I start up for example imagine I need to change the ownership of a directory. And so I need to have you know the ability to change ownership of all directories all the time, which isn't really true right so we can add this temporal context to our thinking about security right. If you only need this privilege for a short period of time, then an init container right especially on initialization typically than an init container might be a really good option, because you can give that specific container the permissions that it needs and you can take those permissions away from your long running service which is the one that's going to be at risk So a quick look here then at locking down pods. I'll go ahead and bring back up our environment here again and just make sure there's nothing running. Okay good. Alright so we've got no pods running at the moment bounce over to our pod demo here so I'm going to go ahead and run a pod. I'm just going to use q kittle run, which in Kubernetes 118 which is what we're running creates a pod once again like it did in the storied old days. No more generators. And so we've got a basic pod up and running. And if I type ID, I can see that I'm root. So that's not optimal. Right. Also, let's say for example, let's take a look at our capabilities. So the Linux Linux kernel system calls for the root user have been broken up into these capabilities and these capabilities are identified by bits in this mask here. And so these are our effective capabilities. This is what we can do as this shell that we're running here. And this is what capabilities are going to be inherited by our children which is anything the shell creates. So we can see that it's not, you know, all f's right obviously there are some bits that have been turned off these get turned off by the container manager and since Docker. What while the world is sort of, you know, focusing on the open container initiative OCI as the standard Docker really created the foundation for all that stuff right and so when Docker containers are executed. So a lot of capabilities are removed. Clearly all four of these bits have been removed. Right. Three of these bits have been removed. Three of these bits have been removed and so on. And so there's a lot of things you can't do for example you don't have net admin capability inside a container. If you tried to, I don't think IP and tables is even installed in here but if you tried to you know dump IP tables or something like that. You wouldn't be allowed to do that you're not supposed to do those sorts of things right as a normal service running in a container, but a lot of people get used to having many of these capabilities and so removing those capabilities is something you know we're thinking about. Now, if you remove all those capabilities one of them is change owner. And so, for example, if we needed to change the, the ownership of let's do just a simple example of this www directory. This is just, you know, totally contrived but to give you an example, assume that the web route is owned by the root user so if we changed the owner of this or the user of this container to some unprivileged user lost all of our capabilities. We would also lose potentially the ability to write to the web route which maybe we need right because there's no right permissions for anyone other than root. So how could we address that well we could fix that kind of a problem with an init container. So let's take a quick look at a way that we can handle that. So I'm going to go ahead and run that same command again. So this will just sort of show that we're really, really just doing this exact same thing again right I'm cubicle running the busy box image, we're going to call it demo, but I'm telling the, the cubicle command don't actually do this and output it in the YAML and stuff it in this file. So in Kubernetes, we provide it with these declarative manifests with these YAML files to say this is what I'd like you to do. And so if I open up the pod YAML, it looks something like this. And so the creation time stamp will get generated for us automatically. We're not going to request any specific resources and we don't care about the DNS policy. Let's also make this guy never restart. And then we will get rid of any status that's going to be provided by the system, but I've got a bunch of stuff here that we're going to tackle on. So let's drop this in. Okay, so as we kind of talked about we want to run this pod as a non root user so we can specify at the pod level a security context. So at the pod level we're saying run as user 10,000, run as group 10,000 and run as file system group 10,000. So a lot of people are under the misconception that you can't run a Linux process as a user that's not an Etsy password or something like that. Linux just knows processes by ID, and it knows the owners of those processes by ID. So if you provide it with an ID, it'll give that ID to that process. So if you can't find any references to that process as special features or powers or permissions, it'll be just a plain vanilla unprivileged user. So there's nothing wrong with that, right. And then when we run this container, we're going to have it tail minus F dev null. And we've already got the reaps start policy up there so let's get rid of that. But I think that that should get it going right so this this container is going to run. And it's just going to sit there doing nothing but we will be able to, you know, shell into it and inspect it and see what's happening because busybox shell and stuff violating some of the other things that we were talking about, but this is a container designed for experimentation. Okay, so we've got our pod updated. Now let's go ahead and apply that guy. And then we're going to fire up a shell and we'll rerun those same commands. Oops, we got to delete the old guy. This will take a few seconds to let our finishers run. There we go. All right, and so let's try that again. So created. And now if we cube CTL exec into the pod demo. And start up a shell that we want to interact with. Now if we type ID, we're no longer root. And if we say, for example, grep for capabilities, see which capabilities we've got. We're down to zeros. Right. And so this this basically suggests that we're we're an unprivileged user and therefore we're going to have issues if we try to write to var, right, because the container image. And this is something that's an interesting thing to think about the container image is the root is the root file system bits, right for your container. And so when you build that image, that's got all of the bits for your root file system in it. And those bits include the owners of all of those files. And this is why everybody's used root for for so long or has, you know, often use root traditionally, because if you're going to provide a public image, what's the one user you know is always going to exist root and what's the one user you get by default when you docker run a container root. So a lot of production environments, you know, completely switch this around and say, no, there's a system user that or a service user that you need to use in production. And therefore, when you build your images, perhaps you can build those images with the right user in advance, right, you can set up that specific user the same ID, right, 10,000 or whatever it is that you're going to have in production in your image. And that makes, you know, this problem kind of go away to some degree. But it comes back as soon as you start using external volumes and as soon as those volumes get accessed by different users. So we'll do one last example here and then I'll kind of talk about the system side of it and policy where we can be impacted as developers, they can get through this in the last few bits that we've got of time here. So here is another version of this pod spec. So let me just remove the old one. And this is much like the old version, except that we now have at the pod level run as user 10,000. But we have this problem, right, we want to mount this volume from the host or from some storage engine doesn't really matter where it comes from, right, we want to mount this volume and we know that we're not going to have the right permissions. Yeah, because it gets provisioned with the wrong permissions or you know there's different permissions that need to be set and only the pod knows what they are, but there's lots of possibilities. And so our normal busy box container here is going to mount that volume under VAR WWW and is going to have issues because he won't be able to write that volume. So what we could do is we could set up an init container and the init container can run as a different user. Right, so this guy's overriding the run as user at the pod level and providing himself with the permissions that he needs, but we don't need all capabilities right. These privilege would say only give yourself the capability you need. So we need the change ownership capability. So we add that and we drop all the others. So this means we can't do anything except change ownership. And then the command that we're going to run in the init container is going to be change the owner to 10,000 for VAR WWW. Now this guy's going to mount WWW the same volume here change the owner. And then when the main container starts up, we'll have the volume that he needs. All right, so let's give that guy a try real quick. Shut down. There's another shell environment. Here we're getting used to create. So that guy's terminating. So let's do a cube CTL apply minus F of our new pod. The pod is still going. Let's go back here and check it. Give it another second. Okay, there we go. All right, so let's go ahead and re reapply this guy. Okay, and now if we take a look here, our pods up and running. And so if we cube CTL exact into the pod demo, you've got exactly what we were looking for. Right, we've got the ownership change for just the directory that we need to write to, but we don't have a long running container that has elevated privileges. Okay, so that's a, you know, kind of a quick once over and locking down pods pod policy again, you know, 45 minute webinar. So we've only got so much time to talk about, you know, these things. I'm just going to jump into pod policy real quick and mention that governance may apply constraints, right, they may force you to do some of these things. So you've got, you know, running privileged containers might be allowed or disallowed. You might have different kinds of volumes that you're allowed to use and not allowed to use. You might have, you know, users that are going to be required. And so on. So there's a lot of this could they can be enforced as policy and of course that's a great idea. If you want to create a better security posture inside your cluster, you want to make sure that accidentally we don't end up with you know, services that are violating these these kinds of policies. So got to demo for that. But, you know, a little bit out of time here. So last kind of concept I'll mention is network policy. We talked about this in the beginning. And so you can create minimal container images. And you can deploy them. But then you have this ability to limit ingress and egress to them with the various ports that we've, you know, kind of talked about breaking up. But you can also go further than that with network policy and the network policy is actually on the plate of the DevOps team deploying the service because they're the ones that know what that service needs to consume. And the features that it's exposing that will be consumed by third parties. And so these network policies are also useful tools for controlling ingress egress and also giving us access to services based on selectors. So we can identify individual components within Kubernetes by labels. And that's even finer grained and gives us additional capabilities also using namespaces and things like that. All right, so at the end of the day, you know, there's a lot of things to think about as a service developer, we have to, you know, consider the actual construction of the software. We have to consider the packaging of it. We have to consider the deployment of it. And then we've got policies to enforce many of these security concerns. And then we also have network policies that are actually back on the plate of the service developers because they're so fine grained that they're actually controlling ingress and egress for that particular pod that's being deployed. And so those are all the kinds of things that, you know, in general people should be thinking about. So thanks for attending. I will pass it back over to Kristi. Yeah, awesome. Thanks, Randy, for a great presentation. And let's just go ahead and do one question really quick here since we're just at the top of the hour. Um, Rosh asks how to deal with inflex unfixable excuse me vulnerabilities and the base image reported by image scanners. Hmm. Great question so image scanners come in lots of flavors and some of them are looking for CBEs and if you're using a library that has a vulnerability and that that vulnerability is is is not repaired. Which can happen in the world that we live in with all the open source and things. There's, there are there are things that you can do if you understand the vulnerability. But the tricky part is it's going to be case by case right it depends on what the actual problem is. Maybe there's an issue with, you know, with a particular attack pattern that you can control. By adding, you know, kind of a bit of code to your application or configuring a proxy in a certain way. Maybe there's, you know, for example, a way that you can lock off traffic to, you know, to to a specific port to control access to it. So there's there's a lot of different ways that you can, you know, solve these problems, but the range of problems is pretty broad. So it kind of depends. And at the end of the day, of course, you'd ultimately like to go back to that project and see them repair that that vulnerability. One of the things I can say I'm a involved with the Apache thrift project is that a lot of the vulnerabilities that come in are pretty darn subtle. You get some security researchers out there that are, you know, have to they set up a very, very complex, you know, environment within which to exploit some vulnerability. Right. And so if you can defeat that environment, right, if you can make it so that any one of those things, you know, and that that is required in order to exploit that vulnerability is not possible through network policy or through controlling, you know, the features of your pod, or by removing, you know, key tools that are required that that can mitigate the problem. Yeah, great question. Great. Well, that takes us to the top of the hour. Randy's email is here on the last slide. So if you have your further questions and didn't have time to answer it, feel free to connect with him via email. A reminder that the recording in the slides are going to be available on the CNCF webinars page later today that CNCF.io slash webinars. We look forward to seeing you at a feature CNCF webinar. Have a great day and stay safe. Thanks everyone.