 All right, next up is going to be a new is talking about Kubernetes security All right, how's everybody doing? Um, oh, I got somebody here react to me So today I'm gonna be talking about Kubernetes security best practices This is gonna be pretty high-level overview. I'm not gonna get too Deep on this stuff, but By way of introductions, I'm Ian I work at at Google I'm based in Tokyo, Japan, and I do a lot of you know, Kubernetes related stuff go and I have a background mostly in Python development So I heard earlier that some folks were kind of waiting for Python 4 before they migrate So just Just to let you know that I'm I'm waiting for Python 4 or 2. I'm a kind of a Python 2 person I kind of skip the odd numbered versions And if you're a Python 3 person that really likes that, you know So how many of you have actually touched Kubernetes have run kubectl? Okay. Yeah, I like pretty much everybody So I'm I want me to tell you that you know Kubernetes describing Kubernetes as a as a container orchestrator is kind of a misnomer. It's it's a lot of things I like to think of it as an operations framework So it like provides a lot of features that you can use to do operations make things like, you know more fluid That sort of thing, but you guys know all this already So today what I'm going to talk about mostly is I'm going to talk about several different best practices that you can do when you're deploying applications and So I'm going to use this little guestbook app that I wrote It's kind of a microservice the architecture. It's got a little message application That's it's basically just a g rpc service That you can use to save messages In a list and that saves it in a backing Redis store and then we've got an ng word Message service and that's basically just to check if like you're saying bad words in your messages You know we want to keep things g rated So there's that service and then we have like a little web front-end application so To look at it. It looks just like this, you know, everybody's seen one of these things you'd say like high-fossed M And you like run that and it writes it into the into the list yada yada yada Okay, but this one also has like, you know, it's Has like a little ng word check system. So like if you do Ho gay which is this is a word that I have registered in there I didn't actually registered bad words and because I wasn't going to do that during this talk But you know it pops up and says like hey, you know, you can't write that all right so So we have this little application. It works like this, you know, like this is the kind of like flow So the first thing that like you might do when you're thinking about like attacking a system like this That's running in Kubernetes is you might think hey, why don't I just like try to take over the API server? For the cluster like typically or a lot of times folks will have an API server that is exposed to the internet and you know normally you would have to authenticate with it using search and You know, it's not too big of a deal, but it's exposed to the internet, but You know one way that you could attack it is attack An application is to attack the the cluster itself So if you're familiar with Kubernetes, you'll know that that applications each pod can get a service account And that service account can be used to authenticate with the API And so one way that you might do that is to you know, if you've got control of say the front end It's to take that token and then like access the API And then use that you know to get secrets to then further attack the system So you might do something like don't judge. I have a cheat sheet here but So here what I'm gonna do is I'm gonna assume that you have like Access to the to run just basically arbitrary commands inside of the the front end you've broken in There was some sort of bug in the front end that you used to broken break in And so what you're gonna do here is we're just gonna run kube control exec Into the the front end and we're gonna cat out this token that's saved from the service account And so what you can do from that is Then I have this like long command here To actually this will run and call the Kubernetes API to get secrets and then parse out the Using that token and then parse out this the reddest password for our database And so we get out this database here and so we can do like telnet Let's actually break. Let's actually go into that pod first So if we do this Oh, yes, I need to say Been so I've got like a little Equals so we'll do telnet redis and 6379 And then we'll just like off with this password that we got up here Right, so then I can just be like hey You know give me out all the messages and then we'll say like zero negative one And that like you know gives us out our messages. You can imagine maybe like we have a hey Somebody else has been posting into my Yes book Yeah, that's a public IP address. So if you so if you want to play along at home, you know You can you can post stuff there But let's see That's the idea. I mean if you have like some Maybe some user data or some other like kind of stuff that would be exposed then that would probably be a bad thing for you so One way that you can kind of mitigate that is to let's go back to the slides here To you is to use our back so our back is a is you know role-based access control for kubernetes And this is allows you to basically create Roles that you can assign to to users and then that role gets assigned some permissions that it can do so like in Usually that's like some sort of verb to a type of object. So like get secrets Is one that you probably don't you want to give away sparingly? this particular cluster is Already set up to use our back So I'm kind of emulating the fact that That I have access to everything but if you have if you actually were not using our back then By default your service accounts just kind of get access to the full kubernetes API So here I'm just gonna like delete the the web binding that I or the bind cluster role binding that I had To for that service account and now if you if I go back and try to do this again you know I Probably don't need to do that but If I try to do this guy again, I should get back that I don't actually get the password back and if we look at Just this part. I think let's just get this part We can actually see the result that comes back That it's like hey, you know, you can't access the secrets So all backs of thing you should use it That's pretty much, you know if you're using a modern a recent version of kubernetes. It's pretty much Always or it's pretty much enabled by default Let's see like the next thing is to actually Put a firewall in front of the API server so if you want to you can like kind of Limit the the folks that can actually access the API server by IP addresses or by networks So if you're somebody that's using it on, you know, GKE or on at Google Then you can create your cluster and set the the authorized networks that can access the API server Or if you're like running this on-prem or whatever then you can kind of Use whatever method that you want to firewall that But that's another idea because that will make it so that folks can't actually access the API server externally Of course, they can still like, you know tunnel through the the front end if they have access to it Another idea is to use network policy So network policy allows you to at the network level say like This pod can access this pod or this pod can't access this pod and that sort of thing and you can set that up by Pod labels so here we have a pod selector That's it that matches the redis server and then we'll say that like That only guest book applications or the guest book application can have access to Ingress into the redis cluster. So like if I go back here and I say like Apply network policy Like that and then I say bashed and I try to do that tell that thing again 379 Yeah, like I just can't connect to it anymore No, tell that you just don't like me. Okay. All right, so that's Actually This type of thing getting access to the API server So once you've like kind of blocked off access to the API server you've locked that down Another thing that you might try to do if you were an attacker would be to kind of break out of the container And then try to get access to the host and take over the host Maybe attack other containers that are running on that host maybe attack the cooblet that's running on that particular server And so those type of things you can kind of try to mitigate a little bit better Using a few features So if you're familiar with things like sepcom and app armor and se linux Those are Things that you can use to kind of help mitigate this So your app running just by itself is fine, but like when you put it into like a container it's it helps it to So that it's it's harder to break out of it, but it's it's not always It's a little bit porous in the sense that like if the if there's a If there's a kernel exploit or some sort of problem that you could have within the kernel That would that could potentially allow you to get out And so what you do after you've put your application into a container or you've caged it up You put it on a lot box and then you put it in another lot box So these So using these really help it so that once that even if you're able to like kind of break out of the container You still can't really do very much that's useful and so Sef comp does that by allowing you to you know create a filter a whitelist filter on the the syscalls that you can do app armor and se linux are both the What is it the security frameworks for Linux So they can stop you from like doing things like mounting things extra stuff inside the container or like writing a proc or doing things like that So Kind of by default Kubernetes doesn't really a lot like have you set this or doesn't set a second profile for you so it's Pretty recommended at least by me to go ahead and put in that because otherwise You'll have things like Where we are going here. Let's go back into our little guy here and we can say like okay. Well, we're gonna just unshare The user container the users name space. Oh wow I can just do that You know so like if there's a put that doesn't really necessarily by default like or as it is Let me do very much. That's useful But you can imagine if like there was some sort of exploit that through Unsharing a namespace that allowed me to break out of the container that this would be actually a bad thing for me to be doing Like most applications don't need to do this And so if you just say like well, you can't do that at all Then that will help, you know mitigate those type of problems So we can go in here Let's see And We'll say in the web app deploy Let's go ahead and add that in this is really So now I updated that so that it's so that the web This web server now has a set com profile like the default set com profile running on it so I could say like control exec like so in here I can try like Unshare again So like by default like so if I use the default set com profile for talker that is not white listed So I can't actually do that if I'm actually applying the the default set com profiles so when you want so if you're Most applications will like will run just fine with these And so you'll want to just go ahead and do that if you're If you're running an app application like also the app armor depending on the which You know provider you're using or whatever like However, your your cluster is set up. You might have app armor installed or you might have se linux installed And so you can kind of go ahead and use the the default the doctor default or the runtime default Profiles for app armor and se linux as well So here the app armor one is just the default and then you can set your own se linux options within the security context in a pod So those are things that are like pretty Pretty nice to just go ahead and do Another thing is that you might want to restrict the permissions that the cooblet has so If you're say trying To once the person has got out of the container and then like gets access to the cooblet and the cooblets tokens and certificates Then they can basically do whatever the cooblet can do and so if you restrict what the cooblet can actually do then that can Help you kind of mitigate problems so There we go. So one thing is to enable like the RBAC mode for for the cooblet and then Set node restrictions. So that means that Each cooblet only has access to things like secrets that are running for pods that are running on that node And so it won't have access to secrets that are available on other nodes Or for pods that are not running on that node also Setting the certificates to rotate is also a helpful so that like if somebody extracts the certificate for that cooblet And then tries to use it against the API server that you know It will eventually get rotated or it'll get rotated relatively quickly so that they won't be once you've kicked out the attacker from your From your cluster, they won't be able to just still use that That certificate in perpetuity or in perpetuity so another issue that you might kind of run into is Listening for listening to traffic within the cluster. So like once they have access to the web front end The They might be able to like say listen to the traffic between the Our message service and red is and like you know pick up the password or pick up like some data That's being transferred and use that to attack or just extract the data So that sort of thing is also Not particularly good and you can also think of the the host Perhaps making requests to other services within the cluster. So like you know Creating essentially fraudulent requests to other to other services So this is something that like you I mean even if you use to your PC, you're not really That's not really gonna stop somebody from Sending just your PC requests to to the server They're a little bit harder to do than just sending a curl requests But there's nothing stopping an attacker really from doing that So that's where like things like service meshes come in so so Istio is a service mesh that's You can run on top of Kubernetes and What it does is it includes like an Envoy proxy as a sidecar for each of your containers and then The Envoy proxies for each of the container each of those sidecars Essentially are the ones that talk to each other and all of the applications just talk to the sidecar So you go from the application to the sidecar proxy Then that sidecar proxy goes to the other pod sidecar proxy and then up into the application and Everything in between the sidecar proxy is authenticated and encrypted so The way that it looks is it looks kind of something like this where you have this little sidecar proxy running on each of your applications And then the communication in between them is authenticated and encrypted So you can't really from the web front end like listen in between the What the the message service and the Redis service are doing and then You also can't as easily just send Requests to a particular service because you're not out there indicated with that service so As part of this you can create like policies for which services can talk to which services And so maybe our front end can send messages to the message service But if you had another service out here that message used And the web front end didn't access directly You wouldn't be able to send from the web front end Messages to the other one because you know you would run into the policy would forbid that And so that makes it a lot easier so that like people can't Can't really do very much once they're inside the cluster so The other another issue is like that once you've taken taken control of the web front end the The attacker could then potentially gain access to the sidecar the actual envoy proxy running the sidecar. Okay, yeah, and The So this also does like the cooblet and does rolling surface configured updates So like even if you've extracted that certificate you can't really continue to use that sort of kid certificate forever And that's it really So So probably I have five minutes left, but maybe not four minutes for questions You sort of skimmed this over this fast with the RBAC stuff at the start once you have access to a specific Thing like the web front end right can you not from there? Just then talk directly to Kubernetes API server and for instance kubectl exec on another machine. How do the No, so like the the reason why I can exec into it is like because I'm a cluster administrator myself and So I'm I'm just assuming that like I can get into the front end like in this case I'm exacting into it But like that wouldn't be like how an attacker is actually getting into it They would be doing it through like however like through like some sort of Like tunnel that they One of our clusters at the moment has a Jenkins thing inside it Right, then have permissions to start up and destroy the containers Right, which then means that any of those pods have permissions to do basically everything Not necessarily that something needs to be set up via So your Jenkins server for sure like we'd have permissions to do a lot But you're the pods that Jenkins creates doesn't necessarily don't necessarily have to be able to do everything You can set up the rules in such a way that they only are able to do what they need to do So one of the things Docker really got right was having a good default second profile Do I need to explicitly enable this in Kubernetes? Can you say that again as I want things to go right with having a good second deep right profile Do I need to explicitly enable in Kubernetes to get a default second profile? so If you kind of install it by default right now like yes And I believe that's mostly because it's like it's still like kind of an alpha thing The I imagine that that will become like kind of a default in the future And it also depends like it could depend on the vendor or like whoever's like setting up setting it up for you like so I Believe right now like GKE doesn't do that But in the future it will be ill you'll probably have default set default security policies that Enable those default second profiles and defaults Like app armor profiles and stuff like that Who are the also like with the CRI and all these new container runtimes? Well, they also have to support the same format of second profile, right? They would they would have to support Setting the set comp profiles as part of like their execution framework or however they they start up the containers But yes, like the way that that it was said there say like runtime defaults is so that you Basically have a way of setting the default that that runtime whatever runtime gives you There's also like you can also put it in if you know it's Docker. You can just say Docker slash default Yes, this is just about on what he was asking a little bit How many of the recommendations that you suggested are actually are implemented by a managed communities like GKE like especially like the cubelet we don't We're not really GKE users can't really start Adding flags start-up flags or not to my knowledge and right we kind of have like everything Like the API server and the schedule are kind of hidden away. So there's no really way we could actually install like some Yeah security. Yeah, that's Things are kind of like getting better as like things are going on but like the so things like if you create a server that's Like on GKE for instance, like if you create a 1.8 cluster like by default our back is there and like everything works fine so if you if you create a pod like with a that that pods service account like really can't do anything But So as things go on like more and more things will become default but Yeah, you don't really have at the moment like a lot of flexibility in terms of like turning those flags on and off Unless that particular vendor supports it Network policies I think are an alpha feature and alpha clusters kind of get destroyed after a month, right, right? Network policy I Believe this actually not in alpha anymore. It's in beta fight 2% traffic loss that I'm seeing constantly when I enable that Yeah, it it that depends on the network implementation really like I mean you're probably using Calico or something like that, right? Yeah Network policy is it's actually implemented by Calico on GKE. So like yeah, like and I believe that it's it's probably I believe it's using IP tables underneath the like and setting those rules so that you can access but Yeah, hopefully that'll improve in the future. I don't I don't know offhand like Yeah So I believe that'll get better But yeah, there is a little bit of network cost for things like network policy All right, I think that's all the time we have it is yeah