 So thanks everyone for coming. I'm super excited to be here. My name is Alex Brand. I am going to be talking about certificates and how we use certificates in Kubernetes and all you need to know about certificates in Kubernetes. The only caveat is that I thought I would have enough time to talk about everything certificates but half an hour is really not enough so that all there should really have a star and what we're going to be talking about just a smaller scope we're specifically going to be talking about how to use certificates to produce a secure Kubernetes cluster. So just to give you a bit of background about myself and about Apprenda we've been working with Kubernetes since early 2016 when we got involved with the community we actually created the SIG Windows so the special interest group that's leading the Windows implementation of Kubernetes so you can run Windows containers on Windows and then I personally have been involved with maintaining our open source project called the Kismatic Enterprise Toolkit which is a set of tools that allow you to manage, operate and bootstrap Kubernetes clusters on-prem or on the cloud. So our initial implementation of Kismatic was released in November 2016 when RBAC wasn't quite there yet so our security model we were secure from the get-go but our security model was very very simplistic so every single component had the same identity all the users were using the same the same users so after RBAC came out in April 2017 we actually decided to adopt it because we thought obviously it's just a better model so we adopted that model and that meant we had to just revamp how we generated certificates in Kismatic. So this talk today is going to be informed on those experiences and what we had to do to revamp the generation process and how to produce a secure cluster. So this is the agenda as you can see there's a bunch of stuff in there hopefully we have enough time to go through it but basically we'll talk about a couple things that you need in Kubernetes and that are related to certificates to have and to run a secure cluster. So I know we're in a security track but I just thought I would do just a quick certificates refresher just in case yeah so the certificates basically allow parties in a conversation to authenticate each other so if you think about a client and a server relationship the client can authenticate the server and just make sure that it's talking to who it thinks it's talking to and the server can also authenticate the client using certificates so this is all this all depends on the on so one way of doing this is using a third party that both the server and the client both trust so if they don't know each other they don't have a previous relationship they can actually trust a third party that issues certificates to be able to trust each other so that is where the certificate authority comes into play so the certificate authority is actually going to issue certificates to all these parties and given that they trust the certificate authority they can then trust each other so if you think about when you access your bank's website for example you want to make sure that you're talking to you're actually talking to your bank servers and not some random server on the internet so what you do is you ask your bank for a certificate and then you validate that certificate and given that that certificate has been signed by an authority by a certificate authority that you trust you basically trust that you're talking to the the the bank's server so that takes us to kubernetes why do i need certificates in kubernetes why do i even care about certificates in kubernetes so how many of you are running a kubernetes cluster or are in an operations team very nice that's awesome to see um how many of you are familiar with how certificates are used in kubernetes and how exactly awesome very cool so as we all know kubernetes is a distributed system this is a very simplistic diagram of some of the interactions between all the core components in a cluster so we have the api server in the middle the api server is the only component that talks to xed and then the api server has a bunch of clients including the scheduler the qproxy the controller manager and the kubernetes are basically the main the core components of your cluster and given that all these components are running on different machines that are connected through a network you just want to make sure that all these interactions are secure and that all these interactions are basically all the components can authenticate each other so i'm going to walk you through how we can build a secure cluster using certificates so the first thing we actually need is a cluster certificate authority so the cluster certificate authority is going to be the the trusted route throughout the entire cluster so all the certificates that are used in the cluster are going to be signed by the cluster c a and this is what's going to enable all the components to be able to authenticate each other so all the components are going to trust that c a and then whenever they they're presented with a certificate they actually can trust it because they've been signed by by the c a okay so now that we have the cluster c a we can actually start securing some of our interactions so the first the most important interaction is the api server as we all know the api server is the main entry point to the entire cluster so we want to make sure that we are exposing this endpoint or the server over htps and to do that we need a serving certificate and the corresponding key and as i mentioned before the certificate is going to be signed by the cluster c a which is going to allow all the components that are using or that are communicating with the api server to be able to authenticate it so there is an important gotcha here with with certificates for the master and that is that when you're setting up multiple api servers when you want to have an ha cluster you actually want to make sure that the load balancers ip address and dns name is part of that certificate because otherwise whenever a client tries to talk to an api server through your load balancer the client is actually going to try to validate that certificate and it's going to complain it's going to say you know what the common name that's on the certificate is actually not what i'm trying to talk to so it's going to complain so just whenever you're setting up multiple masters just make sure that the certificates have the load balancers dns name and ip address this is usually taken care of by your favorite installation tool they usually have a configuration parameter to just set extra extra extra names in the master server so there's actually another api which is really an internal api is just an internal private api as of now and that is the keelbit's api which you also want to expose over htps and then again to do this we need a serving certificate and the corresponding key and this api is mainly consumed by the api server when it needs to get logs or when it needs to get metrics from a specific container and also when it when it needs to exec into a pod so again this cert is signed by the cluster c a and that's how the api server can actually authenticate the keelbit and and make sure that it's talking to the the keelbit similar to the api server the keelbit's api is also protected by authentication and authorization so this means that clients of this api also have to present client certificates so whenever the api server has to talk to a keelbit it actually needs a client certificate and that's why the api server actually has a client cert um for for talking to the keelbit so as you know as you'll notice i have little links at the bottom so if you want to look into it more there's some more docs on it most of the slides have them cool so and so we talked about how do how a client can authenticate a server um so the next piece is how can a server actually authenticate a client so that's where the client cert authentication strategy comes into play so the api server and the keelbit actually can authenticate clients using this strategy so this is mainly used by kubernetes components um although you could use it for user author although there are some important caveats um but the way this works is that any request that presents the client certificate that has been signed by the cluster c a is going to be considered authenticated part of that authentication process is to obtain the user information and the group membership of that user and that information is actually obtained from the certificate itself so the user is obtained from the common name field of the certificate and the groups are obtained from the the organization field of the certificate so as i mentioned before this authentication strategy is mainly used for authenticating kubernetes core components so each one of them is going to have its own client certificate because each one of them is going to have different access levels to the cluster so you want to have each one of these components to have their own identity and the way to do that is to have certificates with different common names so for example the scheduler has its own certificate with the common name set to system kubes scheduler you'll notice that the kubelet cert is actually special and the main difference there other than it belongs to an organization is that the hostname where the kubelet is running is actually part of the certificate so that is very important because you want to make sure that each kubelet on your cluster has its own identity why is that important so so that so the reason for this is that it allows you to enable a couple features in kubernetes which are called the node authorizer and the node restriction admission plugin so what these do is that they actually limit access to the api server from the kubelets perspective so without these features enabled the kubelet is actually capable of seeing and writing and modifying every single resource on the api so you ideally you don't want this you want to follow the principle of least privilege so instead what you want to do is by having each kubelet have its own identity you can enable this feature that will actually limit the access to the api to resources that are actually related to that node or to that kubelet so for example instead of the kubelet being able to modify every single node resource in the cluster it can only modify its own resource in the cluster similar with secrets instead of being able to access every single secret in the cluster it can only access secrets that belong to pods that are bound to that node so we just went through all the certificates that we need and how do we actually produce these certificates so we can either manually create them so an admin can go ahead and produce all the certificates issue all the certificates that we need or there's actually an api in kubernetes that allows us to generate certificates so as of recently kubernetes offers an api that allows you to request certificates through just you know using the the regular kubernetes apis so the way this works is that a client creates a csr or a certificate sign in request and shoots is shoots it off to the api server once that csr is created it actually is going to remain in a pending state until someone manually approves it so the idea here is that you don't want to actually randomly approve and generate certificates because these certificates are going to be signed by the cluster ca so as we talked about before every single component in the cluster trusts that ca which means that is going to trust any certificate generated through this api so the csr actually has to be approved for the certificate to be issued so once that csr is approved the certificate is issued and the user can actually download it so i actually want to show you how this works so yeah let's dive into a quick demo and i have a kubernetes cluster running here in my machine it's a 184 cluster i think yeah 184 and the first thing i need to do is to generate a csr so i've actually generated that previously just in in the interest of time and you can see here that i have my certificate request and what i have to do is to actually send this over to the api server so i actually have to wrap this in a csr resource in the kubernetes api so the kubernetes api has a csr resource and you know similar to apod or any other resource in kubernetes you add some metadata so you can call this csr uh and set the name to kubecon and then i have my spec and the most important piece here is the request field there and that's where the csr is going to be included in this resource so that the api server can actually store it so i'm going to go ahead and copy this yeah yeah i'll take questions in the end and i can just post this to the api server and you'll notice here i'm using cat but it'll just read the csr and base 64 encoded and then just include it in the resource and then there we go so i've just created my csr and i can actually get these this resource similar to how i can get any other resource in the api so i can go and do kubectl get csr and you'll notice that it's in a pending state the other thing to keep to to keep in mind and to and to learn from this is that the requester of the csr is actually stored as part of the resource and this is important for what we'll talk about next which is how the kubectl can use this api through request certificates so i've created my csr and again i have to approve the csr so i'm wearing an admin hat right now and i can go and approve the csr so the csr has been approved and if i get the csr again the condition has changed to approved and the certificate has been issued so as a user i can go ahead and download the csr so if i actually describe or get this resource in yaml you'll notice that the certificate is now part of the resource so i can actually extract this the certificate using a jason path so that's in the status field and the certificate field yep i gotta get my kubecon cert csr and that is base 64 encoded so i can actually base 64 decode and there's my certificate so i can actually start using the certificate so you can imagine building workloads or building controllers that live on top of this api and that's one of the interesting things of kubernetes that you can actually use all these apis to build stuff on top so that's the quick demo of how this process works and now we can talk about a specific use of this api and that's how the kubelet can actually use this api to bootstrap itself and to rotate certificates when when they're it starts our nearing expiration so as we as we talked about before the kubelet needs a client cert to talk to the api server and it also needs a serving certificate for its own api so instead of having the admin having to generate all the each of these certificates one for each kubelet and all the all the serving certs the kubelet is actually capable of requesting certificates when it starts up so this is again this is built on top of the certificates api and also the bootstrap token authenticator which i won't go too much into but it's basically a different authenticator that that authenticates clients based on a short token and these tokens are basically used during the cluster bootstrapping process so this is how this process works so let's assume that we're bootstrapping a cluster and there's an api server and the controller manager it's already they're already running so the first worker node is starting up and the kubelet is registering with the api server and it's going to realize that it doesn't have a client cert so the first thing it's going to do is it's going to create that csr resource that we just learned about and it's going to use the bootstrap token that is used for bootstrapping clusters the importance about the bootstrap token is that's is that it's going to tie the request back to a kubelet so as we saw before in the demo part of the csr resource is who requested the csr so in this scenario the the requester of the csr is going to be a kubelet which is going to be important in in this flow so once that csr is created the api server is actually going to issue a watch event to one of the controllers that that's watching the csrs and there's a controller that's going to check whether the csr was requested by a kubelet given that in this case it was the it's going to automatically approve the csr so as we saw before csrs have to be approved so there's a controller that automatically approved csrs that come from kubelets and then there's another controller that is watching csrs that once there's a csr that's been approved it's actually going to go and sign the csr to issue the certificate and it's going to go ahead and upload it to the api server once the certificate is up in the api the kubelet can actually go ahead and download that certificate so it can go and as we saw in the demo it can go and download the the certificate using the api and it's going to after after that it's going to be able to use it for further api access so just to summarize the most important steps the first thing is that the kubelet creates a csr using that bootstrap token then there's a special controller called the csr approving controller that is going to approve the csr automatically because the request came from a kubelet and then the csr signer controller is going to go ahead and sign the csr which will then allow the kubelet to download the certificate and start using it to talk to the api server so that's how the kubelet gets the first certificate or that's how the kubelet bootstrap bootstraps its own certificate but what about rotation so you don't want to get paged at two o'clock in the morning if when your cluster dies because all your kubelets fail to talk to the api server because the certificate expired so instead of you know getting paged the as of kubernetes api as of kubernetes 1.8 the kubelets can actually request a new client certificate when the one they're using is nearing their expiration date so similarly the kubelet can also rotate the serving certificate and that's actually in alpha right now there's a couple of kinks to work out but yeah if you want to share if you want to follow the progress on that issue I've linked the issues down there so that's basically how we can create a secure kubernetes cluster deployment that's how certificates are used at the kubernetes component layer but as you might imagine certificates are also used for different things on the cluster once you have your your running cluster so one of those those things which is actually which is actually a missing something that's missing is the ability to revoke certificates using certificate revocation lists so just something to keep in mind if you're using certs for user auth you currently cannot revoke certificates so they're always going to be they're always going to be considered valid until they actually expire so if you want to learn more about this or if you want to join the discussion or if you need if you think you need this feature for whatever reason definitely check out this issue on on github so today we talked about certificates at the kubernetes component level so how the components themselves use certificates to authenticate each other but what about workloads so if you want to expose workloads over tls you can actually do so using ingress so ingress has the ability to expose services that use tls and the way to do this is you first have to define a secret that includes a certificate and a private key and then you have to reference that secret in your ingress resource so what's going to happen there is that once your ingress controller realizes that there's a new ingress resource defined it's going to go and use that secret to configure that tls listener so if you if you don't want to do this manually there's actually an interesting project called cube lego which can actually automatically generate all these certificates using the let's encrypt api so if you want to check that out that's pretty cool and then similarly at the workload layer there's actually a working group that is working towards allowing containers to prove their identity outside of a kubernetes cluster so inside the cluster each workload has a service account which they can use to talk to stuff that's running on the cluster but what about accessing external services so there's actually a working group called the kubernetes container identity working group which is working towards this idea of being able to use some mechanism that is going to allow containers running on the cluster to be able to access external systems that are running off the cluster the other potential use case although there's a bit of overlap with it something like istio is potentially kubernetes could do service to service you know mutual tls everything out of the box so if you want to learn more about this these are the notes of the working group and they have weekly meetings so i definitely encourage you to to check those out so just to quickly summarize what we talked about today we went through how we can produce a secure kubernetes cluster and how certificates are used to create a secure cluster so certificates are key to the functioning of a secure cluster and the main reason for this is that kubernetes again is a distributed system and you want to make sure that all the components are talking to each other in a secure manner and that's exactly where certificates come into play so certificates allow all of the components to authenticate each other and to establish mutual tls and that is what's going to produce the the secure cluster we also talked about the api that kubernetes offers for generating and requesting certificates so we we saw a specific use case of this which is the kubelet and how the kubelet is capable of requesting and generating and rotating those certificates and again this is an api that can be consumed by anything other other than just the just a kubelet so this is just a quick table that gives you an idea of all the certificates that are in use in a kubernetes cluster obviously at the at the component layer so i mean we talked about all the the api server the controller manager scheduler kubelet kubet proxy and i hope this gives you an idea of how important certificates are and how careful you have to be with certificates so again you don't want to get paged at 2 a.m. in the morning when one of these certificates just expires and your entire cluster is down so i hope that gave you an idea again of how certificates work how they're used in kubernetes and if there's any questions i'm happy i'll be around and i'm happy to take them no so that's a great question so the actual component that signs the the certificates is the controller manager so the controller manager does need access to to the private key of the ca yes it's fairly sensitive so it really depends on how you're setting up your cluster so some installation tools run the controller manager as a workload on the cluster so you could use a secret for that or something like that yeah yes so the the kubelet api yeah so the kubelet needs a client cert to talk to the api server but then if you want to talk to the kubelet's api you also need a client cert yeah so that's a great question so all the issues that i've seen around the kubelet api the kubelet api is really an internal api right now it's undocumented so it's not really maintain or yeah an api that's going to remain stable so the the recommendation is to not really depend on it but i think there are people that are that are using it yeah so when you you're on your bootstrapping your see it when you're sending your first csr i think yeah so you're using a bootstrap token at that point but i don't think it'd be encrypted that's a great question i'm not sure i can definitely look it up and come back to that yeah so the the majority of clusters that at least that i've worked with have a single ca so there is one ca so the ca is a single ca in the entire cluster um the the yeah i guess so you're talking about signing the certificates when they're requested yeah in tech technically the controller manager but yes the kubernetes api and the yeah exactly yeah makes sense yeah the ca continues to be the same ca so there's one ca for the whole cluster yeah a way to automate the generation of keys uh the private keys so that really uh so for example in the community there's kubei the m that takes care of all of this um so if you're thinking of uh creating a cluster kubei the m is a good way kismatic our tool also gives you that capability and it gives you a couple other things so uh we take care of that we take care of all of that for you so you don't have to do that so yeah yeah think so i'm not super familiar with that right now but i believe there is an effort to integrate with external key management software um i believe there is some integration with vault there's there actually is um you can actually use vault to store secrets today i'm not super familiar with that so i yeah there's the next doc there we go yeah for the masters so the master certs are actually not generated using the certificate api so you actually have to create them beforehand right yeah so either manually or the installation tool actually takes care of that for you so for example kismatic will take care of that or kubei the m will take care of that and then it you also right yeah you set the sound for the lb exactly yes sir uh that is a good question uh i don't know yeah i'm not sure yeah yeah yeah i would expect them to be after after they're created yeah i would expect them to be cleaned cleaned up yep yes sir so it really depends so we we actually issue in kismatic our ca is i believe two or three years and the certificates that are produced by that ca are one year no so as of right now the only one that gets rotated automatically is the the kubei one yeah yeah every whenever they they're close to their expiration they the kubei is going to rotate them yes sir not necessarily so you can actually use your own ca you can provide your own ca to kubernetes and your installation tool so for example in kismatic you can actually bring your own ca and we will use that ca to produce all the the certificates that are that are needed for the cluster yes yeah so yeah there's some implications there definitely yeah to upgrade the ca that is a great question i believe from what i've seen you can only use one ca so you would incur downtime there i think yeah he i believe that should work yeah that sounds like a potential worker around there yep so he was suggesting adding the ca to the machine's trusted root list so that the the the binary just trusted because the machine trusted so in the multiple master scenario each master is going to have its own certificate and then the consideration there is to just make sure that the load balancers ip and dns name are part of that cert so that your clients you know can validate that certificate so you can bring your own ca i'm not sure if you're talking about like integrating with another system yeah i'm not sure i don't think so i think you actually have to provide like a file based ca so yeah at this point you have to do that yeah cool thank you so much and enjoy