 Hello, everyone. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Annie Talastro, and I'm a CNCF Ambassador, and I will be your host tonight. So every week, we're being a newsletter presenters to showcase how to work with Cloud Native Technologies. They will build things, they will break things, and they will answer all of your questions. So you can join us every Wednesday to watch live. So this week, we have Lynn and Richard here with us to talk about real-world trust management, liquidity and search manager. As always, this is an official live stream of the CNCF, and as such, it is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. So basically, please be respectful of all of your fellow participants as well as presenters. But with that done, I'll hand it over to Lynn and Richard to kick off today's presentation. Thank you. Yeah, I'm Flynn. I'm a technical evangelist for Lincardee. I'm work with Boyant. With me is Richard Wall, Software Engineer for JetStack. Thanks for the introduction, Annie. Glad to be here. We are going to talk about real-world trust management with Lincardee and search manager. Sorry, Lincardee and search manager. We will also be talking about a thing called trust manager in there as well. That's related to search manager also. You can reach both of us, well, you can reach me on the Lincardee Slack and the CNCF Slack as Flynn, you can reach Richard as Richard Wall on the CNCF Slack. So if you have questions after this, that's the best way to reach us. What we're going to do today is we will talk a bit about Lincardee and search manager, a bit about certificates themselves and how identity works in Lincardee, how the search manager helps with that, and then we will dive into a demo. The agenda sounds like a bunch of things for slides. It will go pretty quickly. We're mostly going to be dealing with the demo here. But there's some background that's important as we talk about, especially when we're talking about trust and certificates and all this stuff. So Lincardee, the very, very quick demo. It's the only CNCF graduated service mesh. Its purpose in life is to bring security, reliability, and observability to every cloud-native application and every cloud-native developer, and it is mostly brought to you by Boyd's, where I work, full disclosure. Cert Manager is a CNCF incubating project whose purpose in life is to automate provisioning and managing certificates in Kubernetes. It's brought to you by Jetstack, which what was it Richard a couple of years ago that Jetstack ended up becoming a part of Venify? Yeah. It was the beginning of 2020. That's right. Yeah. Right. So as we go through this, you may hear things about Jetstack or about Venify, don't be confused, same company. Sort of. Anyway. So why would you use both of these tools? If you look back at what I just said about Lincardee's purpose in life, security features prominently. Security and Lincardee relies on MTLS. MTLS in turn relies on certificates. You can manage all these certificates by hand. You really do not want to. The reason you don't want to is that if you get it wrong, it will cause downtime and then everybody will yell at you. So you really don't want to do this by hand. Instead, you want to use a tool like Cert Manager. Really quickly, let's talk a bit about certificates themselves. They tend to be a thing that is relatively poorly understood everywhere, actually not just in the cloud native world. Certificates rely on public key cryptography. They form the basis of trust in Lincardee. Public key crypto is pretty cool because it allows you to safely communicate and do authentication, irrespective of whether the entity is communicating and authenticating, no much about each other. MTLS relies on all this stuff. Lincardee authentication and identity rely on MTLS. I talked a lot more about this part of certificates in the earlier cloud native live on certificate management with Lincardee. If we could get that YouTube link posted into the chat, that would be lovely if anybody wants to go back and get the deeper dive. I'm deliberately doing the less deep dive on it right now. To make all of this stuff work, though, both in the sense of whenever you're dealing with public key cryptography, but also when you're talking specifically about certificates, there's a private key and a public key. The private key must be kept private. They form the basis of identity in Lincardee. The public keys really must be shared with everybody so that they can be used to verify the identity that is formed by the private key. This creates a problem because keeping track of all these keys is difficult. And to deal with that problem, this certificates thing was invented. More correctly, we're talking about X.509 certificates. They are data structure that take a subject, which is really the name of the entity being identified. And they bind it up with a public and private key pair. Only the public key, oh, there's the YouTube leak. Great, if anybody wants that. Only public information is contained in the certificate. They are always completely safe to share around. There's absolutely nothing in them that's private. This is actually a critical point because you're supposed to share these things so that they can be used for identity verification. They are signed. Here's the signature block for this one. The point of that is that the signature prevents them from being spoofed so you can actually trust them. And another thing that we talk about more in depth than the other one, but we need to touch on here is rotation. There's an axiom in security that the longer you use, a given key, the more valuable it becomes to an attacker, which means that you don't actually want to keep using the same key for too terribly long. Rotation is the name given to the process of generating a new key pair and swapping it in for a certificate so that you don't have to rely on a key for a very long time. All right, we're gonna dive into how certificates get used in LinkerD so that we can talk about how we can get cert manager to help out with that. In LinkerD, keep hitting my mic, excuse me, that probably is very loud. In LinkerD, we start with workloads in a cluster. Every workload has its very own workload certificate which identifies that workload within this entire thing. LinkerD manages all of the workload certificates for you. You should never ever have to think about that. The workload certificates are in turn signed by a thing called the identity issuer certificate in LinkerD. And then one layer up from that, we have LinkerD's trust anchor certificate, which signs the issuer certificate. The reason that we have both of these, both of these, both the issuer certificate and the trust anchor certificate, there are actually a couple of major reasons. The first one is that it makes it very much easier to just swap out issuer certs fairly frequently without having to mess with everything. It also limits your exposure correspondingly if one of these certificates gets compromised, having the two layers there makes it a bit operationally simpler to deal with that. The other thing going on here is that it tends to make it easier to insulate these identities from the network topology. So for example, the reason that you would have two issuer certificates is if they're in two separate clusters entirely, this works fine with LinkerD. This is the basis of LinkerD's multi-cluster world. The requirement for multi-cluster is basically the trust anchor has to be the same across your entire network topology. And then everything else will just work. Now, normally when we are doing demos, we will install LinkerD with the CLI. And if we install LinkerD with the CLI, we will get a self-signed trust anchor. We will also get the characteristic that all of the certificates shown in gray are things that you have to manage by hand. We talked earlier about the fact that you don't really wanna manage certificates by hand. Another thing I'll throw out is that using a self-signed certificate for the trust anchor is really not ideal for LinkerD in production. So better not to do that. What we're gonna do today is focus on a case where we're using cert manager instead where we've got the workload certs and the issuer cert in the trust anchor, but we use cert manager to provide an extra layer for the certifying authority route. The CA route then signs the trust anchor and maybe most importantly, all of the certs in gray are now going to be managed by cert manager instead of managing them by hand. One last point, talking about how often should you rotate? The time that you choose to rotate things is always a bit of a balance between basically protection against a key being compromised and operational complexity. The point, the biggest thing to remember here is that you're probably not going to know if a certificate is compromised. So once it's compromised, it will stay compromised and you probably won't know about it until the next time you rotate the certificate. So if the time to the next rotation is very, very long, that gives your evil doer longer to do evil things. If it's very short, it mitigates that risk there. It mitigates the damage they can do. The countervailing pressure here is that if you're spending all of your time rotating certificates, you are not spending your time actually accomplishing things that you want to do. And so there's a balance to be struck there. I cannot tell you what that balance is. I can tell you that you need to evaluate your risk tolerance. You need to evaluate how much operational complexity you're willing to tolerate and then come up with something that works out okay for you. I will point out that by default, for example, when you use the LinkerD CLI to install LinkerD, that will create a self-signed trust anchor that's good for a decade. That's too long in my opinion, I would not recommend doing that. The other thing that's very important here, and another reason why I would not recommend Xperia Times out to a decade, is that you really have to actually do the rotation so that you know how to do it as opposed to just going, oh, we wrote it down, everything will be great. Because what will happen if you don't actually practice the rotation is that something will happen, if you will be compromised, it will suddenly be an emergency and then you'll end up scrambling around in practice trying to figure out how to do this while you're under time pressure or monetary pressure or people yelling at you or all of the above. I would love to tell you that I did not learn this the hard way, but in fact, I have beaten exactly the situation, don't make my mistakes from younger me. It's really much, much better to just automate it all. Whether you automate it or not, actually practice the rotation thing and make sure that the people who are still at the company know how to do this. I think we're about to dive in on the demo stuff. Richard, was there anything that you wanted to add to all this? No, I muted. Richard, I think you muted. Sorry. I was curious, it's a bit of an off the cuff question, but in the default LinkerD installation, when it creates these self-signed certificates, does it store them in Kubernetes secrets? It does, yes. Okay, in the cluster. In the cluster, and that actually is a really good point that used to be in the slides and is no longer in these slides from when I was editing things. Generally speaking, so let's actually back up a moment. LinkerD must have access to the secret key for the identity issuer certificate because that has to be used to issue new workload certificates. The workload certificates get rotated very frequently. LinkerD does not actually need access to the secret key for the trust anchor. And because it does not need it, LinkerD should not have access to that. In a perfect world, the secret key won't even be on the Kubernetes cluster. In an imperfect world, it'll be stored on the cluster, but it'll be in some place where Kubernetes RBAC prevents everything except cert manager from seeing it. Yeah, I was hesitating to ask because I know that in the demo, it's a slightly imperfect version. It is slightly imperfect, yeah. And we'll talk about that as we go through the first chunk of the demo. But yes, that is an excellent point. You really want to have things to not have the secret keys on the cluster where you don't absolutely have to have them. That's just part of that is because Kubernetes secrets themselves are far from ideal, let's just say. If you have RBAC access, you can go and see the secret in all its glory. You can read it back. You can do whatever you want to it. And yeah, so secrets are a little bit less secure than their reputation often implies. But the other one is just a general security principle, the principle of least access. If you are trying to do things that are where security is important and security is always important, you should always arrange it such that a given entity does not have access to things it doesn't need to. That's much less about reducing temptation and much more about reducing the blast radius from the inevitable mistakes that will be made. Yeah, secrets aren't too bad as long as the RBAC is correct. Right, and we all know how easy it is. In my experience, that's the difficult bit. Yeah, we all know how easy it is to get RBAC perfect, right? Yeah, I've never made mistakes with RBAC, never at all. It's a tough mechanism to work with, it really is. Well, from the perspective of the cert manager maintainers, we know that when you install cert manager, to make it easily installable and to work out of the box, the RBAC that we ship with cert manager allows it to read and write every secret in the Kubernetes cluster. Yes. It needn't do that, but there's no really good mechanism to allow it only the access to the secrets that it manages. At least we haven't figured that out yet. And part of the problem there is that the secrets the cert manager manages could be in any namespace, for example. Exactly, yeah. So you can't just, it just doesn't work to ship cert manager with default RBAC that only allows it access to the cert manager namespace that wouldn't work for the case that we're about to demonstrate, for example. But yeah, pay attention to RBAC. That's a good thing. Actually, we do have a bit more to talk about there. We already talked about this, making rotation easy. Don't do it by hand. That's how you make it easy. Richard, do you wanna talk a bit about some of the background for cert manager, trust manager? Yes, I'm not as good as you at speaking fluently, but I'll try. Well, cert manager has existed for about four years, I think, and it was born out of a previous project by Jetstack called Cube Lego, which is based on the Lego Acme library that it forego. Anyway, cert manager is a controller, is an operator that you install in Kubernetes. And it has a bunch of CRDs, custom resource definitions, which allow you to configure the desired state of a TLS certificate, which will be signed by an external issuer or by cert manager itself, depending on what your requirements are. And the issuers that cert manager you can interact with are quite extensive. There are five built-in issuers and there's a plugin mechanism which allows people to write external issuers for projects such as Google's CA service. And so you can supply a custom resource which tells cert manager the desired TLS certificate and it will put that into a secret in Kubernetes. Trust manager is a relatively new tool which we've produced, which is responsible for distributing the public CA certificates, which may or may not be related to the cert manager certificates, but it will take a source, that could be a secret or a config map, or it actually can retrieve public CA certificates from Docker images and it will publish those two config maps in one or more namespaces in the cluster so that those CA certificates can be consumed by servers and clients in Kubernetes. And we will be using both of those. We'll be using both of those, yes. I'll stop talking. It wasn't really what I was hinting at. We've also talked about this, but just to reiterate, and I was wrong earlier, I moved this slide instead of deleting it. I hate it when I do that. But yeah, the most critical thing here as we were talking is to pay attention to our back. The purpose of trust manager is to be able to arrange things so that cert manager can manage a certificate within its own namespace that can be locked down very tightly and then trust manager can copy the public half into another namespace so that another piece of the system can access it without also having to have access to the private key. Yeah, that's the bit I should have said, yeah. What people have done until the invention of trust manager, cert manager will put the CA certificate in the same secret as the private key in the signed certificate. And people have tended to mount that secret in for the benefit of the clients that need to verify the servers that actually need to consume the private key in signed certificate. But that's not ideal. They should only, those clients should only have to mount a config map. It's not secret material. So it can, it should live in a config map and they only need access to the CA certificate. Yeah, and just to reiterate, the private key must be kept private. The public key is public. You can do anything you want with it. So you should be very, very careful about not letting anything see private keys that don't absolutely need it. Okay, we will now go and do a demo and hope that the demo God smile upon us. There's a bit of a white lie in this slide, which is that Richard and I were going through and editing a bunch of things in the demo. So it's not actually available at that PKI and production director yet, but it will be shortly. And we will be using a few tools here. We'll be using step. This is the step CLI from small step. We're going to only be using it today to examine a couple of certificates. It is a remarkably handy tool for working with certificates. We will be using Helm to install a bunch of things, including cert manager and trust manager. And we will be using the Linkerd CLI as well. We will not be using it to install Linkerd today, but we will be using it to go through and look at things and run some checks. So you will need those tools. Once this is all published and you go a look at it, you'll need those tools to go through and actually walk through this. And yeah, I believe, yes, if we could get that GitHub Link posted into the chat as well, that would be lovely. Yep. All right. Let's see if the demo gods are going to smile upon us. I have already set up a cluster. I'm actually using a Civo cluster today to run this in. Flynn, should you say, I had never heard of Civo. Oh, very. So you introduced it to me. Tell us about that. They're a cloud, wow, sorry, can't talk. Civo is a cloud provider for Kubernetes clusters. I personally find that it's usually quite a bit faster to spin up Civo clusters and recycle them and such than if I'm trying to do the same thing with GKE or the like. And I like that. So yeah, I tend to use them when I want cloud things. This demo should also work fine. Yes, Achmed is correct. Civo is, I don't know all the details, but I do know that they are using K3S under the hood to make things quick. This demo should work just fine if you do it in K3D or if you do it in kind. I have in fact done it in K3D, but today we're going to use a Civo cluster. I tried it in kind, it did work for me in kind. Excellent. Glad to hear it. All right. So we're going to start by setting up Helm so that we can use Helm to install LinkerD and CERT Manager and Trust Manager. And I'm pretty sure that these repos all exist already for me, but there we go. Usual, add the repos, do a repo update, off we go. Next thing we're going to do is install CERT Manager itself. We need to install CERT Manager first because we're going to use CERT Manager to create all the certificates that LinkerD needs to do everything. And so there we go. We're, there's nothing particularly crazy about that. It's just installing CERT Manager. There is a newer version of CERT Manager. I was just about to say, I really should have checked with you what the proper version was. It was released in early January. It's 1.11 is now out. Well, I will update the demo and I assume everything will still work. After CERT Manager has been installed, we will switch over to install Trust Manager or CERT Manager Trust. I guess it's official chart name is. Same thing, we're going to install and then we're going to wait for it to come up and be running. Yeah. We'd be agonizing over the name of it. Trust, it used to be called Trust and now we're calling it Trust Manager. That's the name. Do you think that the chart name will change sometime soon? Yes. I think my colleague Ashley is attending tonight. Yeah. Okay. Excellent. All right. So now we have CERT Manager installed and we have Trust Manager installed. Next up, we're going to set up CERT Manager in a slightly less than production way. We're first going to use, so if you remember the slide from earlier, actually let me get to that slide from earlier and then I'll bring it back up. Okay. So you remember this slide where we've got a CA root certificate being managed by CERT Manager, which is going to sign our Trust Anchor, which is going to sign our issuer, which will sign the workloads. In this case, we're going to just set up a self signed CA root with CERT Manager. This is not 100% ideal, but it is a very simple way to demonstrate all the concepts. And then after we do that, I'll hand it over to Richard who can talk about how we're going to tweak this a bit to make it better. All right. And actually, let's put back over here. We use CERT Manager to create a self signed root CA. We will then use the root CA to create a Trust Anchor certificate signed by the root CA and backing up to our previous conversation about private keys and such. CERT Manager will be creating the Trust Anchor certificate in the CERT Manager namespace and Ligardy will have no access to it at all, not to the private key, not to the public key, nothing at all. We will then use the Trust Anchor to set up yet another CA, which will use to sign the issuer certificate. The issuer certificate we will create in the Ligardy namespace because Ligardy actually must have access to his private keys so that it can use it to issue workloads certificates. Finally, we'll set up Trust Manager to copy the public key for the Trust Anchor out of the CERT Manager namespace into the Ligardy namespace so that Ligardy has access to the public half of the Trust Anchor's keys, but only the public key. So off we go. The big payoff, just to make it clear, is that the Ligardy Trust Anchor is no longer self signed and no longer has to be managed by hand. We can let CERT Manager do all of the rotation for us. Okay, so you'll recall a point where I said that we're going to use CERT Manager to create things in the CERT Manager namespace, but we have to copy them into the Ligardy namespace. That implies that we need for the Ligardy namespace to exist, so we'll go ahead and create it first. Next up, we're going to use, you're gonna set up that root CA and we will use a CERT Manager custom resource to do this. It is a cluster issuer, which I believe means an issuer for things that live in the cluster. Is that the semantic there, Richard? Yeah, yeah, it signs certificates using a private key which it finds in a Kubernetes secret in the cluster, yeah. Excellent, and as Richard mentioned, there are many different kinds. Oh, I'm sorry, I misheard the question. You were asking about cluster issuer versus issuer. Right. The cluster issuer is a non-namespaced resource. So an issuer lives in a namespace. You got it. And it can be referred to by certificates in the same namespace, and any credentials associated with that issuer must be in the same namespace as the issuer. So we actually might want to, we might want to go in and swap this for an issuer, but we'll do that later. And a cluster issuer is a non-namespaced, a cluster-wide resource. And it's, any credentials associated with it need to be in a single designated namespace, which is CERT Manager by default. Got it. So in this case, we're going to name our cluster issuer a Linkardy self-signed issuer because we're going to create a self-signed issuer. And this self-signed bit just says, so for this issuer, create a self-signed root CA. So we'll go ahead and apply that. Also, I should point out this, it is easy with CERT Manager to swap the CA out. So you actually can do things like bootstrap with this self-signed issuer like we're doing here, and then go through and tweak things without having to completely tear the cluster down. All right, so let's go ahead and apply that. And now that we have that self-signed root CA, we will tell CERT Manager how to create the trust anchor certificate. This is a little bit more complex, so we'll just walk through this. This is a certificate. Its name will be Linkardy trust anchor and it will live in the CERT Manager namespace. The common name, which is to say the name in the subject of the certificate, like we saw earlier, must be root.linkardy.cluster.local, because that is the name that Linkardy requires it to be. CERT Manager will create this as a CA certificate. A CA certificate is a certificate that can be used in turn to sign other certificates instead of only being useful for being at an entity. In the x.509 community, you will very commonly hear people talking about issuing certificates. Outside it, you will often hear people talk about signing certificates. They are the same thing. CERT Manager talks about issuers, so the issuer for this cert will be our cluster issuer that we just created, the self-signed one. We will specify that it will use a 256-bit elliptic curve digital signature algorithm, private key, and it will be stored in a secret named Linkardy identity trust roots, which again will live in the CERT Manager namespace. We didn't say anything about durations or renewal times or anything like that. I should actually double check this with Richard. This comment's been around for a really long time. Is this actually the correct default value here? Is it 90 days and renewed at 30 days or is that no longer true? Caught me off guard there. I think it is still a default. I shall check. Excellent. Okay. After creating that trust anchor certificate, and we're going to define a second cluster issuer, that instead of being self-signed, we'll use that shiny new certificate that we just created to issue certificates. And that will be called the Linkardy trust anchor cluster issuer, but again, that will be in the CERT Manager namespace. Okay, so we're going to apply both of those. Now, issuing certificates is fast when you're dealing with CERT Manager, which is kind of nice. So in fact, the trust anchor should already be there. And we should be able to look at it using kubectl describe secrets. We can see that, well, it's there. It has the right name. It's in the namespace we expect. It has a ton of annotations, which is all right. It also has various keys. TLS.key is the private key in the secret. TLS.crt is the public half of that key. And ca.cert is the issuer's public key. You will notice that in this case, ca.cert and TLS.cert are the same size. That's because they're the same thing. So if we go through and look at that, this is one of the annoying bits about, one of the annoying bits about secrets in Kubernetes is that the actual value of an X509 certificate, it's usually rendered as a PEM format, which is itself base64 encoded, then Kubernetes base64 encodes it again. So if we want to though, we can use this horrible bit of kubectl stuff to pull out the public key and then decode it and then pass it to step certificate inspect so we can see what that certificate actually looks like. And it actually looks like this, where it's got a subject of root.linkard.cluster.local and we've got the public key and we've got the signature algorithm. Everything is great. All right. Now, even that we have that trust anchor certificate and we have a corresponding CA going with it, then we will tell cert manager how we want it to use that to create the identity issuer certificate as well. The big difference between the certificate resource that we use here and the one that we used for the trust anchor is that this certificate must be created in the linkard name space rather than in the cert manager name space. So if we look at this again, we're in the linkard name space. The common name must be identity.linkard.cluster.local rather than root.linkard.cluster.local. And once again, that is the name that linkard requires it to be. We've done a couple of things where not only is this a certifying, not only is this a CA certificate, but we've explicitly said some things that it is allowed to be used for, mostly to show that that's possible. We're also gonna explicitly say that this certificate will only be good for 48 hours and it must be renewed no more than 25 hours after issuing. This is a relatively common thing. It is possible for the renewal to fail in the real world in practice. So if you have something that expires at 48 hours, don't only try to renew it at 47 hours, renew like halfway through your actual validity period that gives you lots of time to figure out if something is going wrong and to fix the errors. It's probably also worth pointing out that I've never actually seen cert manager fail its renewals, but it can happen. It can be, especially if it will retry, but if you're issuing via, it's not applicable here, but if you're using let's encrypt or some external service and cert manager fails to contact that service for whatever reason, it will retry, but you want it to be going through that process well before the current certificate is due to expire. Flynn, the renew before, it's a little bit confusing. I may be confused, but it's actually the number of hours before the certificate is due to expire. Oh really? See, this is why we do this. So manager will renew the certificate 25 hours before the certificate's end time. Today I learned. See, this is why we do this with people from other colleagues across the industry is because even the presenters always learn things while we're doing this. And the default, as you said, going back to what you asked me earlier, the default is 90 days and the default renewal is at two thirds of two thirds of the way through the certificate lifetime. That's 30 days before the end. Got it. Great. So presumably if we said duration 48 hours and then we didn't put renewal before, it would renew 16 hours before the end of the period. Yeah, yeah. Yeah, okay. All right. Hang on a moment. I'm going to make a note of that so that I can correct it. Renew before is the time before erasure, not time into lifespan. Okay. Excellent. Right. This one will be issued by the Linkardy trust anchor issuer, not the root CA. Still doing a 256 bit ECDSA key, but this one will be stored in a secret named Linkardy identity issuer again in the Linkardy namespace, not the server management space. So we'll apply this one. And once again, creating certificates is fast. We can see that that certificate is there in a secret. And we can repeat pretty much exactly the same thing that we did earlier with grabbing data.tls.crt. It's probably worth pointing out. The backslash here is important because the dot is actually part of the key name here. It's not the separator, not a hierarchy separator. Thank you, Kubernetes. Base 64 dash D to decode it, step and certificate inspect to look at it. And we can see here that the subject is identity.linkardy.cluster.local. It was issued by root.linkardy.cluster.local. We can also, I didn't actually go and look at the validity period for the trust anchor and should have, but you can see that this one is in fact a 48 hour expiration. That finally brings us to trust manager. We're gonna set up trust manager to copy the trust anchors public key from the cert manager namespace into the trust anchor bundle config map that Linkardy uses in the Linkardy namespace. This is, that sounds a lot more complex than it is. The name of the bundle, the name of the bundle resource becomes the name of the config map. It must be Linkardy identity trust routes. We will fetch the tls.crt key from Linkardy identity trust routes. And we will write that into the config map Linkardy identity trust routes with a key of cabundle.crt. And that config map is actually always the thing that Linkardy looks at to figure out how to verify the certificates. It always looks into a config map, not a secret for the keys for the trust anchor because as we were pointing out earlier, there's nothing secret about the public key. We don't need a Kubernetes secret, a config map is just fine. So you were gonna say? Just to say that Kubernetes itself puts the CA certificate of the API server into a config map, doesn't it, Flynn? Yeah, it does. So it's quite a standard procedure to put that material into a config map. Yes, there actually is, if I remember correctly, there is a slight additional expense on the API server for secrets instead of config maps. And so where they don't need secrets, they don't use them. So creating, again, trust manager is fast in copying things into the config map. So we can look and see that. There you go. We could go and pull out the key and run that through step certificate inspect, but I'm not gonna bother because we've done that already so many times. Finally, having done that, we get to install LinkerD. We are not gonna use the LinkerD CLI for this. We will use Helm. First off, because Helm is arguably a little bit better for production installation, it makes it a bit easier to do upgrades and things like that. But also because you're gonna see how to use Helm to tell LinkerD to use cert manager CA rather than spinning up stuff on it's end. First things first, we must install the LinkerD CRDs. Then we will go and install the LinkerD control plane itself. We will install that into the LinkerD namespace which already exists. We will tell it to use an external CA for identity and to still use the normal Kubernetes IoTLS secret for the identity issue server. Can I, maybe one thing I'd say, I noticed I agree with the way you're installing the CRDs first. And that's actually what we advise people to, how people, that's how we advise people to install cert manager too. When we installed cert manager earlier, we used the install CRDs flag in the Helm chart, but that's not what you should do in production. Because the thing is that if you uninstall the Helm chart, after installing it that way, you remove all the CRDs which will cause Kubernetes to garbage collect all of the certificates and the custom resources, which is not what you want. Yeah, that's actually a really excellent point. And that also is a thing that we can go and update. So you'll notice that I didn't use dash dash weight on that Helm install command. So Lingardee has been merely installing in the background. Oh, look, it actually finished. And an interesting thing here is we can see that the trust anchor is valid. The trust anchor is valid for at least 60 days. In fact, it's valid for 90 days. But we can actually see that our issuer cert is complaining in Lingardee check because it's going to expire in only two days. That's okay, that's what we told it to do. Just a thing to be aware of. All right, now, one last thing we're gonna do before handing it over to talk, having it over to let Richard talk about more production real setup is we're going to install our EmojiVoto demo application. And the reason for that is that I want to show y'all how to look at the workload certs as well and make sure that they are signed by the correct things. There we go. So EmojiVoto is now running and we can use the Lingardee CLI, the Lingardee Identity Command to look to show us identity workload certificates within the EmojiVoto namespace that match the label app equals emoji service. That will be exactly one workload in the EmojiVoto namespace as it happens. And if we do that, we can see that it has a subject of emoji.emojivoto.serviceaccount.identity.linkardee.cluster.local. We can see that it lasts for 24 hours. Actually, one minute more than 20... 30 seconds more than 24 hours. That's kind of fascinating. I'll have to find out what's up with that. But that it was issued by identity.linkardee.cluster.local. We can also see if we look down into these extensions, you will see that this is not ACA certificate. It cannot be used to sign anything else, which is good. Okay, so that is all about going through and setting up CERT Manager with a self-signed issuer but having CERT Manager do all of the certificate management for us. And now, Richard, I think you were gonna share your screen here, right? Yes, and talk just a bit about doing this a bit more realistically with Vault instead of a self-signed issuer. I'll share my screen. And... And when I do this, I think I'm not gonna be able to see the chat and the comments, so tell me if people are asking any questions. I'm gonna stop my screen share. And... There we go. Excellent. Can you see my screen? I can. Okay, perfect. I love it when technology works. It's so nice. Yes, so, well, Flynn has really covered the problem. He's already explained the shortcomings of the demo we've just seen. And those are the trust anchor certificate and its private key. Although they're not accepted, they are in the CERT Manager namespace, but they needn't really be in Kubernetes at all. The private key, at least, needn't be in Kubernetes at all. And so Flynn challenged me to figure out whether it was possible to use CERT Manager with an enterprise issuer like Vault or like... And yeah, we're getting a comment from Dionys. Could we get a bit of Zoom in? So they're having some difficulty seeing. Oh, I guess. Like, enlarge the view a bit. There's a new laptop. I'm struggling a bit. That's better, at least. Yeah, we can try. We've done anything. Yeah. Okay. And if the audience, you feel like you can't see even with now, just let us know and we'll think about the solutions. Well, I'm going to switch now to Emacs, which is probably going to be even worse for people. And so, yeah, so let me switch to a single screen. So what I was saying was that it is possible to configure CERT Manager to sign CA certificates, to get signed intermediate CA certificates from software like Vault, how she called Vault, or from Venify TPP. And so... And various others. And various others, yeah. And in the interests of making... Venify TPP is a piece of enterprise software, which I'm familiar with, but it's not free for anyone to use. And it's quite difficult to set up. So we chose instead to hash a quote Vault, which is available for people to install themselves. And it's quite easy to set up inside Kubernetes, and that's what I've done. I deployed it inside Kubernetes using its Helm chart. I've configured it to... And I've deployed it with some very unsuitable for production configuration. Just to get it up and running fast, I'll show you what values I used. Do you basically followed the Vault Quickstart, right? Exactly, yeah. It gets it running inside Kubernetes without much security, and with a fixed API token. That was just... So we installed it. We configured its PKI system. And we created a root certificate inside Vault called root.linkadee.pasta.loop, which mirrors what you saw Flynn create earlier in his demo. And that certificate and its private key are not held in Kubernetes secrets. They're inside Vault and totally inaccessible to the workloads running in Kubernetes. And then what you can do is you can set up a cert manager to be able to connect to that Vault server. In this case, we've done it in a very simple way by providing its API token in a secret. But you wouldn't do this in production. You would use like a workload federation system, which it supports as well, where you can use an ephemeral Kubernetes service account token to authenticate the Vault. Anyway, you would set up a different type of cluster issuer, which I'll show you here. It's very similar to what Flynn showed you earlier. The difference is that you use a slightly different stanza in the cluster issuer spec. And what we do here is we're telling cert manager to connect to this API endpoint, this Vault server, which is the address of the Vault server inside the Kubernetes cluster, telling cert manager to authenticate using the token which we've put into this secret. This API endpoint is not the standard API endpoint. It's a dedicated endpoint for signing intermediate certificates. So that's important. And as Flynn said, we'll publish this to the repository later once we've tidied it up. Really, we were doing this. I was doing this right at the last minute, so apologies for not being more prepared. Yeah, we never do that for demos. The certificate, though, is basically the same. It's exactly as Flynn showed earlier. The common name, which is significant, it must be this common name for linkerd to use it. It must be marked as a CA certificate. But at this time, in the issuer ref, we refer to this Vault issuer, which I showed you. And when we deploy this, cert manager will, let's look at the certificates, kubectl, get, cert, minus n, linker, e. We should probably point out here that this is a different cluster than the one that I was using, where just in the interest of time, Richard went through and set all this stuff up and then is walking through. So this is a kind cluster, right? Yeah. Yeah, again, shouldn't really matter what kind of cluster it is, pun not intended. And so there's a certificate, and cert manager has marked it as ready. And that means that there should be a secret, a Kubernetes secret associated with that. And it is here. Excellent. We can use, Flynn, you were showing the step command. I was going to demo the cert manager command line tools. Oh, great. So cm, ctl, inspect, allows you to decode the contents of the CLS secret. And presumably that works with any secret, not just any cert manager. Any TLS secret, yeah. You got it. Good enough. And what I wanted to show was that this is a CA certificate that it's been issued by root.linked.cluster.local, which is the CA certificate inside Vault. And it is issued with this common name, the common name of the, what's the term, Flynn? The issuer issuing. It's a common name of the subject. Yeah. The name of this particular linker DCA is called the issuer CA or the issuer CA. Oh, I'm sorry, the identity issuer. Identity issuer CA, yep. Anyway, so with that in place, we then use trust manager to copy the, this time we copy the CA cert file from the linker ID identity issuer. So when cert manager, sorry, go ahead and go on Flynn. No, just in this case, trust manager doesn't have any access to that root CA either. But trust manager does have access to the intermediate certificate where it's CA.CRT is the public half of the root CA. So we can get it out that way, right? Yeah. Cert manager, as well as putting the key in the science certificate in the secret, if it can, it puts the CA certificate that it receives from the issuer, from Vault in this instance, into a file called CA.CRT. Yeah, it will soon be running on a time as well. So if we can, let's start kind of getting towards a wrap up, but you still have a bit of time, a few minutes, so. That really is it, and all I was going to show then was a linker ID check, show, sorry, verify. No, linker ID check is. What is it? Yeah. Show that linker ID is in the. All green check marks. Yeah. Nice. Oh, you even have a business installed. Nice. So that's it. That's a slightly different way of achieving this integration between cert manager and linker ID. And again, we should emphasize here that the Vault setup that Richard was using is not really the production Vault setup, but the way in which cert manager is pulling all this stuff out of Vault, the way cert manager is interacting with it, that part will work fine with a properly production ready Vault instance. Yeah. Excellent. Can we flip? Let me share my screen again. If we can go back to that really quickly, there were one. Oh, lovely. Look, my browser window is a bit frozen again. We'll hopefully it'll come back. Well, there we go. Perfect. OK. I wanted to point out the first linker ID day ever is happening at KubeCon in Europe. If you're going to be there, we would love to see talks and things like that. There's a truly atrocious URL that I could post here. But instead, I'll just say, go to linkerty.io, click on the community menu, and you'll see the linkerty day 2023 EU thing to click on. Also coming up is KubeCrash, as well brought to you by a number of people, including Boyot, and Sevo, and Carcroach, and Fairwoods, and Jetstack. KubeCrash.io is the URL there. And again, we would be delighted to see everybody there as well. And other than that, you can reach us by email fluentpoint.io or richard.well.jetstack.io. And hopefully we have a few minutes for questions. We are out of time, but if we had a super quick question, we could have taken it, but I don't see anything in the chat currently. But I do like the fact that we're sharing your contact details here, so if anyone has any questions, please go ahead and jump over to Flynn's and Richard's info and ask the questions there. And I think this works, and this bit of stalling for me and no questions came in that time either. So we're perfect. So yeah, jump to those contact details with your questions. And we'll start wrapping up. Really good content, though. I really loved it. Yeah, so thank you very much for joining the latest episode of Cloud Native Live. It was great to have a session about real-world trust management with LinkerD Insert Manager. We also really love to view audience comments that we have and always look forward to those. But as always, we bring you the latest Cloud Native Code every Wednesday, and in the coming weeks, there's even more sessions coming up, and they're going to be great, so tune in. So thanks for joining us today and see you all next week. Thank you, much appreciated. Thanks.