 Hi everyone, welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Annie, and I'm a CNC ambassador, as well as a senior product marketing manager at Camunda, and I will be your host tonight. So every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer all of your questions. So join us every Wednesday to watch live. So this week, we have Lynn here with us to talk about certificate management with LinkedIn. And as always, this is an official live stream of the CNC app, and as such, it is subject to the CNC app code of conduct. So please do not add anything to the chat or questions that would be in violation of that code of conduct. Basically, please be respectful for all of your fellow participants, as well as presenters. But with that over, I'll hand it over to Flynn to kick off today's presentation. Hi there, so yes, I am Flynn. I'm a tech evangelist for buoyant, working with LinkedIn. These are some ways that you can reach me. The best one's probably the Slack link at Flynn on the Linkardee Slack. You can also tweet at buoyant or find me on GitHub if you really need to. I'm not sure why you would. This is a workshop. We do have a bit of background first since we're talking about certificates and TL and all this kind of stuff. But we are gonna be spending most of the time going through doing hand-on, either working a demo or breaking things. We'll find out which. Tools that you need, if you want to follow along from home, and I hope you do, you would need a code control command. I'm gonna be using a K3D cluster. Doesn't really matter what kind of cluster you're using as long as you have one. We will be using Linkardee. You can get that from linkardee.io. We'll be using the step CLI, sorry, the Linkardee CLI in particular. You will need that. You'll need the step CLI from smallstep.com. We're gonna use that to manipulate certificates. And you'll need the Helm CLI because we're gonna use Helm to install cert manager later on. Okay. So I'm gonna go into the background stuff now and hopefully that'll give everybody a little bit of time to collect tools if they need to get any installed. Background. If you're unfamiliar with Linkardee, it's the only CNCF graduated service mesh. Its purpose is to arrange for cloud native developers everywhere to have free access to the tools that they need to make sure that their applications include observability and reliability and security. This talk is all about security. So we're gonna start off by pointing out that security and Linkardee starts with identity, starts with knowing which workload is talking to which other workload. That's the core of everything we do so that we can make good decisions. So we need to go through and talk a little bit about the technologies underlying that to understand how that actually works. First one, public key cryptography. This is the thing that forms the basis of trust in Linkardee. It's a very simple idea instead of having one key that you use for encryption and decryption, you have a key pair. You have a public key and a private key. These two keys are linked mathematically with a few properties in particular. First one is anything that you encrypt with the public key, you must have the private key to decrypt. Conversely, if you encrypt something with the private key, you must have the public key to decrypt it. And finally, if you only have the public key, recovering the private key is impossible. The air quotes are there because if you want to get technical, it's merely astronomical unlikely. In practice, it's not gonna happen. These three properties give rise to some really interesting aspects of public key crypto. The first one is, if Alice were to encrypt some data with Bob's public key, only Bob will be able to read that data because Bob is the only one that has Bob's private key. Like as if Alice were to encrypt something with her own private key, then only she could have written that because she's the only one with her private key. This business of encrypting with your own private key, by the way, is actually really common. Typically, Alice would actually take some data to send to Bob, encrypt a hash of it with her private key, and then he could verify that if he can decrypt the hash with Alice's public key and it matches the data, he knows that she's the one who sent it. That's so common, it's called signing, it has its own name. So given all of that stuff, couple of things. Couple of things, one is the private keys have to be kept private and they form the basis of identity. If only one person has the private key and we can verify that it really is that private key that did something, that's a good proxy for identifying the person holding that key. Converse the keys, can and really should be shared with everybody because the more people that have them, the easier it is to do verification of identity and both of these things are important. Also, the more widely say Bob shares his public key, the easier it is for Alice to get it and send him some data. And that's really the name of the game is that public key crypto allows Alice and Bob to safely communicate with each other but also to safely know who is on either end of the channel and they can do this even if they haven't interacted before. MTLS happens to rely on this and Linkardy authentication ultimately relies on MTLS. All this stuff is important. It's also worth pointing out that one of the interesting problems here is we're using a private key as the root of identity but we haven't really talked about, okay, how do you manipulate the private key? How do you share the public keys to make all of this work? That leads us into the world of certificates which are usually everybody who's trying to work with anything with public key but they're also fairly simple. A certificate is a data structure that takes a key pair and associates a name with it. The name is called a subject. The association is done via a signature so that you can know that this identity, this name really does belong to this particular key pair. Certificates only contain public information so they are safe to share. This is important, you have to share them. People will be able to use them. The signing again prevents them from being spoofed which allows them to be trusted for identity and we're gonna talk quite a bit more about the whole manner in which they are signed. But there's also a last bit here which is rotation. There's a basic axiom in cryptography that the longer you use a given key, the more risk you've piled up with that key and the more attractive it becomes for an attacker to try to steal it. So to mitigate this, it's very important that you rotate certificates periodically to change the private keys. Of course, if you're changing the private key and we're using the private key as the basis of identity that creates kind of an interesting problem that gets solved by having the signer of the keys stay the same as you're rotating the key underneath it. But rotation is a very important thing. This is one of the things we're gonna look at in the workshop is how you rotate keys with Linkardy and it's important that you do it periodically where we'll talk about what periodically means in a minute. So any questions so far? I realize this is kind of a fair amount of stuff and just kind of going over things fairly quickly here. All right, onward. So let's talk about the guts of Linkardy's operational model and trust. Everything with trust in Linkardy starts with certificate which we call the trust anchor. The trust anchor in turn is used to sign issuer certificates. The issuer certificates then get used to sign workload certificates or leaf certificates which are associated with workloads. The immediate reason why we don't just have one layer of signing here is that this way we can rotate the issuer certificates very often without having to do anything with the trust anchor. Also, the workloads get to use the workload certificates to protect calls from workload to workload but by having multiple issuer certificates we also get to separate this completely from the topology of the network so that you can have workloads that might even be in different clusters as long as there is a relationship with the trust anchor then this works fine. And workload two for example can just use workload three's private public key to encrypt data is sent to workload three, workload two can sign it with workload two's private key, workload three can use the public key for workload two to verify everything and it all works. None of this relies on the topology of the network. None of it interacts particularly much with the topology of the network. As long as the workloads have a pathway by which they can talk to each other everything works fine because identity is a function of the certificate not of the network topology. Okay, so with that in mind I mentioned we can easily rotate the identity, the issuer certificates without having to mess with the trust anchor. So that permits us to rotate the identity issuer certificates very often. There's a trade off to be made here. If you rotate certificates quickly it limits your risk because if a certificate gets compromised it will be compromised until at least the next rotation. That means that if you rotate every two months an evil doer has potentially two months to go through and mess with you. If instead you're rotating these identities every few hours it makes it much, much, much harder for an evil doer to actually use the certificate effectively. There are two really, really big counterpoints here. One is if you're rotating the certificates more often you're gonna be spending more of your time messing with certificate rotation. The other is you have to actually practice the rotation mechanisms. You have to go through and actually do it to make sure that it works and to make sure that you actually know how to do it or you'll find when the time comes that in fact you won't be able to do it correctly. The converse to that first one by the way is that if you rotate on a longer cycle you can spend more of your time focusing on your work instead of focusing on rotating certificates in your platform. So there's not necessarily a right answer here. It's just a balance that you have to strike for any particular application that you're working on. Okay, workshop time. Again, I encourage you to follow along. There's the URL for the repo that I'm working out of. You'll be able to see this stuff in the L5D certificate management directory in the service mesh academy repo. And again, we're gonna be using the step CLI, we'll be using Helm, we'll be using cert manager and we'll be using, of course, the Linkerd CLI. There's a note about my audio. I guess I'll hope it gets better. Annie, does that sound okay to you? There is a bit of breaking in the microphone but at least for me I can still kind of understand you clearly. If there's any quick fixes to be done, we could try those but I think- Well, I guess we'll see if this gets any better. Guess we'll see if it gets any better. So fingers crossed. All right, in the workshop, we're going to cover a few things. There's the basic skill of generating and inspecting certificates using the step CLI. We're gonna do a staging setup where we create a cell-signed trust anchor and then use it to create the other certificates to get Linkerd running. We're going to talk about the operational skill of rotating kits. And then we're going to do a production setup where we bootstrap all this stuff with CERT Manager to make life easier on us, the people who are trying to get work done. For homework, you'll get to fully automate the identity with CERT Manager. We're just going to do the bootstrapping because once it's bootstrapped, going to fully automate it is usually a lot easier. So let's get to it. And you get a early look at the KUCRS thing which you should sign up for but we'll come back to that later. Okay, you make certain that this is going correctly. Here we go. Just change some settings in the backend as well. So hopefully the audio will get a bit better as well. But yeah, fingers crossed, you never know. Fingers crossed. All right, but everybody can see that shell window, right? Yes. Excellent. Okay, so basic skill. How do you use step to generate and inspect a certificate? First thing to remember here is that step dash dash help is easy and useful and gives you a lot of good information. So here's the basics for generating a certificate. This looks a lot uglier than it really is. We're going to break all this down a little bit later but this is the name that goes into the certificate. These are the files where we store respectively the public and private halves of our key pair. This tells us what kind of certificate we're going to create. This example is showing a root, basically a trust anchor, the thing at the root of a certifying authority that can be used to sign other certificates. And these are things that are important for demos and not necessarily a great idea in practice. They are saying that they, you don't want to protect this cert with a password and you know that that's insecure because it is. But that's what we're going to do just so we can go ahead and get this going. It generates it quickly. It saves it. If we try to just look at this certificate, it's kind of useless. This is what the step certificate inspect command is for. If we use this, we see a bunch of things. Again, we're going to break this down a little bit later. But some particular things are, you can actually see the public key. You can see a signature that was made with the private key. You don't get to see the private key itself. That's private. Okay, onward then. So we're going to use this to create a self-signed trust anchor certificate, which we will then use to create those issuer certificates, which we will then use to set up LinkerD as if we were kind of doing this in staging. Now in staging, it is often okay to be doing manual rotations at a long expiration times because again, in staging, you probably, or development, you probably want to be concentrating more on the experiments and the applications that you're working on, rather than spending a lot of your time going and rotating certificates. On the other hand, you could always go through and set all of this up with CERT Manager, even in staging, that would be another fine idea. In this case, we're going to do it all by hand so that we get to go through and get some experience with manual rotations. So we start by building a self-signed CA trust anchor. So here, route.linkerd.cluster.local must be the name that you use for a Linkerd trust anchor. We're going to save these, the public half in trust anchor.CRT, the private half in trust anchor.key. Again, we're going to pick the root CA profile. Again, no password insecure and we're going to make this not expire until St. Patrick's Day of 2060, just so we don't have to worry about it for right now. Let me remind you, this is not a good idea in production. We're not talking about production yet. So save that, we can use inspect. If we look a little bit more in detail here, we see that the subject does indeed have our name. The issuer is the certificate that signed the certificate. It has the same common name, that's what CN stands for, meaning it's the same certificate. We've used this certificate to sign itself because we're going to just declare that this is the root of trust in our world. It expires in 2060. It is valid not after March 17th of 2060. The other thing here is, it's okay to use this to sign certificates. It's okay to use this to sign certificate revocation lists, which is a particular kind of thing that you can use to spread around to the world and say, hey, I have found a certificate that was compromised, don't trust it anymore. This is part of a certifying authority. And this is something we're gonna talk really briefly about because it's gonna come up with the issuer certificate. Path length of one means, this certificate can sign another certificate that signs a leaf certificate. That path link of one is, how many other signing certificates will be in the chain from us to a leaf? Okay, any questions so far? Not so far, but that's actually a good reminder. If anyone has any questions, you can ask them throughout the session as well. So we'll get those sorted out. But there, I think there's some time reserved in the end for Q and A as well, so we can take it as well. There is, yeah, all right. So we're now gonna create an issuer certificate signed by the trust anchor we just did. This looks mostly the same. The name is different. We must use identity.linkard.cluster.local. The files are different. This is an intermediate CA, not the root CA, meaning that we are going to have to explicitly say, here's another certificate that you can use to sign this certificate with. And we will make- And then there's an audience question immediately. What is critical? Excellent. Let me come back to that when we inspect this certificate. This one is expiring in 2050. It's important to note that in general, you want the issuer certificates to expire before the trust anchor does. And the reason for this is that it's much easier to rotate the issuer certificates and they tend to have to have their keys live in the cluster. So it's also more important to rotate them more frequently. So let's go ahead and generate that and then inspect it. So I believe that this is what the question was, right? That critical means that any attempt to violate these constraints is a critical error. The operation will instantly fail. We don't usually have to worry about that too much because the tooling that you work with, like step knows how to interpret these things and it won't let you do something that would violate a critical constraint. But that's really going on. Can I ask what tool you use to speed the command on the terminal? In fact, you may. It's a thing called demo magic.sh and I'll make sure there's a link to it in the repo. I love it, big fan of it. Okay, so now we're gonna use the certificates that we just made to install LinkerD. First up, need a cluster to install LinkerD into. I am using a single node K3D cluster just running locally on my Mac. Should all work pretty much no matter what kind of cluster you have. So you do you for whatever cluster you want. We also need to know that this cluster is actually capable of running LinkerD. The LinkerD CLI has a command that makes this very easy. You can run check-pre and it'll go through and do all of the checking to make sure that we have a cluster that we can go and install LinkerD on. And everything is good. We get green check marks across the board. This is what we wanna see. So now this bit's important for people who've used LinkerD before. As of last night, LinkerD 2.12 is out. That's what I'm using here. In LinkerD 2.12, there's an additional step. When you're doing installation, you have to explicitly install the CRDs by hands now because CRDs are getting more complicated to wrangle. So let's do that. We'll go ahead and install the CRDs. Okay, next up, we install LinkerD proper. If you've messed with LinkerD before, you might be used to just running LinkerD install with no arguments. But in this case, we want to be explicit and tell it. These are the certificates to use so that we can go mess with them later. One other note, we hand in the trust anchors public key, but we don't say anything about the trust anchors private key. And the trust anchors private key is actually never going to be in the cluster at all because the trust anchor is used for verification. It does not have to sign anything. It does not need access to its private key. So it's much better to not have it in the cluster at all. Keeping it out of the cluster is safer. Sorry, there's a question here because it's manual install. Cause which I'm missing some context for that question. I'm sorry. No worries. Kenneth, if you can provide a bit more context, we'll get the question answered. And in the meantime, we'll start LinkerD installing because this next bit where we run LinkerD check to make sure that LinkerD is up and running, this might actually take a little bit of time because this is doing actual real work. No worries. That leaves plenty of time for Kenneth to type more context in. Or for anybody else. Yes, true. Yeah. But what's going on here is that we installed LinkerD, we applied all of its resources and now LinkerD check is going through and waiting for LinkerD to be fully run. Again, helpful command. It's a good one to be familiar with and use early and often. Perfect. And then there's context. So from Kenneth, the CA, it was manual. So you flipped to the fuller command. Exactly. If you don't use those arguments on LinkerD install, LinkerD actually generates temporary certificates for you. And this works great, but in this particular case where we want control over it and we want to do our own certificate rotation we basically, we want the whole thing in our hands. So we have to explicitly tell LinkerD these are the certificates you use. Also, if you use Helm to install LinkerD you always have to supply their certificates. So you have to generate them on your own. So hopefully that answers that question. We have green checkmarks across the board again. So we are actually have a running LinkerD, which is great. Now, in the real world, we would then go through and set up some workloads and get them into the mesh and test our applications and we're gonna skip all that because this particular one is talking about certificate management, not workload management. So I mentioned earlier that we were generating certificates to expire in 2050 and 2060 and that's kind of insane. It's really just, even for a staging cluster that would be too long for me to really be comfortable with. So what we're gonna do here is generate new ones which with much shorter explorations and then we'll use those to demo how you rotate new things in. Let's see, a couple of points to remember here is it can be really, really difficult to actually know when a key gets compromised especially in the Kubernetes world. So you do better to err on the side of assuming that compromises can happen more often than you probably think. Somebody stealing the trust anchors public key out of your cluster is irrelevant, it's a public key, shout it from the mountaintops. But the issuer certificates also have their private keys in the cluster because the issuer certificates have to generate and sign those workloads certificates. This means that the issuer certificates are at significantly greater risk than the trust anchor and in turn that means that rotating the issuer certificates fairly often is a really good idea, especially in production. We also mentioned briefly the CRL, the certificate revocation list. Certificates can get revoked. This is probably a little bit more common in enterprise worlds where maybe your trust anchor is being given to you by your corporate security people. It's a little bit less likely to happen if you're doing this all by hand than a five-person startup, but still it can happen and it's better to get accustomed to actually rotating certificates and treating them with some care. So let's go ahead and do that. We're gonna generate a root CA with a validity, not a validity, oh dear. A validity of two months, 1,440 hours, and we'll generate an issuer with a validity of one week, 168 hours. Again, there are no hard and fast right answers for how long is too long or too short, but it's a thing that you need to make a decision about based on what's right for your environment and the way you work. We're gonna pick these in the real world, we could put these in and then if we decided it was too much friction, change them again. It's not a big deal. Let's see, question about certificates. Daily generated, I have daily generated self-signed certificates for the same subject. I need to manually accept the trust anchor certificate from them needs to be from the certificate issuer. So let me answer this, assuming that your daily generated self-signed certs are for one of the linker D, are for one of the certificates in the linker D world. If you want to use a self-signed certificate that you rotate every day as your trust anchor, everything will work great. And you'll see the process for doing that in a minute. If you want to use a self-signed certificate as your issuer certificate, that will not work. The issuer certificate must be signed by the trust anchor and I expect you're probably not trying to use it as a workload certificate, but if you are, it would have to be signed by the issuer certificate. And if that doesn't answer your question, then let me know. If it's not part of the linker D trust chain from the trust anchor to the issuer to the workload cert, and it's just some other certificate that your application needs to use to head off to a browser or whatever, your golden, you don't need to have that signed by anything in particular from linker D's point of view. Your, the rest of your application may have its own demands, of course. Okay, so this is pretty much exactly the same command that we did to generate the first trust anchor. We've changed the names of the files and we've changed the not after argument. We do those, we can pay attention to validity. Now it's not valid after October 23rd of 2022, which is considerably more reasonable. We can then use our new trust anchor to generate a new issuer certificate. And once again, we've changed the file names, we've changed the trust anchors file names for the signature. And again, we've changed not after. If we inspect that one, if we inspect that one with the wrong file name, then we'll still see the 2050 expiration date if we were using the correct thing, we would find that it was out on August 31st. So we're just gonna pretend. Okay, yeah. They say to never precede a software demo with any comment more predictive than watch this. So this is why. All right, so talking about rotating certificates, we've just added a bunch of security by having these things expire in two weeks instead of 30 years almost. This means that we have to pay a lot more attention to rotating them. We must rotate certificates before they expire. Repeat after me, I will rotate my certificates before they expire. This is very, very important. The reason it's important is that if your certificate actually expires, you hit downtime, don't do that. You will also need to rotate certificates in order to replace them. In this case, we wanna replace one with a stupidly long expiration with a much shorter, more secure expiration. You might also, maybe the corporate ID people, the corporate security people have handed down your shiny new trust anchor from on high and you need to replace it. Same thing. When you talk about rotating a certificate, that really means replacing it. The actual process you have to follow varies depending on which certificate you're talking about rotating. If you rotate the trust anchor, that's the hardest one because you have to rotate the trust anchor and then rotate all the issuers and then let them do all the LEAF certificates. If you wanna do the issuer certificates, you can do them without touching the trust anchor and the workload or LEAF certificates. Honestly, you shouldn't have to worry about it all because that's part of Lincardy's job. So the first thing that you always wanna do here is to check the state of the world. And here, once again, Lincardy check comes into play. This is the dash dash proxy argument to tell it that you're gonna focus specifically on the proxy layer, the data plane. You might see all green check marks like we've been seeing in the past, but you might also see some other things. You can see this thing complaining about how you have trust anchors expiring soon. Basically, Lincardy will complain any time a certificate is gonna expire in the next two months. This is a warning because you may very well have time to fix this without incurring downtime. All you have to do is rotate the certificate before you actually get to the expiration. If you actually do get to the expiration, this is the one you'll see, you don't wanna see this. This is a pain. Your cluster is probably not working anymore and you have to go through and replace all the certificates starting from the trust anchor all the way down. So again, don't get there, keep an eye on your trust anchor. You can also see these for the issuer certs. Again, expired issuer cert, still gonna mean downtime, but it's at least easier to replace that one than it is to have to do the entire chain. So let's take a look at what our cluster looks like. Ours is okay because we didn't generate its certificates long enough to go, actually because it's still using the old ones. We haven't rotated it yet, sorry, nevermind. So all green across the board. All right, now we're actually gonna rotate the trust anchor. This is one you have to be careful about. There are multiple steps here. All of the steps that I'm showing you are actually necessary. So, you know, read through it, take it easy. Don't try to rush it and it'll be fine. It's also another reason to keep a careful eye on your trust anchors because yeah, if you get this process wrong, you cause downtime. So we would start by generating a new trust anchor. We're not actually gonna run that in this particular case because a few minutes ago, we generated a 60-day trust anchor that we can use. The next thing that has to happen is we have to take the new trust anchor and we have to bundle it with the old trust anchor. Bundling is literally taking the two certificates, taking their public halves and stuffing them into the same file. The reason we need to do this is that right now, at this very moment, the issuer certificates in my cluster are signed by my old trust anchor. If I just swap out the trust anchor entirely, my cluster will stop working because the issuers are no longer valid because they no longer have a valid signature. If we bundle the two of them together, then we can use issuer certs signed by either one and that lets us bootstrap the whole thing so that we can rotate the trust anchor, then rotate the issuer certs and not take any downtime in the middle of it. And there's another comment or question from the audience. Is there a way to generate a trust anchor from the certificates? It needs to be a, okay, let me back that up. You can use pretty much any CA certificate as your trust anchor, whether it's self-signed or not. Linkerdee actually isn't gonna care if somebody else signed it that Linkerdee doesn't know about. That'll work fine, but it is very important going back to those constraints. It must be a CA cert, it must be allowed to sign certificates and CRLs and it must have a path length of at least one. Preferably exactly one. But if those constraints are true, you can just stuff it in as your trust anchor and then follow the rest of this rotation and it'll be fine. So hopefully that answers that. In the meantime, we're actually going to read the current trust anchor out of the cluster. In this particular case, we already have the old one lying around on the file system, but if we were doing this for real, maybe it's been a little while, maybe you're not quite sure. This is the way that you can make absolutely certain that you're working with the correct original trust anchor. Remember, certificates are only public. It's perfectly safe for them to be in the cert and it's perfectly safe to pull them off and write them to the file system. So we'll grab that, we can inspect that just to show that it really is a certificate and it's really the old one. This is the last time I'm gonna bother inspecting one though. It is a certificate and yes, it is our trust anchor that expires in 2060. So that should be the correct one. If we wanted to be really certain, we could check these against stuff we'd saved ahead of time, but yeah, not gonna worry about that right now. Okay, so we're gonna use step certificate bundle to bundle the certs together. We're gonna take the original trust anchor and our new trust anchor and we're gonna save them into something called bundle.crt. There we go. And then we will deploy that bundle as our new trust anchor back into the cluster. Cross your fingers. Hey, there we go. All right. If we were doing this in the real world, this would be the point where you would need to restart all of your mesh workloads because if you were to run LinkerD check dash proxy right now before restarting any of the mesh workloads, it would complain that some of your mesh stuff no longer had the correct trust bundle, but everything would still be running. So once your restarts are done, you should be able to run this and then see all green things. We actually don't see all green things because in the time that I've spent talking about this, my exactly two months certificate slipped to being expiring in slightly less than 60 days. So that's why we see this warning. That's okay. But yeah, it's a risk of doing this sort of stuff. Okay. Actually, before we go there, I don't see any other questions. All right. So we've rotated the trust anchor. Now we have to rotate the certificate. The issuer certificates rather. And again, we would do this by generating a new identity certificate signed by the new trust anchor. And like last time, since we already did this, we're not gonna worry about actually doing that. But once we have done that, once we've generated that in this case, we'll use our one-week identity issuer and we can take that and apply it by running liquidity upgrade again. So there we go with that. At this point, everything will be using the trust anchors that have been updated. Nothing should still be using the old trust anchor so it's safe to get rid of it. So we do exactly the same thing as rotating our trust anchor, but instead of feeding it the bundle, we feed it only the new certificate. And we can take a look again and yeah, all right, fine. Now we're seeing warnings for both the issuer and for the trust anchor because yeah, they both happen in less than 60 days, but we know about that, so that's okay. Okay, now's a good time to ask anything else that comes to mind. The next step is gonna be to tear down liquidity. Actually, you know what? I lied. The next step is for me to go through and read this bit. This is actually, you know, this is relevant, right? Every time you have to replace the trust anchor, you're gonna have to replace the issuer afterwards and there are three distinct steps to all this one. First, you have to update with the bundle that has both the old trust anchor and the new trust anchor. If you don't do this, downtime. Second, cycle out the new issuers. You don't need to do a bundle or anything like that, but you do have to rotate the, to the trust anchor bundle first. Last, clean up, go through and get rid of the old trust anchor entirely, switch just to the new one. All right, and now is the point where we talk about rotating expired certificates. If you let your certificate expire, any of these certs expire. If you manage to avoid downtime, it's blind luck. So don't let yourself get into that situation. You basically, if only the issuer certificate has expired, it's not that bad. You just do a rotation because if your trust anchor is still valid and you still have its private key, you can just generate a new issuer certificate signed by the trust anchor, rotate it in and all will be well. If your trust anchor has expired, then you have to go and rebuild the entire world. So you have to build a new trust anchor, use it to build a new identity certificate and at least you get to avoid bundling the two trust anchors together because if you try to bundle in an invalid certificate, it doesn't gain you anything. But yeah, this is a situation you don't really wanna get into. And in fact, we're not gonna demo this because some of you might be using clusters that aren't just demo clusters and I don't wanna assign something that's gonna cause downtime in those in a live stream. This you can do for homework. There's a guide in the repo. Yeah, and if you have questions, then join the Slack and hit me up. Okay, now let's get on to a slightly more practical production setup where we're using cert manager to automate a bunch of this stuff. Basically, as you've probably seen by now, managing certificates completely by hand is kind of a pain. You have to pay a lot of attention to the expiration times. You have to constantly have the keys available to work with. It's just kind of a pain. This is landing us in the realm of PKI, public key infrastructure, which is pretty much all about trying to automate this stuff and trying to put policies around it and make it so that the humans can pay less attention to the certificates and more of their attention on getting their job done. A reasonably good PKI setup will be able to handle both bootstrapping identity in a brand new cluster and automatically rotating certificates for you. Cert manager can do both. Since we're kind of pushing on time a little bit, we're only gonna show the bootstrap step here. Setting up automatic rotation once you have identity bootstrapped is both a good learning exercise and usually pretty easy. So we're gonna leave that one as homework. As far as bootstrapping, we will start by throwing LinkerD away because we're gonna reinstall it differently than we did when we were doing everything manually. LinkerD cert management with JetStacks. How do I compare LinkerD cert management with JetStacks cert manager? I am about to be installing JetStacks cert manager. LinkerD's certificate management, they're complementary. They're not competitors. LinkerD's job really centers on those workflow certificates. Sorry, the workload certificates out at the leaves of the graph so that LinkerD can make good decisions about what it should allow or not allow when one workload is talking to another. Cert manager is all about the problem of managing rotation of certificates, which LinkerD, other than the workload certs, LinkerD doesn't worry about that at all. It assumes that you're gonna want to set something up where that's their core thing, that's their core competency. Rather than competing with it, we'd rather just work with them. Does that answer the question? Hopefully, if not, let me know. And in fact, as mentioned, I am going to install cert manager right now. So we make sure we have the repo and then these are pretty much straight out of the quick start for cert manager, where we just go through and install the cert manager home chart. Next time I do this, I'll set up some background music for times like this. It'd be good to have some elevator music or jazz. Once it's done. Yeah, pretty old-life jazz, it'd be good. We're also gonna install Jetstack's trust tool. Basically, it makes it easier for us to tell Jetstack exactly how we want the trust hierarchy to be set up within the cluster. All right, now, this next bit gets a little bit odd. We're gonna go look at a bunch of the CRDs underneath this configuration. We're not gonna talk too much about them because they're gonna be pretty much the same for most LinkerD installations. But the core of what we're going on here is that there's a config map called LinkerD eternity trust routes that must have the trust anchor in it. And there, we need a secret that will hold the issuer certificate later. So let's take a look at some CRDs to make cert manager do that. To set up the trust anchor, we're gonna ignore a lot of this, but critical bit, remember that common name, root.linkerd.cluster.local, that's gotta be there. And we're gonna use cert manager to create a certificate and then store it in this secret. You know what I said, secret not config manager will get back to, sorry, not config map. We'll get back to that in a minute. To set up, well, actually we'll go ahead and apply that one. And we can see that cert manager has in fact made us a secret for the trust anchor. Don't mount this in a pod or in general, don't give it any way to go through and be visible from inside your certificate. Let cert manager worry about that. This actually contains the trust routes secret key because cert manager can't do its thing without the trust anchor's secret key. But you really, really don't want things poking at that when they don't need to. This is in the cert manager namespace. So, you know, set up our back so that other people can't look at it and don't mount it in a pod or anything crazy like that. So next up, the identity issuer. And again, you know, raw CRDs where here's the name, the name that we used earlier, identity.linkerd.cluster.local. And we're gonna go through and set that up. That put a secret in the linkerd namespace, linkerd identity issuer. So that's good. Sorry, that secret name was given up here. And then the last bit is that we have to go through and set things up to link this back to the config map and chain everything together in terms of the trust hierarchy. I'm not really gonna break this down. It's kind of more of the same where you'll see these things and you'll see that we're looking at these secrets and all that. So we'll apply that. And we have the linkerd identity trust routes config map that we need for all of this stuff to work. So now we can install linkerd using the cert manager for these certificates. So we have to start again with linkerd install dash dash CRDs because we're using linkerd 2.12. And then you'll note that this is quite a bit different instead of passing it the certificates, we're telling it we want it to use, basically we want it to use external management of the certificates instead. So we'll let that go. And as always, run the check. If this turns all green, we will know that cert manager actually did its thing and set up a shiny new trust anchor and issue assert and all that for us. And if it doesn't turn green, we'll know that something went wrong. Again, this is another great time to ask questions if anybody has any. Yeah, ask way people, we only have about 10 ish minutes left. So if you have any questions in mind to send them to the chat so that we can get them answered. Yeah. And everything is all green, that's good. You do see a warning up here because since cert manager is messing with all of this for us cert manager can use very, very short certificate expirations, it just, it's a program, not a human. So it's going to be rotating every 24 hours. And this is a good thing for security. You know, you do need to deal with the fact that you are gonna see a warning about that. So when all of a sudden done, if you've done this correctly and if cert manager is functioning, you will end up with the linker identity trust roots secret in the cert manager namespace. That's the one that you wanna make sure other people can't get at. Other workloads should not be able to see this. You will see the linker identity issue or secret in the linkerty namespace which is being created by cert manager. And you will also see the linkerty identity trust roots config map in the linkerty namespace. If you don't see those, something is wrong and linkerty check should tell you about it but that's what you'll end up with. So there you have it. That's automated cert bootstrapping like we already saw since there's a program managing the issue or certs it can do them every day will really increase security there. The only downside of doing it this way exactly the way we did it is that the private key for the trust anchor actually is in the cluster right now. You have to protect that a better way of doing this would be setting it up so that it's stored off cluster and all that but that's very much out of the scope for this presentation. So with that in mind, do we have other questions? Not so far, but usually at least in the past when we start saying last call then there's gonna be a bit more so we can wait a bit for those but yeah, eight minutes left and then we have to finish. So there's only a time for you questions. And here we come, here we go. The first one. Complement Istio or does Istio have its own way to work with cert manager? I would find it a little strange to run Istio and Likerti in the same cluster. They're kind of doing the same job but doing it in very, very different ways. So not gonna lie, I don't actually know about Istio and cert manager interacting but yeah, sorry, that's not, that doesn't feel like a great answer but that's what I got. A bit more on the trust anchor the issuer certs and other certs. Let me see if I can't get that slide back, Klee. Perfect and no worries if there was no info on the first question. I guess people can also reach out to you via Likerti Slack or Twitter or some other places if they really, really need some answers in the future as well. But yeah, we see Tahir saying no problem, you're right. So, we'll answer. So, this is the overview of the relationship between the trust anchor and the issuer certs and all that, where the trust anchor is the root of everything. The trust anchor signs the issuer certificates which sign workloads certificates which are attached to workloads. And the trust anchor- Wait, you're sharing something? Oh yeah, I am sharing something. Yeah, so we can see if we get to that end. There we go. Okay, sorry about that. Yeah, so the trust anchor certificate is the root of all trust for Likerti. It signs the issuer certificates, the issuer certificates sign workloads certificates, workloads certificates are attached to workloads to protect communications between them. The trust anchor, it doesn't really matter who signed the trust anchor, but it must be allowed to sign other certificates. And in fact, it must have a path link of at least one, because it needs to sign the issuer certificates which sign the workloads certificates. And in X509 speak, that's zero, one, and then you get to a leaf. Hopefully that answers the question. Perfect, and then there was a question from Muaythairo about is there a possibility of getting a prerecorded form of this video tutorial? This precise tutorial that we have recorded today will be available as on demand, like a recorded version in the cloud native live playlist in CSEF YouTube. It should be available pretty much immediately after this ends. So no worries, this precise one, you can rewatch any time you want from the YouTube. But if there's any other material that's taking, yeah, if there's any other material they should check out, obviously, then you can shout them out. The repo for the workshop is probably the best place to go for resources. There's not video in there, but there's a lot of pros that might be a lot easier to read than following along on the video. Perfect, and then we had a comment. Thank you so much, you guys are the best. Thank you so much to you as well. My thank you. Yeah, you are the best as well to be asking as well. Yeah, and then we have a link. Would you have the link for the repo or the slide with the repo link? I have the slide with the repo. Here we go. All right, so there's waiting for the screen share. Can't see, yeah, yeah. Let's wait for the backend, perfect. There we go. Yeah, so everything for this workshop is in the buoyant IO service mesh academy repo on GitHub, specifically the L5D certificate management directory. There's a bunch of stuff already there. I'm probably gonna update a couple of things later today, but it's okay. Go ahead and look. And I should have had a link to the Slack as well. Let me just make sure I'm about to give the right thing. Yes. Here's the link for the Slack. I'm not sure I can post. Yeah, you will have to put it to the private chat and then, yeah, then we'll be here. So I've just posted the link, just posted the link for the Slack. There we go. If you go there and join up the linkerty Slack, you can find me there. I'm Flynn. Perfect. Easy to remember now after this. Yeah. Hopefully. Okay, any final, we have three minutes left to any final questions in the audience or any final comments from you, Flynn. Oh, this one that we saw in the teaser. I had a final, yeah, one final comment. Kube Crash, it's happening before KubeCon. There's some cool stuff going on there by all means go over and take a look at that. We'd love to see you there. It's virtual, it's free, easy to make. And also, we've mentioned the Service Mesh Academy a couple of times. We do a bunch of stuff like this that's focused on linkerty. Also, love to see you there. Perfect. Great final notes for today. But thank you so much to you as well as all the questions. Thank you, much appreciated. Yeah, so thank you. And thanks to the audience for showing up and paying attention. Yeah, exactly. That's the important part. So thank you everyone for joining the latest episode of Cloud Creative Live. It was really great to have a session about certificate management of the linkerty. And thank you everyone for the interaction and thank you everyone for the questions. And as always, we bring you the latest cloud saving code every Wednesday. And in the coming weeks, we have more great sessions coming up. So thank you for joining us today and we will see you in the coming weeks. Thank you.