 Helo. OK. That sounds correct. We're going to get started. So, hello everyone. Thank you for coming. I'm Ash. I'll be starting us off. And as you can see, I need a new picture on this slide. I've had a bit of a haircut since then. Welcome to Cert Manager in five levels of difficulty. Today we're going to be taking you through Cert Manager in five levels of difficulty. You've got me. You've got Myel here and Tim. We're all at Venify. We're all Cert Manager maintainers, which is good because of the content matter. To start us off, we've got the typical slide where we show you a load of numbers. We like all of these numbers. One of my favorite numbers on this is one million plus daily downloads. The problem with that number is we don't actually know if it's true. Well, we know it's not true actually. We know it's bigger than that. We don't know how big because our container images are hosted on key.io or quay. I don't know how you're meant to pronounce it. And every time we try and look at the analytics, we actually crash the analytics page. We get a 500 error. So if anyone is from Red Hat and can tell us how many downloads we have, we'd love to know. But I think with the second biggest project, we'd love to know. The other important thing on this slide obviously is that we're a CNCF incubating project. That's why we're here today. That's why you're here today, I assume at least in part. We're really hoping at the moment that we're going to be hitting graduated status soon. Obviously, that's the same level that Kubernetes itself is at. It'll be awesome to get there. We're really hoping we'll get there before KubeCon in Paris next year. The security audit for that is currently ongoing, which is awesome. Watch this space. Hopefully, the next time we're giving a talk, we'll be a graduated project. So here are the five levels of difficulty that we're going to be talking about. I'm going to do a little bit of audience interaction, not much. If you think you're at one of these levels, just raise your hand and then put your hand down if you no longer understand what we're talking about. So if you've used CertManager at all, you've probably done something like ingress and gateway annotations. If you've ever issued a certificate using CertManager, put your hand up. Lots of hands. That's great. Have you used a certificate resource? Yeah, private PKI. Anyone doing private PKI, especially if you're using CertManager? Yeah, a few hands there. CSI drivers, approval policy. Okay, a lot less hands. Yeah, this is the real stuff. This is the meat of the talk now. Level five, has anyone written a custom issuer or a plug-in? There is a hand up at the back. That's awesome. We'll get to all that stuff. First, we need to make sure we're all on the same page. This is the bit we do every CertManager talk where we talk a little bit about what TLS is, how certificates work. So everyone's starting from the same bit. The idea is you have some application, like you're a bank, and you need a certificate to prove that you own a domain. So a customer, a client connects to your service and you hand them a certificate that says, yeah, I am this bank and this is the proof. If everything goes wrong, then the client will see your connection is not private thing and they probably won't do banking with you. If everything goes right, hopefully they see this and it will all be working. This means that you had, well, the client had some way of trusting your certificate before they actually got it. They have a trust store on their device, which is what's on this laptop, on my phone, on your phone. And they, using a chain of certificates, a chain of cryptographic signatures, if you want to be technical and fancy, your client device was able to verify that the certificate was actually issued to the bank and you trust that because you trust the root CA, we say, or a certificate authority. Essentially, you want to get certificates for your services and Kubernetes and CertManager helps you do that. Another thing that's worth pointing out is that this isn't just one way. So, we talked about a bank that is running a server that gives you a certificate. They can also then say, hey, and you can give me a certificate and you can send back yours, so that Manager can do that as well and we'll learn a bit more about that as we go further on in the talk. But that's enough from me, you've heard enough from me, so I will pass over the mic to Tim. Thanks a lot. So, we will actually look at the simplest way to use CertManager. For that, we will look at the main CertManager use case. That's namely provisioning a certificate for your application. So, basically what we're doing, we allow your application to request a certificate through CertManager, then your certificate authority will respond with a certificate and that will be provisioned by CertManager in a location that is accessible by the application. So, the simplest way to do this, to do this flow, is by doing this for ingress resources. So, in this example, I have an ingress resource and we are serving example.com slash bar here. So, if we go to that URL, we see that we are serving this service here that returns the current time. But you might have also noticed that this is served over a non-secure connection. So, if we try to go to the secure variant of this DNS name, of this endpoint, we actually get an error. So, this is the thing that Ash explained. We get an error saying that this connection is not private. And this is because, by default, ingress will actually use like a cell sign certificate and we haven't actually configured proper certificates for this endpoint. But, thanks to CertManager, it's very easy to do this. So, what we'll do is we will add the ingress resource, we will add an annotation that points to a CertManager issuer. So, in this case, the Let's Encrypt Production issuer and CertManager will detect that this ingress exists, that it has this annotation, OK, for this ingress I have to do something. Additionally, we will add a TLS section in the ingress resource and this TLS section specifies where it has to find. It's a secret that contains the TLS certificate and the private key material. So, normally what you would do is you would configure this and then you create the secrets that contain this information. However, in this case, of course, it doesn't exist because we haven't created it. But the great thing here is because we added the annotation, CertManager will automatically generate a secret for you. So, after we configure this, we apply this ingress and we refresh the website. We see that the endpoint is now searched over a secure connection. So, this is actually the easiest way to use CertManager. A summary of this is you basically annotate your ingress behind the screen or behind the scenes. What's happening is that CertManager will detect the annotation. It will create a certificate resource, a certificate resource that points to the secrets that we specified in the TLS section. That certificate resource then in itself will be reconciled by CertManager. Certificate Request resources will be created so that the certificate request resources will be detected by the issuer, which then returns the signed certificate, which is then placed in the certificate together with the private key, so that it can be used and picked up by the ingress controller. Now, Myl will talk more about the secret resource. No, the certificate resource. Thank you, Tim. We have seen that the ingress resource, in itself, if you add one annotation, you will get a Let's Encrypt certificate for free. But sometimes you cannot do that. Sometimes the ingress annotation system won't let you do all the customization you want. This is going to be level two. Level two is creating your own certificate request. To give you an example of what I mean by customizing your Xivo9 certificate, I'll be using a client certificate example. Radis, the Radis server, if you install it with Helm, it will require you to give a secret. The secret is expected to be a server certificate. On the other hand, I chose to go with a spring boot application that connects to this Radis database, because it's a Java application which has a tiny quirk to it. In this configuration, we will have two certificates, one Radis server cert, and cert manager will create a secret for it, and that's the secret that will be used and that you will have configured in your Helm top. On the other side, you have the certificate client, the Radis client cert, itself creating a secret loaded into the deployment and the spring boot deployment. Now, to dig into why the certificate resource is interesting to you, I'd like to show a few things on the certificate resource itself. Now, the secret name, the one, the secret that cert manager will create a private key and store it into, as well as storing the sign certificate, sign certificate, and then we have the usages, that's also something you cannot do with the inquiry presentations, so you will want to use a certificate instead for that. On the left side, you have the server auth usage and then on the other side, you have the client auth. Finally, the small perk I was talking about, sometimes your application will need a special encoding for the certificate and by default, cert manager will store the sign certificate as a perm-encoded string. Here, because it's a Java app, it's PKCS12 and that's something you can do with the certificate resource and something you cannot do with the ingress resource. So that's the certificate resource and finally, it's looked a bit at the secret that was created by cert manager. We have, again, since I was talking about Java, P12 that resulted, because we set in the previous slide, I was showing something about PKCS12 and most importantly, we have the chain of trust, the certificate chain of the cryptographic signatures. I made it. So tls.tst, that's the thing that is served by the client and by the server when authenticating and the CA.CRT the CA.CRT field that I didn't highlight here, which is a bit special because we can't, we recommend not to use that for distributing trust. So don't use the CA.CRT and you'll see how to use this properly in the next level. So to summarize, in level two, we learned that the certificate resource makes it possible to customize your X509 certificates and most, for example, in the case of Java for having something compatible with your application and lets you do client certificates and I'm now handing over to Ash to talk about trust. Thank you very much. So we've just seen two levels which tell us how we can configure what certificates we get, right? You do something in your cluster be that an annotation or you write some YAML for a certificate and you get a certificate that hopefully matches what you requested as close as possible. But what about if you need a bit more control over the issu inside of it? So we've given you examples with let's encrypt, right? That nothing wrong with let's encrypt at all. I love it. I can't recommend it highly enough, but it's not the right use, the right tool for every job. I'm not going to go into the detail of this slide because it's quite heavy and there's a lot of stuff here, but in summary we would call let's encrypt a public issuer, right? That's because you already trust it on all your devices in this room. I would assume. That's great. That means it's easy to set up because your clients will trust it already. But it does mean that it comes with other burdens if you like. So rate limits are the big one with let's encrypt. It's also a little bit more complicated to issue because you need to prove that you own the domain before let's encrypt will give you a cert. So what you can actually do is use private PKI which is sometimes called self-sign certs but certificate nerds don't like that because self-sign means a different thing there. Private certs, private PKI gives you complete control over everything. That means if you want to add some weird thing into your certificates you totally can do that. You can add whatever domain you like so if you want to make a certificate for example.com or google.com or whatever you can totally do that, maybe you shouldn't but you can. There is obviously a trade-off with that which is that private PKI is more that you have to do. You have to manage this CA certificate you've got another one that you need to manage and I would like to highlight that rotating those certificates is complicated and needs planning for. It's not as trivial as just rotating a certificate like cert manager would do for you in the background. Rotating a route is actually quite difficult. So as a summary of that we've seen how cert manager can provision for you this certificate chain that you need to do TLS stuff be that a server certificate or a client certificate or whatever. The question we're asking is well how do we trust the CA certificate that issued those and for your devices as I've said before you already trust them. It's provided by your operating system or whatever but it's there. In Kubernetes we think the answer is trust manager. So trust manager is like cert manager but for trust. We're not very inventive with names. So trust manager essentially takes the CA certificates that you need and makes it easy to use them in Kubernetes. The way it does that is through YAML like everything else in Kubernetes. We add a new resource which we call a bundle. It's actually cluster scoped. It's a bundle of resources which could be secrets in your cluster, config maps or that default trust bundle, the same one that's on your phone probably. It stitches those things together and then outputs it into a config map that you can use in your cluster. Or a secret now as a version 0.7 which released recently. The idea then is that you take that output and you mount it into your part or whatever it is that you're running. You can then trust anything. So it makes it really easy to get started with private BKI. I think it's really cool. I love playing with this stuff. A lot of organizations do this in production and it is incredibly powerful to have the ability to do whatever you like with these certificates. You can put a lot of useful stuff in there that you might not imagine. It's not for everyone. I think from level 3 onwards it's fair to say that we might be encountering stuff but it is good to have the option and we certainly provide those. I'm going to pass over again in a sec but I'd like to say at level 2, Myel is here today talking but Tim and I were preparing yesterday for him not to be here because he was in bed all day ill yesterday. Shout out to Myel for actually being able to do this. I will let him bask in the glow of that while I pass over to Tim for level 4. Thanks lots. We talked about how to obtain a certificate using an annotated Ingress and how that results in a certificate resource but also how we can directly create certificate resources. Important to note here is that a certificate resource is tightly coupled to a secret resource where finally the result of the issuance will be placed so that that secret can be mounted somewhere. I'm going to pass over to myel. I'm going to pass over to myel. M returning user registration error1 gyda it3 The default ribbon can be placed so that that secret can be mounted somewhere. An alternative method that you can use to request certificates is a CSI mount, ac roeddaf yn unrhyw gwirionedd ac roeddaf yn uned gan addysg hwnnw ben yn gwneud haniyniadol. Mae anodd o'r ffordd i'w amrwy o'r adegion artennig ac eich ffordd i'r adegion arwaith y bydd y ffordd hynny. Mae anodd arwain o ffordd i'r adegion eich rhai yn dda, ac mae hynny ymwysig oherwydd mae'r adegion arweinysig. Yn ceisio am y gynnwys i'r adegion, mae anodd yn cyrfryddio'r adegion. Mae hanfodl yn debyg o'r adegion, The CSI driver will actually provision for us a certificate at the specified path that we mount this volume in, and the certificate will have the specified DNS names, and it will be issued, of course, using this issuer. So this is a great way to obtain certificates without using secret resources. So this way it's more secure. It is also tightly coupled to the POTS resource, so basically when you have like five different POTS, they will all have five different certificates. Also the private key material does not leave the nodes, so it's kept in memory in the nodes, and then the memory is mounted as a temporary volume in the container. So it's really a more secure solution. The private key material is not kept in the cluster state, but there's also kind of a disadvantage in case your issuance or like issuing a certificate is pretty costly for you. This is probably not the method that you should use because this is like an ephemeral method, so every time your POTS restarts or it is scheduled on one of the nodes, you'll have to issue a new certificate. Then another extension that we provide with CertManager is a approval policy. So if you look at the issuance flow of the certificate request resource, the first step in there is actually to make a approval decision on the certificate request. So we basically decide whether a created certificate request is approved or not, and we only proceed with the issuance when we see a certificate request that is approved. By default CertManager actually comes with a approve all approver, which approves all the certificate requests for that target issuer that is entry in CertManager. You can disable this default approver and you can replace it, for example, with the approval policy add-on that we created. And this add-on is actually a more advanced policy system where you can specify using these certificate request policies, what certificate requests are allowed and what certificate requests are denied. So in this example we actually say that all certificate requests have to have a common name with the value example.com, have to have the address names that match the specified values, and then we also specify the selector, which basically selects what certificate request this policy applies to. So there is still a bit of work here that we are doing, so we are adding support for CL. We are trying to make this much more intuitive, because I think we can agree that the UI currently isn't super great. But generally the idea of this approval system is that you let your application teams create certificate requests freely, and then you specify through a policy what certificate requests can actually be allowed and what certificate requests are denied. Okay, let's move on to the level five. It's, I guess, the most difficult level, but that's not the way. Level five is how to extend CertManager. Because we're using Kubernetes, the Kubernetes API makes it very easy to extend things. The first extension point we have is the issuer. And if you are a company that provides certificates to your users, signed certificate requests, then you will probably have thought about creating your own issuer for that measure. And by the way, we have created a new library called issuerlib to help you in the process of creating an issuer, because historically it was very hard to build one because you needed to know everything about Kubernetes controllers and the certificate requests, lifecycle, and all that. And this is all simplified, hopefully, with this library. The second extension point is the DNS01 solver extension. If you are, again, a DNS provider and you want your users to be able to solve DNS01 challenges through CertManager via your DNS, you will want to look at this webhook example and to provide your user to do that. And finally, the two last extension points is we talked about the approval policy project previously. But you can go way beyond that. You can build your own approval workflow. You could even approve your certificate request from your phone if you want it. It is totally first-class, but in that case, you will need to look at the approval policy code base because we don't have any other example of such an approver. And finally, the CSI driver is also a place where things can be extended. There is the CSI lib that allows you to build your own CSI driver. There is one example of such a specialized CSI driver. That's the CSI driver SPIFI that issues specifically SPIFI identities instead of any type of certificate. So that's it for the extensions and level 5, and I'll hand over to my hash. So let's finish this off. I'm very aware that we've just dumped quite a lot of information in that talk. It's very information-dense, a lot of different projects we've mentioned. Fortunately, you don't have to remember it all. This is being recorded anyway. Aside from that, anything we've just talked about should be documented on our website, cert-manager.io. So I'd like to highlight that before I go any further. Another thing I'd like to say is we're going to have time for questions, so now's the time to start thinking of them while I wrap it up. We'd really love if you could come along to our cert manager booth if you have not already. I recognise a lot of faces in the crowd that have actually already been, but you can come again. This is a picture of our booth in Detroit last year, but our one today is actually in a much better location. It's really nice. It's really great to have people coming along and talking to us, and the fun thing here, which you can actually just see in the picture, is we're actually printing real live certificates, X509 certificates that you can take home as a souvenir. Come get yours. They've valid for 30 years, so it should be good. We even have an actual wax stamp of the cert manager logo that you'll get on it, so please do come along and get yours. A lot of fun will be here until the end of the conference, so come get that. I'll ask questions about cert manager and talk about stuff. That's cool too, but a lot of people come for the certificates and that's okay. The other thing I would say, and I've said this to a lot of the people who've come to the booth, is we have meetings. If you're based in North America, our daily stand-ups are going to be while you're asleep, or at least I hope you'll be asleep because it will be really early for you. But every couple of weeks we have a North America friendly meeting where people can come along, ask questions, the agenda's open so you can add in your thing and come talk about it. It's a really good way to move things forward if you've raised a PR, or if you've got questions, or you can get started in contributing. We've done a lot of work as part of the graduating stuff in defining different levels of how you can get involved. We've probably got a maintainership level for you if you want to get involved in the project or get involved with Open Source. Slack is always available. We have a channel on Kubernetes Slack. You can always come and drop in there. KubeCon asked me to add this QR code for feedback. We'd love to have your feedback. We'd also love if you could stop by our colleague Sacha if you're here in person. He's back there in the corner. He's doing a survey on Kubernetes and he'd love to talk to you as well. With all that said, we have seven minutes for questions. If anyone has anything burning, please do feel free to come to the mic and ask us. Now is a great time to do it and I've immortalized forever. Oh, we've got our first contestant. Sorry, trying to get here quick. It meant you had the slide on adding a new issuer. What issuer has come out of the box? How does the issuer side of this work? It's interacting with a root CA outside of Kubernetes, correct? Yes, exactly that, yes. The issuers that come out of the box are Acmi, Venify, HashiCorp Vault, and a built-in issuer that will do private PKI inside your cluster. There's tons of external issuers. Too many for me to list here that have already been made. The idea is basically a wrapper around any API which can sign certificate signing requests. So we take the encoded request, send it off to the API, get back something. Usually that works pretty well as it happens. The big ones are like AWS private CA. They have an API for this. It's just a wrapper around that. And so the manager does the plumbing in the middle to make sure it all works. Thank you. Thanks very much. A question for that trust manager. How do you guys handle the idiosyncrasies between the trust stores for different distros when you're actually mounting them into the pods? Is that just up to the user to deal with that? Or do you guys have a solution there? I passed over the mic on a trust manager question. I shouldn't have done that. So we don't. We just create a config map and you mount that wherever you like. We'd be super interested in solving that problem, but it's incredibly hard to do. I believe that there's work going on upstream. There's a new cluster trust bundle resource that will be coming out. And trust manager will support that. So we'll write that new resource. I believe Kubernetes are going to take a stab at this. But yeah, we kind of just as an open source project with limited resources, we've kind of said like writing the config map is enough and then you'll have to work out the rest of it. To answer that question generally and solve the problem generally, it's just too hard for us at the moment. Gotcha. Yeah. Thanks very much. Hello. Good afternoon. I've been using your product for nearly three years. So thanks. I wonder I've been using with the certificate. I saw the CSI over there in the presentation. Where is the secret actually going? Because like you said, you're not going to have the secret anymore, right? So that's the first question. The second question will be, do you guys have any idea or any plans to have some sort of like integration with Azure Key Vault? So we could push, for example, a certificate. To Azure Key Vault. Thanks. So let's maybe start with the first question. Basically with the CSI driver. So CSI lip is the library used to create the communities at the certain manager CSI driver and the certain manager CSI driver spiffy. So what you as an unsusable use probably is CSI driver. So certain manager CSI driver. That's why you search in the GitHub. Basically the idea here is that there is no secret resource. So what actually happens is the CSI driver locally generates a private key. It signs the CSR with that. And then that CSR is put in the certificate request resource. That's like all the way on the right. And that certificate request resource is picked up by the issuer. So the issuer integration can be like folds. And then a science certificate is created based on that. So the science certificate is then put in the status of the certificate request resource. The CSI driver sees that and it takes that science certificate and places the science certificate next to the private key material that it generated locally. And so now the CSI driver has this information and it mounts these things in the volume that it's basically the volume is mounted in the container in the port. Could you repeat maybe your second question again? If you guys have any plans to do any Azure Key Vault integration for example? I'm not sure if there's any answer for that. I think no is the quick answer. We do hear a lot of requests for that sort of thing. It's kind of hard to do with the current architecture of cert manager. Love to chat about it more. It's maybe a booth question if you could make it along or one of the meetings. It's an interesting problem space and there's a lot to explore there. Thank you. Great. Anyone else got any questions? Now is your chance. Okay. Some people looked away awkwardly so that they didn't make eye contact and make them ask a question. So all is left then is for me to say well first feedback please. And finally, thank you all for coming. It's been great. Thanks so much.