 OK, I make that five minutes too, so let's get started. Quick admin beforehand, we've heard a few alarms going off. It's because the French government are testing the alarm system apparently. You can't really prepare for that with the talk, so we'll see how that goes. If your phone goes off, it's OK. Please just get it if you can. It's no problem. So let's get into the actual talk. Welcome to cryptographically signed swag. Thank you all for coming here. Bonjour. I hope you've had a good KubeCon. The scheduling means that obviously you've come here after the project pavilion is closed. So we're talking about the booth, which you cannot go to. That's a shame, but what we can do is tell you what happened at the booth and maybe it's all Lake City in November. We'll be able to see you there and you can get a certificate there. So we are three third manager maintainers. My name is Ash, this is Tim and this is my arm. We all happen to work at Venify and we spend at least part of our time maintaining Sir Manager. Quick show of hands. Who has heard of Sir Manager? Yeah, that's what we like to see. Who runs Sir Manager? Who runs Sir Manager in production? Yeah, great. So you're all experts already. You don't need to listen. Just in case anyone doesn't know, Sir Manager is what we like to say is the best way to get certificates in Kubernetes. So you're used to writing YAML to get a pod. Sir Manager adds the ability to write YAML to get a certificate that you can use for your website, say, or for interpod communication with encryption. And the key really is that once you've told Sir Manager how to get a certificate, you can actually, like it already knows, so it can renew the certificate as well and it won't expire and cause an outage and that's always good. Of course, we support various different ways of getting your certificate. That includes ACME, let's encrypt, that's what most people use. But we can also do private PKI, which is kind of what we'll be talking about today, in a way. A Hashicot Vol and a bunch of other, including Venify. There's a picture here and you might recognize some of the faces on it, certainly my shiny head. This is what the booth looked like and this is what we'll be talking about today. And this is the result of that. So this is a stamped Sir Manager certificate that was issued by Sir Manager and then printed out in physical form. I can't show you the other side because it has a QR code on it which would let you download my certificate. So I'm being really careful to just show you the front here. We're gonna be talking about the journey that we went on here and how this works behind the scenes and how we've improved it here for Paris. Here's some big numbers. There's the standard slide. 20 million monthly chart downloads, we think. It's all good stuff. A big milestone to point out there is that our graduation process has started now. We were hoping to get it done in time for Paris but not quite there, we're still incubating. Hopefully by Salt Lake we'll get there. Of course, there's other big numbers here if you enjoy big numbers. So there's a brief introduction. The best way to start now is to tell you a bit about the history of Sir Manager and how we got to here where I'm holding a physical Sir Manager certificate in front of you all. So I'll pass over to Tim. Awesome, thanks Ash. All right, so how did we get to these impressive numbers, right? I'll try to kind of explain that to you by guiding you through the history of Sir Manager. So big disclaimer here. I wasn't there for all of it because I'm still pretty young. So to give you an idea, in 2016 I was only 17 years old. So that's actually where the journey started. So Sir Manager actually comes from this project called Kube Lego, which is basically a controller that was created by this company called Chatstack. And basically what the controller did is it looked at ingress resources and at the annotations on those resources and based on those annotations it got, or it created certificates. And those certificates it got from Let's Encrypt which is still actually an exact use case that we still support with Sir Manager. And this was the predecessor of Sir Manager. So that's kind of where the idea was born. Then one year later, the actual term Sir Manager was coined and the project was created. The first release was made and this release was already a bit more cloud native. So included here was, for example, the notion of like a certificate resource and some issuer resources. And so that was in 2017. And this API of these resources has been improving for the next couple of years. And then in 2020, the first stable release of Sir Manager was created, the V1 release. And we're actually still supporting that V1 release. So that's like still the version that we are building on. So if you have a version or if you have like resources from then they will still work with the Sir Manager from today. In the same year, we also started our CNCF journey. And that basically meant that we became a CNCF sandbox project. Then two years later, we actually got to the incubating state which is already a great achievement. And since then we have been working hard on getting to the next stage which is basically to become a CNCF graduated project, right? And one major milestone towards that goal is to complete a security audit, a CNCF sponsored security audit. And I'm very happy to announce here at KubeCon that we actually completed that. So we have passed the security audit and one day we actually published a post where you can find out more information about the security audits. So I would like to thank the team from Adalogix who actually performed the audit. And of course, my colleagues in the Sir Manager team to fix all the issues that were found and to make Sir Manager a better product. So the next step of course is to get to the graduation state. And that will be hopefully for later this year. Okay, so I mentioned before about the Sir Manager booth and the things we've been doing there. This has been going on for a while now. We're going to talk later about what's changed but first you need to know what's happening behind the scenes. And for that we've got the primary expert on the matter. Myel. Hi everyone. So I'm going to talk about specifically what how these came to be. And the first thing I want to know is has anyone in this room gotten something like this? Oh, that's, yes, that's a lot. So these certificates, these cards are made of on the front. You can see a receipt of a 659 certificate. And on the back you have a QR code that contains a certificate. And on the front you have this nice stamp. The idea was to somehow mix the physical and digital into one card. So digital obviously the X599 certificate and the physical is the seal that you usually see. You would see 200 years ago or more. I'll be honest, when I created this demo in Valencia 2022 it was this stamping idea which frankly let's say that's the most fun of this demo. It came from a person called Marcia. Marcia was a maintainer of the Sub Manager Project and she had this great idea for the V1 of Sub Manager to do thank you cards with actual walks. And with this melting pot and candle and she would pour it on the card and there send it to all the contributors. I'll be honest, I tried at home, I bought everything and then I contacted the CNCF event team who told me that candles aren't allowed at QCon. So I bought this glue gun stat which isn't ideal either because it stinks like the plastic when you keep the glue gun turned on all the time it really does this weird smell. Now, we have done this demo over four different QCons that was Valencia. The first one we made over 225 printed certificates. That was awesome because everybody was talking about it on Twitter when it still existed. And then we continued in Detroit, Amsterdam and Chicago with over 800 certificate printed. The moment I remember the most in these, while staffing the booth was this time in Chicago where over 20 people were in line waiting to get a printed certificate and we were like rushing, printing, stamping and I was crazy and I was, yeah, that's super nice to see especially for a project pavilion booth. Now to the specifics of what you would see on the booth if you came to the booth, I'll tell you exactly what was on the table. There was this printer and a Raspberry Pi. That's essentially what is the demo. On the Raspberry we have Kubernetes running with K3S. With Soutmanager obviously we are using Soutmanager and a UI running also on the Pi and a Kubernetes controller. Where you would fill in your email on the UI then it would create the UI if we didn't go with create a certificate resource in Kubernetes which would then be issued by Soutmanager using a self-signed root CA on the Pi, not really secure but whatever. And the user would click prints and the controller would pick up the certificate and send it over USB to the printer. And obviously we would stamp it. Sometimes we would miss and the stamp would be weird but that's what we usually say is you get a unique certificate. That's every single time. And there is even one person for some reason I stamped on the back also. So on the front and on the back. One weird or crazy or whatever choice I made in Valencia because I could was to write the Kubernetes controller in Bash using Kube Kotlin and that's it. Kube Kotlin and JQ. It's not ideal because editing this file as any Bash script is terrible. Error handling is really hard. You have error handling and printing the errors is just weird. But I think it's a great way of getting started. If you don't know Go, you definitely know Kube Kotlin and Kube Kotlin get watch, the flag watch can get you started with reconciling objects in Kubernetes and you can do it right from your terminal. You can see an example on the printer search GitHub repository if you want to see more. Finally, finally about these QR codes on the back. In Valencia and all the other Kube cons except this one I chose to have the QR code contain the certificate but not the private G. The whole certificate was printed on the QR code that was great because we didn't need to store anything but the only thing that you could do with this QR code was to show the certificates in the browser and that was it. But Ashley had an idea to do something with the private G and that's what he's going to present now. Thank you, Mayor. So yeah, we noticed while we were doing this booth before a lot of people were saying what can I actually do with this apart from take it home as a souvenir and we wanted to do something about that and add a way for people to actually use them. So we love the souvenirs and there's nothing wrong with them but we wanted to do a bit more and we wanted specifically to be able to use these certificates for TLS because that's why everyone's using CertManager. That's why you use it in production with all those people that had their hands up. We also just wanted to have a bit of fun with it and it was great fun. Here is an example of one of the certs that we actually issued. This one's mine, this is the one that I held up here and didn't show you the QR code for. So what's new with them? Well, the first thing is you can get the private key now and with the private key, you can use the certificate. So the QR codes have been updated and actually it links to the same page that you would get if you were at the booth. That page allows you to download a tar ball with the certificate in it, with the private key in it and with a little bonus script which I'll talk about later. This completes the loop. It means that you can actually use the certificate that you get. Another little advantage is ECDSA. If there are any other certificate nerds in the audience, I'm a big ECDSA fan. Our issuing chain always use ECDSA so the signatures were always using elliptic curves but the printed certificates actually had RSA public keys. Now we're using ECDSA everywhere which has the advantage that the certs are smaller and faster and I get to say elliptic curves which is always cool. As an example of that, like a visual illustration, this is the difference in size between an RSA key which is on the left and an ECDSA key which is on the right. You can actually see how much smaller they are in a textual representation at least. I've talked about the chain there. Another thing that we changed is actually the chain itself. So before, we used to sell the booth essentially from scratch for every KubeCon. We've now issued a stable root certificate. I've been telling people for the whole conference that it's 100 years long. It'll expire in 100 years time. I didn't actually check that but I'm pretty sure it's true. In any case, it expires in a, it's the next generation's problem. What we're doing now is generating new intermediate certificates in each location. So we can use the same chain and we can actually keep on expanding this demo to future KubeCons. So in Salt Lake City, we can do something else fun and who knows what that will be, where future ideas are always welcome in the cert manager community. It gives us a lot of flexibility. But the real fun thing is that there's not much point in just giving you access to the private key. You need to be able to use it. So what we did is created a guest book and it's super simple. It's just an endpoint that you can post to and you can only post if you have a certificate that we issued at the booth. So either you've got a certificate from us or you somehow stole our root certificate and I hope you didn't do that. So I mentioned a script earlier that comes in the table. That script is actually a quick way of being able to sign the guest book. You can customize the message if you like. I know there's some people here that have done that. I got to be first because I wrote it. And you see my message up at the top there. Obviously we've obfuscated the email addresses for obvious reasons, but it's a really cool way of completing the circle and going back and sort of taking what you got at the booth and actually interacting more, learning more and seeing how this works. If we can do this with certificates in person, there's no reason you can't do this for machines in your production clusters and do mutual TLS everywhere. Of course, we've also got Easter coming up very soon and we've added some Easter eggs as well because while we were changing things, we couldn't resist. We've added them to both the certificates that you've got and the chain itself. Of course, we set the country code to France. We couldn't resist that. Obviously in Salt Lake City, we'll set it to US. We set the organization in the certificates CNTF and the organizational unit to cert manager because of course the cert subject is kind of a bit pointless now because it's not really conveying any useful information for doing like website security, but it is a great place to store Easter eggs for booth demos. We actually used to put the email address that you gave us into the subject itself. The modern way of doing things with TLS is to use the subject alt name extension, which we now use. We put your email address in there and we put your name into the subject. And so that's a few little things that we've changed. Basically, it's from start to end, a demo of how you can use cert manager, obviously all running on cert manager and it's also still a souvenir and still a lot of fun. The only shame really is that we didn't do this sooner because I know people have been to every KubeCon in the past and got a cert manager certificate and they're collecting them like infinity stones. So it would be nice if they'd still be able to use those old ones, but that's in the past. So I've talked about what we've done here, but there is a serious point to this. It's not just souvenirs. What we're gonna do now is I'll hand over to Tim who's gonna tell you what we can actually learn from this demo and show you why it's sort of a good representation of how cert manager works in the real world. All right, yes, indeed. So please pay attention to this section because this is a section that you have to tell your boss about when you try to explain that you listen to talk about physical certificates. So basically this is what the demo looks like. The issuance flow looks like for the demo. And what we can learn from this basically is that once you understand how this issuance flow works, this issuance flow kind of reoccurs in every scenario where you're using cert manager. So it reoccurs when you're using cert manager, your production cluster, even if it's like a thousand nodes and a thousand clusters. All right, so to quickly go over the demo again. First, you have the user interface. There you enter your name, your email address. Those are like the details that you want in the final certificate, right? So you specify them in the user interface. What actually happens behind the scenes is that this information is passed into the certificate resource. So the certificate resource is a Kubernetes resource that cert manager is actively looking at. And once it sees that such a certificate resource is created, it will actually create a certificate request resource, which is like a one-time resource that's used to basically tell an issuer, actually, I would like to get a certificate for this certificate request. After that is done, the issuer creates the certificate, it signs it, and then cert manager puts the result in a secret resource. So this secret resource is then used to print a QR code on the back. And that's kind of the issuance flow that reappears everywhere with cert manager. There are, however, two ways to kind of deviate from this issuance flow. And the first way to do that is to change the way you request the certificate. So by making these alterations to this issuance flow, you can actually make cert manager work better for your specific use case, for your specific scenario, for your specific business, basically. So in order to request the certificate in a different way than just creating a certificate resource, you can, for example, annotate an ingress resource. And if you remember, that's actually what I talked about with the Kube Lego controller. So this is the original use case that's still supported. You add the annotations on the ingress, cert manager creates a certificate resource for you, and then the certificate is issued, and put in a secret. The same applies to like gateway resource, which are this new cool API. An alternative that we have for certificate resources and secrets is a CSI driver. So we have like a cert manager CSI driver. We also have like a cert manager CSI driver, Spiffy, which is another cool buzz term. But basically what this does, it lets you through the CSI properties. It lets you specify what you want in the certificate, like DNS names, subject, and so on. And then instead of creating a secret, it actually directly puts the certificate in a volume for you. So you can mount the volume in your container. And this is like a no secret solution. So not sure if you've ever heard of like no secrets policy in your company. I'm sure if you've heard about it, you will remember. But basically there's a solution for that. Another way to request certificates is through integrations with service meshes. One service meshes in particular that we support is Istio. So we have this solution called Istio CSR, which is basically an integration between cert manager and Istio. And it lets Istio directly request certificates from cert manager, and it puts those certificates directly in the sidecarts that are used by Istio. Right, and then the second way to kind of change this issuance flow, and to make it fit better for your use case, is by changing the issuer integration. So in our demo, we actually requested a certificate from a CA issuer, at least that's what we call it in cert manager. And basically what that means is that somewhere, in your Kubernetes cluster, there is actually a CA certificate that lives in a secret, together with this private key. So it's like a public private key pair that lives in a secret, and you point cert manager to that secret, and you say, okay, well, now all certificates that I request, just sign them using this certificate, the CA certificate I have in the secret. But of course, that's not really a good solution, because if you accidentally get rid of this secret, you just lost your CA, right? And if that's like a root CA, you have to rotate all your clients. So that's kind of an issue. Also, if you have multiple clusters, and you want to share a CA across multiple clusters, you could kind of copy the secrets across all these clusters, but that's also not the recommended way, I would say, since copying private key material across clusters is not ideal. So basically, I would advise you to use one of our issuer integrations. For example, we have like, for private PKI, we have like Ashkopf fault integration, Fanify integration, AWS integration, Google Cloud integration, and then for public PKI, we have this Acme solution integration. I'm sure you've all heard of it, since it's the most popular one, definitely in combination with Let's Encrypt. So this is actually an open API. You can add your own integrations for issuers, and there's a full list of these issuers, but like a lot more that are created by our community, and you can find the list on the search engine website that by just clicking on the link in this presentation. Thank you, Tim. The slides are available for download as well, if you didn't get that link. So we've talked a bit about the booth here, and we've sort of gone over how actually, this is kind of a silly demo, but really it's how every production cluster running CertManager works. And it's not just through veneers. Like I said before, this is a full CertManager demo. If we can do this for people, you can do this for machines. Like more encryption might be the right solution in your cluster, it's worth evaluating. If you're already running CertManager, it's actually very easy to start doing like expanded stuff using CertManager. It's too late to go to the booth. I'm really sad about that. If we've been on Wednesday morning, then if this talk had been on Wednesday morning, then you could all rush off and get your certificates now. You can't, sorry. But you can join our community. And we have Slack channels, and we have regular meetings, which all documents are on our website. Again, these slides are available for download. We love it when people get in touch. Our regular meetings include daily stand-ups, which are EU time zone friendly. So if you're based around here in Europe, then you can come and join in every day if you want. We also have bi-weekly meetings, which are great if you have sort of longer form questions. That lasts for an hour. I'd like to say thank you all for coming, or merci. We thought ahead this time. We know that sometimes people don't like to ask questions. There are microphones that may be behind you. So we actually printed some certificates and brought them with us. So anyone that does ask a question can still get a certificate, and these are the last ones that will be available in Paris. Like, don't all rush at once, but of course, thank you all for coming. Thank you for coming late on a Friday. You've been great. Thank you. I really thought there'd be a rush for questions. This is good. That's the question, I suppose. Yeah, I kind of get a certificate, he says. There are microphones back there if you do have any questions. I feel like an air hostess. Your nearest microphone may be behind you. Oh, we've got another one here. Yeah, hi, thank you for the demo. I already have my cert, so I have a question. Like, when generating the certificate, if we, like, generated with the same name and same email, so would it be generated, or is it like unique, or how does this work? So, you're asking about the booth specifically. Or in general, like the certificate. So, the identity of a certificate is actually a surprisingly complicated topic. I mentioned about the subject, alt names, extension and certificates. That can hold many DNS names, email addresses, IP addresses, that all works. Cert manager can generate multiple certs that have the same subject, alt names, it'll handle all that just fine. In the demo, we have a unique constraint on the email address just so that people can't spam us, but cert manager can generate anything that you want, really. It's all good. It's just, it's like certificate, it's a Kubernetes resource, right? So, you can't have two pods that are identical that with the same name, but if you have a different resource name, you can have two, that's fine. Yeah, thank you. Thank you. Do you want one? Anyone else? You can come and ask us questions down here afterwards as well. We'll still be here, of course. Oh, someone's walking over, yeah. Go, fire away. Can you hear me? Yes, we can. All right, quick question. So, I saw a paper saying that they were able, somebody in a specific country, I'm not gonna name my country, they were able to decrypt one of the certificates, the RSA 2048. So, even if I put everything using M-T-L-S, how can I protect myself against AI? Well, so they can't as far as I know. The question was about people claiming to be able to decrypt RSA 2048. That is not as far as I know possible. That will be possible if quantum computers come out that are powerful enough. There are no quantum computers today that can reliably do that. It requires factorizing very large numbers and quantum computers just can't do that yet. Some people have claimed to do it. I've not seen any actual evidence of that being the case. You're certainly correct that that is a threat that looms on the horizon. Cryptographers differ in how soon they think it will happen. I'm not concerned about it in the next 10 years, personally. I'm not a cryptographer, but I'm not concerned. But we are starting to think about that as a thing now. A certain manager in the future could well be quantum resistant. That is a thing that we could do. Certainly, it's something to keep an eye on. It's a really interesting area of research. It's something we'll look at. Quick follow-up. I was not concerned about AI a year ago. AI. So AI cannot, as far as science knows, AI cannot decrypt RSA 2048 as far as we're aware. AI might claim to be able to, because it hallucinates, but I don't think that that is a threat that I've ever encountered. I'm not too concerned by that. I think we're still secure for now. You might want to look at this series called Silicon Valley, where this actually is one of the... Oh, maybe I'm spoiling now, but this is one of the interesting parts of Silicon Valley. Yeah, maybe. We still have time, yep. See you at the mic there. Go ahead. Hi, thank you for your demo. Is SERT Manager a solution suitable for multi-cluster environments? It is, yeah. So there's not really any orchestration beyond you installing into multiple clusters. What a lot of people do is when they set up Kubernetes, they install SERT Manager at the same time. They treat it like it's a standard part of the cluster, if you like. It's not what we like to think that it almost is. That will work if you sort of just apply install SERT Manager across many clusters. There's not really any orchestration beyond that, but it does work just fine in multi-cluster environments. There might be some hiccups with things like DNS challenges, but mostly it just works in air quotes. Also, like I mentioned before with the issuer integrations, you can have one issuer that you share across multiple clusters. So then basically all your clusters, they share one CA. And that's really how you do multi-cluster setup because then you trust this one CA and you can kind of communicate with any of the clusters. It doesn't really matter because they all share this one trust domain. Excellent point. I think we've still got time, yeah. Down there. Hi, Al's Rushing. Thank you. Hi. Hi. Is it possible to use SERT Manager to manage SSH certificates currently? The question was, is it possible to manage SSH certificates with SERT Manager? Unfortunately, the SERT in SERT Manager only applies to X509 certificates. We don't do SSH certificates. That would be a very cool project. SSH certificates are awesome, but I find the documentation is very challenging with them. So it could do with an SSH SERT Manager. It's something that would be super cool to talk about at one of our meetings if you ever wanted to come along. Unfortunately, we don't have that today, but it sounds cool. I think, yeah, two minutes. So if you're quick, we've got one last question. I just can confirm that to be successful around SERT Manager in several clusters with SERT Encrypt and issuing the certificate for the same domain. They obviously conflict for a challenge, but then they retry and eventually get this. My question is, can I build a CA authority, some kind of stack? I don't wanna have the private key even for intermediate certificate on every cluster because intermediate certificates are, I cannot limit intermediate certificate to only issues certificates for subdomains. So I would wanna have some secure cluster where I will run certificate authority and other clusters use this one as a shooter. So the question was about having one trusted cluster that stores the CA and then having other clusters consume that. That's possible with issuers like Venify and Vault. With a CA, you're sure it's not possible. It's not designed to work like that. I'll talk to you more about that after this because I think we're out of time. Thank you again. Merci beaucoup.