 Hi, welcome to the Key Factor DevOps Workshop. Today we're going to cover securing your containers using EJBCA. My name is Eric Meisel, I'm the Field CTO here at Key Factor and my good colleague Sven. Hello, Sven Reiler, the International PKM and of Mystery. I love that title by the way, right? Sven is an awesome PKI guy. So if you ever need help with PKI, this is your guy. So for today, we're going to cover a few things. We're going to talk about how to deploy EJBCA in a container. We're going to talk about how to issue certificates to an Istio service mesh using the CSR API. We're going to show you how to use a Vault plugin so that you don't have to change code and you can actually get certificates from EJBCA through existing mechanisms where you get searched from Vault today. And then we're going to finally finish up using CERT Manager with this new external issuer that we built with EJBCA. So that's our agenda. So kicking it off, the first workshop, we're going to talk about how to deploy EJBCA in a container. And I'm going to hand it over to my good friend Sven to walk us through it. All right, thank you. So we will first go into the image here that we've got and we've got our Jupyter Notebook open. And what we've got in the first section here is deploying EJBCA using Helm. So Anton, one of our professional service engineer put together a Helm chart for us. And what we have are the different sections that really are kind of important to this, the first one being the environment and variables that we deploy EJBCA with. We've got the JDBC URL and the password set to connect to it. We're going to set audit log to true. So that means our audit logs will actually be stored in the database. And then we're also going to do the TLS setup enabled equal simple, which means for this tutorial, we can just log into EJBCA without having an MTLS CERT at the time. And then we can lock that down later. The next section we have is the image area, where we're going to be pulling the container from Docker Hub. And you'll see that new tag of the 8.1.0 beta. I got us a special community release with the new APIs to use for this tutorial. And then the next section is how do we access EJBCA? So with that, we'll be using the proxy AJP along with an ingress. And what that will give us is the ability to access inside the cluster for containers running in there, as well as us accessing outside of the cluster. That way you can run EJBCA in the Kubernetes cluster and have both the external and internal access, all from the same instance of EJBCA. And lastly, we've got the resources section, which specifies how much CPU and memory we'll be allocating to our container. So you can tweak this around to your own settings in your environment. All right, so with that, we'll run our command to deploy with the Helm chart using the values YAML that we just went over. So we'll click the Play button, and that deploys. All right, the next step is where we're going to deploy a patchy HTTBD, which we're using as a reverse proxy inside of our Kubernetes cluster. And this is serving a couple purposes. Number one, it's how any container running internal will go to access EJBCA through it, as well as pulling any revocations, such as CRL or OCSP, or if you wanted to download the CA certificate inside the Kubernetes cluster. And then the ingress that we added through the Helm will also go in through the reverse proxy as well. So in the configuration, we have another ingress that we actually added outside of the Helm to route the OCSP, CRLs, and the AIA access into this Apache proxy, too. So it's kind of multifaceted with what we're doing. And you can take a look at the YAML file in there, too, if you want to see all of the magic of what I created. So with that, let's go and run that guy. So we will click the Play button on step two, and it launches. And then we can give that a few seconds like this. Because the next command we're going to run will actually go and check the logs of the EJBCA container. And we're looking for a string that says the health check now reports the application status at EJBCA public web health check, EJBCA health is up. And when we see that message in the logs, that's telling us that EJBCA is ready to use now and the container is up and running. So let's go ahead and give this a try and see what we get back. All right, so it did come back with a new health stamp. So at this point, we are ready to begin using EJBCA, Eric. Dude, that took forever. I'm sorry. I'm sorry. I don't know if you guys are about you guys, but taking that long to deploy a CA, that's crazy. So we've got EJBCA up. Now we're ready to go, right? So magic happened, downloaded some bits, got it on the box. So now what we're going to do is we're going to start to look at how we can use EJBCA to serve a bunch of DevOps use cases, right? So whether I want to get searched into Istio service mesh or through a vault or through another way, we're going to talk about those. So first use case up is using the Kubernetes CSR API. Now what's interesting about this is this API has been around, seems like for a while, but it's been experimental for ever. Yeah, and so I don't know. It doesn't feel like it's ever going to launch, but yet I think people are using it. We have customers using it. So the nice part about it is what happens is that when you plug this in, it's going to redirect to call EJBCA. So part of this workshop is to show you guys how to do these different things using EJBCA, where others might be using an inbuilt CA like Citadel or Hashicorp or some others. Or even the built-in Kubernetes CA in the cluster. Exactly. And they're finding that that's breaking policy at their org. And they're like, hey, we need to go to something that's standard, that is compliant. Exactly. So this allows you to deploy without changing code. So we're going to be able to use this plug-in to redirect to call EJBCA. So without further ado, we'll turn it over to Sven and he's going to show you how to enable this CSR API and to go get search from EJBCA, that one we just built. All right, let's go over and check that out. So now we're in the section of Issue Certificates to Service Mesh. So we're using the EJBCA CSR signer integration, which is in our GitHub. And you can see how it's a blue text, so it's a link. So you click on that and it'll take you over to the GitHub repo so you can see the code. We've deployed this with the Helm chart that's in there. And the thing to point out with this is in the EJBCA settings here in that little YAML text snippet we've got, a one-day ephemeral search. So what that's telling us is that with the ephemeral search we've configured, we're not going to be keeping those search in the EJBCA database because they're one day. And with that lifetime like that, that just obviously fill up too much and we don't really need that. This is more think session based at that point because with service meshes things are coming and going up and down. We're always issuing you certificates so we don't need a long lifetime on those. And we've got the values YAML on the OS here in the path and check that out. So now we're in the section of issue certificates to service mesh. So we're using the EJBCA CSR signer integration which is in our GitHub. And you can see the little how it's a blue text so it's a link so you click on that and it'll take you over to the GitHub repo so you can see the code. We've deployed this with the Helm chart that's in there, namespace. So our namespace is where we're going to organize everything like logically. It's kind of like you can think of like an Azure tenant I guess if you're in the cloud world or if you're already familiar with namespaces and Kubernetes, you get it. All right, so then we'll go down and create, we'll deploy the book info app from Istio. And it's using an app that's already there, yeah? Yep, we're using it there. So when we deploy that, that instantly kicks off the certificate issue and so all the containers spinning up in the mesh will be generating CSRs and the CSR API will send that over to EJBCA to give us a certificate. And so as that kicks off, we'll add a gateway in. So we'll click the little play button again. And now we're going to go and tail the logs with the container. So we'll click the play on that one. There we go. So it loads and we'll scroll through the text. So we want to look in here for this begin certificate request, that's good. What is that begin certificate request? That means we're generating certificate requests from the containers booting up and those will be sent over to EJBCA to get signed. So the next thing we want to look at is a begin certificate that we actually got a cert back and that'll be a bigger blob with the cert contents. So it comes through and ooh, there we go. We got some certificates too. So I mean, it's that easy to get certificates with CSR API and a service mesh like that. So you didn't have to change any code, right? No, we didn't have to do anything. Just enable the plugin. Yep. And remember, the policy is driven from the EJBCA side because that's where our cert profile and identity profile are configured with this integration. I think that's the important thing to touch on. So depending on the audience who's watching, developers, I'm a dev by heart, security is not my thing. It was not my thing. It is now, right? But the thing is, with a lot of these tools that we use, Kubernetes, Vault, any of these tools, you can request a cert of any longevity you want, any key length you want, you can do whatever you want and that's breaking policy. And so what we're offering here with EJBCA is policies enforced by the CA. The PKI where it should be. Exactly, exactly. Perfect. All right. Next up. Next up. So a lot of people use hashi. Nothing wrong with hashi, right? Great tool for secrets. Great tool. It actually has an inbuilt CA. Part of the problems there with the hashi is there's lack of missed two requirements. A lot of the InfoSec teams struggle with it because, again, going back to what we talked about, policy enforcement, getting any kind of cert you want, the keys aren't in an HSM. There's different things there that are break InfoSec policy rules. So we built a plug-in that allows us to redirect the calls to EJBCA. So much like you just saw with the CSR API, where it redirects to EJBCA, same thing here. So if you're using an application or your service mesh or your Kubernetes cluster is calling into hashi to get a cert, they'll actually redirect it to EJBCA, where we work the magic. So again, handing it back over to my good buddies fan. Let's go through that. Let's go take a peek-a-boo. So what we have here, we scroll down to the issue certificates through the hash corp vault with EJBCA. So thing to note is, I think like two or three years ago, we released a vault plug-in. And this is the new vault plug-in that we did. As you may or may not know from using the old vault plug-in, it only worked with a single instance of vault. You couldn't do it or could not use it with Vault HA. And with this one in the tutorial I did, I really wanted to take advantage to show now that HA works. So what we did is we did a three-node Vault HA cluster with containers deploying with the Vault Helm chart. We also used the init container in the Vault Helm chart to download and compile the EJBCA vault plug-in, which then puts it just in a volume out within each of the three vault instances. And then we configured that to issue a certificate type of a five-day, I believe. Yes, client-off-cert and a role for that. The big one with this is you can issue multiple certificate profiles and entity profile types with this plug-in, too, by just creating different roles. So it's quite scalable at that point. And from what I understand, too, you can even assert different CAs in this as well. And it's very robust compared to what we had before. It's something that I've been wanting for. And I finally got a dev to help me out with that. And now it's there for us to use. And as well, the link for that repo is the blue text in here. So you can link over to that to be able to go and check that out. So with that, the first thing we need to do is unlock Vault because the VM just came up. Vault is locked. So we'll have to go through all three instances to unlock those. So let's do that real quick by clicking the little Play button to unlock the first instance. And we can see how it goes from true, true to false sealed. So now this one's unlocked. And then we'll do our second instance. And that's true, true. And then the last one should be false. And then we'll do the last instance. All right, so we've got Vault unlocked. So with Vault unlocked, the next thing we need to do is to log into Vault. So we'll get a token to be able to log in. And that will actually let us be into issuing certs. So we'll go ahead and click the Play button to do a log in here. All right, we've got a token back to log in. And we've got a little note in there. If you actually want to see the docs for all the settings of a role, you can run the command just above step five there to check that out. So one of the things, too, as we talked about is, in this instance, for the lab, we're logging straight into Vault and running commands. Whereas you guys would be running, your application would be calling into Vault. Or whatever's calling into Vault, we don't have an app tied into it. So we're just running command. Exactly, yeah. The idea was really, I wanted to package this up so we could focus as much on just issuing certs to this so you guys can go and play with it on your own after this. I guess we should give a little more context. So we have an OVA here that you guys will make available that you can run either in VMware or in VirtualBox that you can go through all the stuff. See the setup, see the configuration, run it, test it, try it, and then you can be able to port it over to your world very easily. Exactly, all right. So we'll click the play button to do our first cert. And there we go, we got a cert back. You can see all the payload of it. And this one was with the PEM bundle, so it's a big blob. So the next format we'll try is just a PEM. So we can click the play button here. And we get another cert there with the CA chain in it, and the issuing CA. And then lastly, we'll go through and do one without the root chain included in that bundle that it's returning. So we'll click the play button for that. And then we get a cert back for that. So really, really straightforward to just be able to get certs right through the vault. And so since we did these and these ones were not ephemeral, let's go take a look in RAweb. So we'll pop up in a new tab here and click on the RAweb link and accept a little cert coming in. Clearly, this is a lab. Yeah, right. And then we'll go to certificates and we can go look at those. So we can see here that we'll go by our cert type to have a little more fun with that. So we want the five-day these guys right here. And we can see the ones we just issued through Vault right here. And view those and take a look at how they came out. Yeah. So that easy to issue certs with Vault and need to be CA now. That's awesome. So the nice part, as you mentioned it earlier, is that you can change. You can have multiple different kinds of certs, serving different use cases by role. But again, no code changes. There's configuration, quick deploy. You don't have to change anything and now your devs can still get their certs. But they come from EJBCA. And what's even cooler in my world is that it was all in containers too. Yeah, we weren't even using VMs. Yeah. Let's say soon next level. Love it, love it. All right, on to the last one, last and final. So cert manager is evolving, right? I think it's becoming sort of the defecto. Exactly, yeah. I mean, we're seeing it with all our customer engagements where we're going and talking to them. And they're like, hey, we're really interested in deploying EJBCA in a container. And my first question is, do you cert manager? Yes. And then I'm writing Eric, hey, Eric, I really need some dev time to get that cert manager integration done. What's nice about cert manager is it used to be just ingress egress with Acme. But now it serves MTLS into the cluster, which is phenomenal. And the ingress egress in Nomura Acme, which is magic for I think anyone. I think the Acme stuff is what people are a little bit tired of. So now we have a provider service that Sven's going to walk us through that, again, another plug-in sits next to cert manager as an issuer service and allows us to get certs from EJBCA. So let's take a look. All right, let's go. So now we're in the issue certificates through cert manager with EJBCA section. So with this, cert manager added the ability to add an external issuer for integrations to come in quicker. So we're actively working to get a pull request and GitHub to get EJBCA added as an external issuer. And on the VM here, we do have the container that was built for this and the source code of the CRDs that are added for this to make this all work. And in the YAML blob here, we see an issuer. So with this plug-in and integration, you can use an issuer or a cluster issuer. Both of them are supported with this. So just like any other cert manager integration we're the same way. You can have issuers for different CAs. And again, the whole a la carte of creating issuers for your different cert profiles, identity profiles for your certificate needs. And the best part is, again, all of the cert policies driven on EJBCA. No configuration for policy stuff on the Kubernetes side with the integration here. I think for all you devs out there that are butting heads with your implicit team, this solves a lot of problems. Exactly. And just makes it so easy because you've already got cert manager in place. So all you're doing is adding another issuer into the Kubernetes cluster to be able to take advantage of. Dare I say a better issuer? Maybe. One ring. That's right. All right. So the first thing we're going to do is just manually deploy a YAML file to issue a test TLS cert. So if you were actually deploying an application and setting that up to get the TLS cert, you could handle that automatically. But we're just really keeping the simple to get the most out of just how easy it is to just issue certs like this. So we'll click Play here to load that. And so that creates our object. Then the next thing we want to go do is we want to go describe that object so we can see that it was created. So here we've got it described. And we can see that it was done and is added like that to issue the certificate. So then we'll go grab the secret names. OK. So we've got our secret here. And it's the secret MTLS test 01. And then lastly, let's go actually look at that secret to see that it's populated. And now we can see that it is populated. And we've actually got data in there for the TLS cert and the key. And yes, it's small. But that's because we're using elliptic keys. So they're going to be nice and small cert sizes. And since that did issue a five-day cert, we can also go back over to our other tab here and check that out in the RE web and just go back to Overview. So the nice things about EGBCA is we can set up certs to be ephemeral. And we can set them up to not be. And so you get a lot of flexibility. So based off your use case, so if it was ingress egress, those wouldn't be ephemeral. But typically within the cluster MTLS would be. So you get both kinds. And there we go. Perfect. Awesome. All right. So what did we learn? How easy was it to play EGBCA in a container? Exactly. That was crazy easy. I can't believe how quickly that came up. And then we talked a lot about getting certs into Kubernetes using CSR API, using cert manager. Looked at how you can put in the HashiCorp vault eGBCA plugin to make it easy for you to take what you were doing and redirect eGBCA. And so for more information, here's some QR codes that you can get information on where all this magic is, right? So that's all out there for you guys. Hope this was useful. It was a great being here, Sven. Rockstar, this is an amazing OVA. We'll make all the stuff available for you guys to grab from a GitHub that will be part of the package that gets delivered when this goes live. Thanks for joining. Yep, thanks, guys.