 Today I am so excited to be here physically, you know, the first service mesh con after COVID. I'm sure, is that for any of you the first time traveling to a real conference after COVID, raise your hands. Yeah, very excited you are here. So we are going to talk about, it's a secret. Manage your workload, search in service mesh, without persisting it. Let me quickly introduce myself. Working at IBM for a very long time, I think some of you know me from IBM. I changed to work for SOLO about a year ago. So I'm working on open source at SOLO. I've been a very long time contributor to the Istio project, one of the founding members currently sits on Istio Steering and Technical Oversight Committee. I wrote a book about Istio. Istio explained to help our user get started with Istio. I'm also one of the new CNCF ambassador. And I run a Hoot livestream, which I learned about service mesh Istio envoy, EBPF, GraphQL technologies with you together with the audience. So this is ServiceMeshCon. I'm sure you've all seen some charts like this, the service mesh architecture. But today we're going to focus a little bit on the most useful benefit. I guess widely adopted reason for people to adopt a service mesh is mutual TLS. Would you agree with that? A lot of people are looking at service mesh because mutual TLS. Raise your hand if that's the reason. Yes, I know. We've heard that repetitively from our customers, right? So we're going to dive into how the certificate is distributed. We're going to dive into how to plug in your external certificate authority. And we're going to also dive into a couple of approaches to not persist to your private key on disk. So since I come from Istio, I'm going to start using Istio as an example to talk through how things work. I believe other service mesh works similarly last time when I look at them. So if you use the default profile that comes with Istio, you will have self-signed certificate, which means Istio control plan is serving as the certificate authority for you that means the certificate for each of your workloads in the service mesh. So this is typical, but it's not really what you want to run in production because typically you already have your own PKI system. You probably don't want anything self-signed that's not integrated with your existing PKI system. So the way it works is Istio has agent which runs in the same as Envoy proxy in the same cycle. And the Istio proxy, Istio agent sends the certificate signing request to Istio control plan. And then Istio control plan, who access the certificate authority, sends the certificate back to Istio agent, then back to the Envoy proxy. So that's how your Envoy proxy gets the certificates and it also handles certificate rotation before it expires. So if you take a look at self-signed root CA, essentially in the Istio system namespace, you're going to see Istio CA secrets. So that's the self-signed generated by Istio control plan at the boot time of the control plan. You will also see a config map that contains the root certificate for the self-signed root. So let's take a look at the generated Istio CA root secret. Essentially, it contains four of the files. One is the CA cert, that's the generate root cert. Also the private key. So this may be a concern for some of your organization because it essentially persists to that private key on disk as part of the SCD as a secret. And it also contains empty cert chain and root cert in this case. So if you decode the CA key file, you will see the private key. So why is this an issue? This is like the key to your house or apartment. You don't want to give that to anybody. So and the other thing I want to point out is by default, the certificate is good for 10 years. So what this means is you give somebody else the key of your house for 10 years. If somebody take it, they can access your house for free for 10 years. So certainly not something you wanted by default. Now let's talk about how the trust is distributed in this model. So when ECOD starts up, it has a namespace controller that propagates the ECOCA root cert as a config map to each of the namespace in your Kubernetes system. And when your proxy starts, proxy as a side card to your full service, it's going to mount that certificate as a config map, mount the config map cert as a volume into your proxy container. So interestingly, if you use ECOCA today, if you do a quick check by creating a namespace full, even though without annotate or label it as PsyCa injector, you would immediately see the ECOD creates the ECOCA root cert for the full namespace. So that's how the root cert is propagated to each of the namespace. Now let's talk about in this model how the trust is distributed. So ECOD agent is going to send a certificate signing request to ECOD. We talk about that's acting also as a certificate authority, which means the certificate and approve the request and means the certificate and send the signed certificate back to ECOD agent. In the meanwhile, ECOD also propagates the config map into each of the namespace, which allows the, I'm sorry, I should talk about in this sequence first. So the first sequence is actually copy the config map into each of the namespace, which allow the proxy to be able to mount the config map at the boot time. And then with the root certificate and also the service account token, the ECOD agent can send the certificate signing request, which ECOD looks check, make sure it's good, approves that, and sends the means certificate back to the ECOD agent. So the workload served by default expires in 24 hours. But that doesn't mean the service manager waits until the last minute to refresh the workload certs. In ECOD, there is a configuration as an environment variable, which we don't expect you to config, but for some reason you might. It's 0.5, which means we will rotate the certificate every 12 hours for you. So way before it expires, we will rotate for you. So now this is how the default works. And now let's take a look at how to plug in your own CA. So if you look at LinkedIn, it's essentially during install time, you can specify your CA cert, you can specify your issue key, whether using LinkedIn, CLI, or Helm. So this is not surprising. With QMA, you can also specify your key and cert as a secret and the configured in the match configuration YAML. So these are the ways you can plug in your external CA. With Istio, there are two ways. The first way is through a CA cert secret. The second way is through API called match config CA certificates. So let's take a look at them. So the first thing we want to look at is with CV certs. In this example, essentially you as an admin who is plugging the external CA, you would propulate the CA cert secret in the Istio system namespace, where you install the Istio control plane, and then you would specify the cert and key and also the cert chain if it's an intermediate cert and also the root cert. So you create a secret, populate that into the namespace, and when Istio debuts, it's going to check whether the CA certs exist. If it exists, it would use it. If it doesn't exist, it would fall back to the self-signed flow, which we just talked about. So in this example, then the rest of flow is pretty much the same as what we discussed early on with Istio CA root cert as a config map propagated to each of the namespace. When issued, though, this one still have private key persisted as a secret. So may not exactly be what you wanted. All right, so with match config, so why there are two ways to do this in Istio? The reason is we had users sometimes once multiple root cert, which is a common scenario as you transition maybe from one root cert to another root cert during transition time, or maybe you want to do a federated mesh that has different root cert. So Istio has a configurable API to allow you to do that through match configuration. So you can configure your actual root cert in match config, which Istio will propagate to Istio agents through the XDS API. So this is an example of how that looks like. Essentially, you can configure CA certificates inside of match config, which is part of the Istio operator API. And you can configure your certificate right there. So one thing is that this is compatible with Spiffy, such as Spell, so very attractive if you're using those environments. So now the next question is, can you plug in multiple CA certs for Istio control plan? The answer is yes. You can actually do that for the scenario I was mentioned earlier to federate across different Istio matches, or maybe in the root transition time frame. So in this case, you would have match config Istio actual root cert config through match config along with the CA certs as a secret where you install into your Kubernetes. So one thing I want to point out is I find that a little bit confusing is registration authority, which does approve the request. As I learned in these all these concepts, the other concept is certificate authority, which signs these requests. It typically happens after the registration authority approves the request. By default, all the flow we just went through, Istio works as RA and also CA. So that's a lot of work by Istio. So now let's take a look at another community project, Istio CSR. This is a very cool project. It's essentially Istio CSR acting as a registration authority. So the certificate signing request was sent to Istio CSR, which approves the request and forward to certificate manager, which acts as a certificate authority. Notice in this case, one thing really interesting is Istio D is not acting as CA or RA. So you have to kind of disable this function in Istio D to tell Istio, I don't want you to be my CA. I don't want you to be my CA. And I guess one biggest advantage of this approach is certificate manager is very well known. So it supports a lot of issuers. So all the issuers that certificate manager supports comes with this approach, like Vault and many of the other ones. But one thing I do want to point out is in this approach, private key is still persisted as secret. The other thing the Istio community is innovating is the Kubernetes CSR integration. So Kubernetes has a CSR API. I believe the API just become stable in the most recent Kubernetes release. But in Istio, the integration with this is still actively under development. It's still experimental. So in this mode, Istio D works as RA and the Kubernetes CA or any other customer CA who choose to implement the Kubernetes CSR API as a controller can act as CA. So this approach, when community first introduced, one of the huge benefit of this approach is it doesn't need to persist the private key in Kubernetes cluster, which is very cool. So this walks through how this approach works. So essentially, the service A of full here would send the certificate signing request to Istio D. And Istio D would act as the registration authority. It would approve the request. And it would actually generate the Kubernetes CSR as a request. So notice the CSR request here from Istio agent to Istio D. It's just normal certification request. And the Istio D to certificate authority in this case is actually Kubernetes certificate CSR request. The format is a little bit different, which I will explain shortly. So in this case, Istio D is going to approve the request and generate a Kubernetes CSR request and then forward to Kubernetes CA or other CA who implements the Kubernetes CSR controller and then eventually issues the certificate back and through Istio D is sending the certificate back to the Istio agent. So that's the flow of this flow, how it works. And when I take a detailed look of how the certificate works in this case, so you can see this is X509 certificate request that I generated using the OpenSSL key command. And this request essentially, on the right side, is a Kubernetes certificate signing request. Notice it's different than the certificate typically standard certificate request. And the request is essentially in the body of here with base 64 encoded format. So essentially what's on the left side with base 64 encoding is part of the request of the Kubernetes certificate signing request. You also have the requester here. In this case, it's the groups. And you also have whoever is the signer for this Kubernetes certificate request. So with this approach, it does require implementation of Kubernetes CSR controller. So by default, Kubernetes has a default Kubernetes CSR controller. I believe third manager also implemented Kubernetes CSR controller, even though I believe they said it's alpha. It's not production ready yet. You could also implement your customer controller. So there are API for implementing controller. So it's all plugable, which is very nice. But one question I would say is there are very limited controller out there, because the Kubernetes CSR API is very new and just become mature. So the controller has not been picked up. Like for instance, certificate manager has a controller. But even with that controller, it's having very limited issue being supported for that controller. The other thing to think through is, do you want your service machine also using a Kubernetes control plane trust domain? Because you may not want to use the same trust domain as your Kubernetes for signing your workload certificates. If so, this may not be a good approach if you choose to use the Kubernetes controller. All right. The other approach, which we're doing as solo, is not changing anything in Istio. Is we actually run a sidecar next to Istio. In this case, Istio control plane continue to act as CA and RA. And the sidecar is taking the responsibility of getting the certificate signed with Vault. So let's take a look how this approach works. So if you have multiple Kubernetes cluster, multiple IstioD, in this case, you would result a less API to your PKI system, less API cost. Because essentially, you're still using IstioD to sign your workload certificate. But the intermediate key and search would be generated from the sidecar to your PKI system, like Vault. So essentially, only the sidecar costs your PKI system. And then each of your service gets their workload certificate from IstioD. So this could potentially allow very large scale of service mesh with many clusters. One last approach I also want to highlight is what we are working in upstream, upcoming upstream, 1.14 of Istio release with spell integration. So the HPE team, shout out to their team, contribute this function in upstream. So for folks who don't know spell, it's really cool that it can attest workload and issue specific identities. By attest workload, you can attest the workload by the image name, by namespace, by part. And it's configurable. So you can tell spell how you want to attest your workload. And spell can be based on the rules and configuration to attest it for you. So in this case, the spell server would act as CA and I for you. And it can support two key management strategies. So you could config spell to say, I want to persist the private key in memory, or I want to persist the private key. I'm sorry, I don't want to persist the private key, or I persist the private key on disk. So that's configurable with spell. And if you use this mode, don't be confused that you still have Istio CA secrets, because that's not really being used, because spell is working as the CA and I, not Istio D. So just walking through real quickly how this model works. Essentially, Istio opened up a hook point to allow a UDS socket on Istio agent. So whenever Istio agent boots, it's going to detect whether the socket exists. If the socket exists, it will detect to fetch the secret, attempt to fetch the secret from the socket. And then the spell agent is going to do a testation on the workload of the service, just making sure it has the right workload with the namespace, service account, and image name. And then once that's all done, it can issue the certificate workload certificate back to the site card. So the normal flow, XDS configuration still goes through Istio agent to Istio control plan. It's just the certification bootstrap flow is going through. Because of the socket existed, it would go through the spell agent in this case. So we actually did a hoot livestream just a week ago. So if you are interested in this, check out our YouTube channel, solo IO. Check out our most recent hoot. I only have one minute. So in conclusion, we talk about many approaches. We talk about the default, how the default works. We talk about how you plug in your own CA is possible, how you plug in your own RA is also possible, how there are some of the approaches out there to allow you not persisting your private key, so such as Kubernetes CSI, control plan site card, and also the external CA integration with Envoy SDS with spell as an example. Now I think I may only have one minute for questions. Do I have a minute for questions or no? All right, there's a question there. Yeah, so Istio CSI is actually very well long in the community. One of the challenges is the persist private key is a security concern. So if that's a concern for you, you might want to see if that's really the right approach for you. That's the main thing. A couple of things I like about Istio CSI, that's super many issues. My main concern is the persisting. Yeah, the private key. Like your private key is like the key to your house. And typically, the root key is good for 10 years. So if you persist on this, if somebody grab it, they can access the whole thing for a very long time. Well, the approach, at least based on the tutorial I've gone through, you still kind of have to generate. Yes, you can provision, you can use certificate manager to create the keys and sorts. But the way Istio CSI is implemented today, it still requires you to create that in the installation directory as a secret if you ever tried that project. Yeah. Well, I hope this is helpful. I'm out of time. Thank you.