 All right, we have Jackie lined up now. So please join me and welcome Jackie to the stage to talk about your wish is over command, declarative certificate management in a service mesh. Thank you. I'm a software engineer at Microsoft and I'm a member of the open service mesh team or OSM for short. Our team wanted to simplify certificate management within the service mesh for users like mesh operators and developers. So in this talk, I'll share some terminology, the problems that led us to seek out a solution, the solution that we implemented, and an example of root certificate rotation with our solution. So let's level set on some common terms that I'll be using throughout the presentation. So for certificate provider, I'm referring to the private key infrastructure or whatever the service mesh is communicating with in order to issue certificates. So this could be cert manager or hashy court vault. For certificate authority, I'm specifically referring to the chain of trust for any given leaf certificate. And then certificate management specifically is the management of certificates within the service mesh. So OSM chose to focus on the problem of certificate management because we recognize the importance of certificates within the service mesh. One of the main abilities of a service mesh is to help enforce authenticated and encrypted communications between services. So this enables users to help strive to achieve a zero trust environment and focus on their business applications rather than on tooling to support certificate management. So despite the importance of certificates, we found that managing those certificates once they're in the mesh is not simple. Performing critical operations requires operators to have deep technical knowledge. For example, rotating a root certificate is often a manual and air prone process. And then specifically for OSM, migrating certificate providers, maybe from our internal self-signed CA to cert manager to better meet business needs is not a standardized process. So failures in any of these certificate management processes have high impact and can result in downtime. So in order to simplify certificate management, OSM took a declarative approach. So the main goal was to move the complexity from the users onto the mesh. The user is responsible for specifying their desired state of how certificates in the mesh should be used and we'll go into how to do this next. And the mesh is responsible for making that desired state a reality. So by coordinating provider migrations and certificate authority rotations, a little preview. All right, so how does OSM support users communicating this desired state? So we created a custom resource or mesh root certificate MRC, which you can see on the left of the slide to help centralize certificate management specifications. So this resource is either created by the mesh on startup using user provided settings on install or it's created by a mesh operator prior to install. The MRC is created to initiate and facilitate rotation but we'll come back to this later. So what can the user describe with this resource? So the provider field in the MRC contains information about the certificate provider the mesh will communicate with to issue certificates. So OSM, like I mentioned before, currently supports sort manager, HashiCorpVolt and our internal self-signed CA which is Tresor. So the role field is the key aspect of what makes the MRC resource declarative. It describes how the user intends for the certificate authority, which is accessed by the certificate provider to be used within the service mesh. So by inspecting this field, the service mesh or MRC controller can determine how certificates should be signed and which certificate authorities should be distributed for validation. Okay, so the simplicity of the role field is best demonstrated in an example of root certificate rotation or a certificate provider. So let's imagine the CA in MRC one on the top has been compromised. So the operator has already created MRC two and it has an inactive role and this is gonna be containing the CA two that will be rotated in in place of CA one. So since MRC one is still active, service A and service B will have certificates signed by CA one and their validation context will only contain CA one and just backing up a little bit. So the active role, this indicates that the certificate authority should be used to sign certificates and should be added to the validation context. A passive role indicates that the certificate authority should only be used for validation and then an inactive role indicates that it should no longer be used within the mesh and this is either in this case when you've created an MRC and you have yet to rotate it in or you've already rotated the certificate authority out. So that's when an inactive indicates. So and then again, notice as we move through this rotation that the green check mark remains throughout and this indicates that the connection between workload A and workload B is never disrupted. Okay, so to initiate the rotation process from CA one to CA two, you can see in the red that the operator updated the intent from passive or from inactive to passive and they're just, this is with a just a simple kubectl apply command. All right, so the update is picked up by the MRC controller, which inspects the state of MRC one and MRC two. So since MRC two is in a passive state, CA two will be added to the validation context of both workloads. MRC controller takes on the responsibility of updating these validation contexts in the mesh and updating Envoy, which is our sidecar without any downtime. And because of how Envoy is configured, these trust chains can be updated and reconfigured without downtime. Okay, so to progress to the next stage or of the rotation, the new MRC twos role is updated from passive to active and the old MRC ones role is updated from active to passive. This change signals that CA two, instead of CA one should now be used to sign certificates. So since MRC ones role is passive, CA one will be included only in the validation context and will no longer be used to sign certificates. Okay, again, the MRC controller picks up on this change and enrolls and begins distributing certificates signed by CA two. However, during this transition period, some services will still have certificates signed by CA one and if they've yet to receive the update and others will still have the certificate signed by the new CA, CA two. Also, although these workloads have different certificate authorities, they're still able to validate each other because the validation context contain both trust chains. So again, throughout this entire rotation, the connectivity between the two services was never interrupted. Okay, so OSM sought out a simplified solution to certificate management experience to reduce manual and air prone processes for high impact scenarios. The mesh certificate is a declarative resource that gives users control of certificates in the mesh and puts the responsibility of updating those certificates and distributing them on the service mesh itself. So, and then as we saw in the rotation example, if we're migrating from one CA to another, the MRC resource allows you to easily make that migration by simply updating the role field. And the pattern in which these roles are updated, which I didn't mention, is enforced by a validating web hook, this pattern ensures that connectivity between the two services is never interrupted as we saw with the green check mark. And the OSM team is committed to providing an excellent certificate management experience and we are working to expand the capabilities of this mesh certificate resource to further improve the operator experience and yeah, we hope that this is a simplified experience for certificate management. So we went through the terminology, we shared the problem, our solution, and we demonstrated our solution in an example of root certificate rotation. So if you're interested in joining more, check out OSM Slack with the very helpful ID and also check out our GitHub and I'll be at the OSM booth all week at KubeCon. So I'd love to chat. Thank you.