 I'm Jian Peng-he from Tetrate. I work in E-Store for the last three years. And you may know my GitHub ID as Zirin. And hello. Good afternoon, everyone. I'm Zhou Huxu from Huawei Cloud. I've been working in E-Store for the last six years. Yeah. And I'm very happy to be here for the first time to speak in person in KubeCon and E-Store today. Yeah, thank you very much to be here for the last topic. And today, our topic is how to do route search rotation with no downtime and no problem. OK, let's get started. First, look at our agenda today. First, we'll talk about some backgrounds to let you catch up with our rotation workflows later. And then we will talk about how we rotate route certificates in previous old E-Store revisions. And then we will talk about our inventive route certificate rotation solution. And then we will have a demo at last. OK, let's first introduce the basic authentication model in E-Store. I think most of you are familiar with these pictures. The first part is that in our Ingress Gateway, we will terminate TRS on the edge with user-specified certificate. And the second is a workload-to-workload communication with MTRS. Yeah, the third part is that the Ingress Gateway will help us originate TRS on the edge with user-specified certificate. And we can deploy, make use of them on demand. OK, let's take a look at how we can set up the certificate on the edge. This is on the right side. This is an example how we can set up Gateway credentials. First, we have to generate a certificate and a current and store them in a Kubernetes secret. Then we should configure the Ingress Gateway using the E-Store Gateway API. On the left side is how E-Store manages the certificate. For more details, we can take a look at E-Store.io. There is a task. Demonstrating this example. OK, then we talk about how workload communicates with workloads in the mesh. So take this example. Sleep needs to communicate with HTTP. Both of them are injected with side cars. This is very, very familiar to most of us. The most important part is that E-Store.io will issue a certificate for both Sleep and HTTP. And Sleep will use the certificate here to communicate with the HTTP service. So the certificate is passed here. And we can see here this is validating duration. And then the most important part is the identity part. Here we can see that the identity is type of specific with this domain and the name space and the service account. The right side is the raw certificate selection. The top part is a workload certificate. And the lower part is a root set. There are two ways of signing workload certificate in E-Store. The first mode is that E-Store.io. We will work as a CA authority. It will help workloads send their certificate with their own identity. For the example, at the last slide, Sleep and HTTP both have different identities. So we send different certificates to them. And the second mode is that E-Store.io or E-Store.io can integrate with Spire. So in production, we actually recommend the more secure and the easy way to sign workload certificate. But we have to make sure that we are familiar with both Spire and the integration solution. OK, the next part, I will talk about how E-Store.io can self-sign the workload certificate. Because most users for Simplity, they can just use E-Store.io to sign their workload certificate. And here, our topic focuses on the E-Store.io self-signed workload certificate. The first one is that E-Store.io can act as a CA. So it don't need any other configurations. When it starts up, it can automatically self-sign the certificate to workload. And then the second part is the RA mode. It will use Kubernetes to send certificate. We can see that it integrate with Kubernetes and use set manager, customer say, or anything, whatever. You can use that to sign the workload certificate. And here, we focus on the second mode. It is also CA mode. But we plug in CA signing certificate to E-Store.io. So this is the simple way we can commonly used to manually manage the bias and revocation of it. OK, let's take a look at what is plug-in CA certificate. In production workflow, we should prepare intermediate CA certificate with a safe CA. So the root CA must be on an offline machine to be secure. And then we need to create a secret named CA-SERV and mount it to E-Store. And the E-Store.io will detect and watch the search and initialize E-Store.io CA server. And then at last E-Store.io, we start up CA server and ready to sign workload search. This is the basic workflow. And we focus on how we can rotate root CA here. So before 1.20, the basic workflow to do root search rotate is signing a new intermediate here, CA-SERV, and then update the CA-SERV secret. And then E-Store.io reloaded for CA server and XDS server. And then we need to restart all the workloads in the mesh to pick up the new search. To mitigate it, we can connect multi-root search in CA-SERV. But this rely on the multi-root search, multi-root root feature support. This feature is, I think this feature is very long, from about 1.6. But there are some shortcomings here. All manually operations, and this is error-prone. So during the rotation, the E-Store.io traffic is very likely to be broken. And when XDS clients connect field, there is a gap between E-Store.io reloading the new search for XDS server and the XDS proxy loading the new CA-SERV. So this is a gap. And the last question is, how long could it take? The whole procedure could be hours, depending on the size of workloads in the mesh, because we have workload search validation for about 24 hours. And basically, we rotate the workload search every about 12 hours. OK, let's turn it to Jianpeng to show us how we can rotate the root CA in details. Before 1.20, I think there was two common ceremony about how we wrote the CA-SERV. This is the simplest way. You both intermediate A and intermediate come from the same route. So you can rotate or change the CA-SERV at any time. And because they are from the root CA, so even in this cramming, you can see the CA-SERV C and CA-SERV C have different workload set from different intermediate routes. But the communication is not broken. But sometimes, you may make a mistake and use a wrong root CA-SERV. And the off-stream requests you to change it from like this. You are running a cluster with intermediate B from route A. And the off-stream requests you to change it from intermediate C, but it was from another route C, or we call it route B here. And I think Christine Poster from the solo and Peter do a very good demo in the past and show how we can rotate it. But the truth is, we need to roll out East2D4 maybe about tries and all more. And then you need to restart all the inject application for maybe tries and all more. So if you have a large scale size cluster, it will be very difficult to run it in a short time. So how can we improve this? Let's take a look at our new very little invocation. So here's the key thing to reload a route CA without anything. Faster, this new feature is got by two feature gates. One is the Prox config XDS agent. And the other one is East2Mute route mesh. With this enabled, East2D is allowed to send a PCDS to the East2 agent when the route CA is changed. And for security reason, for the safety reason, we just allowed to East2D auto-reload the route CA in two cases. First one, the new route CA contains the old ones. It's just like A plus B. And another one is the new one is just part of the new one. It's just like you change from A plus B to just only B. And with this new change, we can allow East2D to reload the route CA with any route and restart. And all the workloads that will be auto-re-signed, you also didn't need to reload the applications. So here's the thing. We start from here. You have a cluster and you have a East2 with the route A. Here, I call route A just as the same as the intermediate A. And we need to route it to route B. And on the left side, you can see there's another two things. The combined route is just the route set A plus route set B. And there's another one called combined route 2. And it was just route A and plus B and one more. So this is the first ceremony. You can see we're getting started from here. The next step, we update the C set with everything from the last, but just replace the route set from route A to the combined route. And after about a set of minutes later, East2D will reload the new CA. And all the workloads will re-sign the workload sets. And because of all the machines, the new workload set is worrying for both from A and B. So maybe you can imagine you want to just the next step is just change everything from route B and with the combined route. But there will be a little traffic downtime if you just do it. So we have a high trick is the next step. We update the C set with the route B, but just replace the route set with the combined route 2. And all the route set and workload set will be reloaded in about just one minutes. And the last step is just change everything to the route B. And maybe one minute or two minutes later, everything is OK with the new route. Yeah, you can go through the demo. I wrote a handbook about this. And also, you can check out the E2E test as a storybook. I don't want to waste your time to go through the handbook here. And you can check out the doc. And if you have any questions, you can select me or just send an email to me. And here's the reference. And the first one is the issue of the all-thing. And the second one is the great book route by Christine Poster in long, long ago. You can see it was just E1, E6. And the rest two is all prerequisites. You can check out the detail. And maybe you can find some mistake made by us. OK, thank you. Well, and good afternoon. Thank you so much. Please just nod your native language. So thank you. Do we have any questions? All right. Hello. And thank you for this feature, because I did a lot of P2 certificate rotation in my life, unfortunately. And it's a pain. I have a question about the part where you have to combine the certificate. When you are doing it manually, so you have to do it in two steps. First, you have to combine your certificate A, please B, into your crystal. Do the rotation. And after that, replace the CRC by just B. So like this, you only have the CRB into your crystal. It is the same thing here. You need to do two steps. So you have two manual steps. First one, combine the old Routier and the new one. And after that, remove the old one. Is that right? Sorry, can I repeat your question? OK. So in this new feature, so first you have a Routier A installed into your system. And I want to rotate this Routier. So I have to combine my Routier, so my old one and my new one. And after that, I have to remove the old one. Or I just need to push the new one, and the system will under it for me. Are you asking if you should remove the old CRC? Yes. Yeah, definitely we don't want the old CRC to exist at the last. Because maybe it has been leaked, or maybe it will expire. Is it planned to have a feature where you just push the new Routier under the deletion of the old one for you? Sorry, I can't catch up with your question. So in this case, I have to move my old Routier, right? So is it planned to have a feature to remove it itself into the system so I don't have to do a manual operation to remove it? Sure. You get the question? I think he's asking, as a user, if he needs to manually remove the previous Routier. Do we have a feature in planning to help? Sorry, we don't have a plan now, because we think the plugin Routier should be in charge of users, not ease to deal. Since you are in charge as a user, so I like it. Any other? Do we have any other questions? I'd like to thank the speakers again. Thank you so much. Thank you. Thank you, everyone. Thank you.