 Well, let's get started. First of all, thank you very much for being here. I wasn't expecting a full crowd like that, so I hope you can enjoy some of the information that I have to present here. My name is Matheus. I'm from Brazil on a company called Cicredi, which is a credit union. Just a few words about Cicredi. Cicredi is the first Brazil credit union founded in 1992. It's a pretty old company. It's one of the largest credit, credit, Brazil credit union with more than 7.5 million members. When we call members, this is a term to the credit union. It really means clients. So we have more than 40,000 employees in its presence on every Brazilian state. It's important to mention that all our critical workloads, or great part of it, run inside Kubernetes infrastructure. So things like instant payment and ATM operations, all that rely on applications that run inside Kubernetes. So just a few words about me. I'm a platform specialist at Cicredi, working for the container orchestration team. I'm in the company for almost 18 years now. I'm a first-of-the-enthusiast. I was always involved with free software open source projects. I was involved with Debian GNU Linux in the past, Debian GNU HURD, and OpenLDAP and stuff like that. So my job is really replace proprietary systems with free software. I am a retired football player who sometimes I insist to play again. I regret every time because I got injured. I'm a father of two beautiful girls. It's a teenager and a child. So I'm not used to solve conflicts in Kubernetes upgrades. I'm also expert in solving conflicts at home. I'm a competitive Pokemon Gen 3 OU player. If you want what does mean, I am Shahua BHC on PokemonShodown.com. They said it's a child game and really is a child game. Sorry. Sorry, mom. I'm a football lover. I'm a supporter of the best team in Brazil, which is Club de Atletico Minero. We call them Galo, which means Rooster in Portuguese. So, okay, let's go. What do you mean, what is really important now? What do you see in this presentation? Well, we will have a brief introduction about Cicredi infrastructure related to Kubernetes. How do we automate our service mesh configuration? That's a pretty important point because they face a lot of challenges. Just before we get started, really, how many of you here, if you can raise your hands, are using service mesh in somehow? Okay. And how many of you are using Cilion cluster mesh? Okay. People with Cilion cluster mesh will see, I think, important things here. The challenges that we faced, the most difficult challenges and our next steps. So, I will start with the infrastructure because it will be important to understand what we have done there. So, we have a private cloud built in canonical open stacks spread across three different data centers. That is on-premise infrastructure. We have public cloud built on Amazon Web Services. Kubernetes clusters are managing using cluster API on a private cloud. And on AKS are managed by Terraform. Here we have, in Cicredi, we have the concepts of stacks for clusters. So, each cluster has what we call in stack. What this really means is a repository in Git, GitLab, containing the information about the cluster configuration setups and stuff like that. So, the Kubernetes clusters that manage it on-premise are basically formed by a cluster API description of the cluster in a YAML format and some Ansible playbooks that run Ansible roles to install clusters components like Cilion. On the AKS for the other side, we have a stack of pure Terraform using, of course, modules like HelmModulus to install the components. We have a dedicated Amazon Direct Connect link with all data centers. This is great because the latency is very low. So, we feel like we are operating in the same data center. This number is a little bit outdated, but I didn't want to change this slide because I already had it updated before. But we have more than 40 Kubernetes and AKS clusters. These numbers are an average numbers because as we have horizontal pod outscaler and cluster outscaler, these numbers tend to change across the day, across the traffic. But in a medium, we have 7,400 pods running production and 500 Kubernetes and AKS nodes running production. I know I saw some huge infrastructure, another talks, and this made me think, oh, this is not so big, but for us, it's really a great number of nodes and pods running. And this thing keeps growing. So, the canonical open stack infrastructure will look like that. We have a region and we have spread across the ability zones, three ability zones. So, the key points are search view lanes between data centers. So, we have fully communication between them. Storage is not shared or replicated among the zones, and open stack control planes are spread among the zones for high availability. This is a very high level description. The idea here is to show what are the components that we have because we have some challenges and you will see further are related to these components, to some of these components. So, we have our data center connected with Amazon Direct Connect. Then we have a transit gateway. And then we have a gate or load balancer. And behind that, we have a Apollo auto fire infrastructure to reach AKS clusters. Of course, this flow is both direction. We have traffic coming from our premise data center and reaching AKS and AKS reaching our premise data center as we work on the fully infrastructure. So, a little bit about our CNI history. We born with Kubernetes in 2018 using Flannel. We changed to a project called Weave in 2019. But due to some problems with Flannel in the excellent type of infrastructure. But we saw that Weave was dying. And we had some problems with it. So, we changed it to Cily in 2020 looking for the security features, EBPF, of course, and the more robust and stable CNI. So, moving to Cily was very important because there was a lot of new applications getting in the cluster. And with more and more applications, security also become a problem. So, most part of the applications were running on Kubernetes did not have any access control. The developers did not write any code to control that. And that was a big problem and an addiction problem for us. So, self-image capability was, of course, an alternative to solve that problem without changing the source code. This is what self-image works for. So, it makes sense. We started proof of concept by the end of 2020 on potential self-image solutions that solved a particular problem. We wrote a sample application that was very similar to all other applications that we had running at that time in our Kubernetes infrastructure to make the tests. And we selected the following self-image tools to evaluate which was the best. So, at that time, Istio was the most prominent self-image tool. We had a very young project called Kuma, which is a self-image by Kong, including they had some talks yesterday, I think. Unfortunately, I could not be able to watch. Celium, of course, which was our CNI. And ConsulConnect, which is the self-image from Consul, because we are already using Consul and some hash cortic tools to configuration and start secrets like Vault. Okay. What was the key point to us? Of course, a feature to block or allow access from one application to another. This was the main point. We would like to have some layers having security policy if this was important. Multi-cluster policy capability. Of course, at that time, we were running multi-clusters, so this should work across multiple clusters. Great part of the self-image tools at that time used sidecars. So, we were a bit worried about the resource consumption and we decided to do some kind of performance test with that sample application that we wrote. A low priority feature that we want to look at is mesh expansion. Mesh expansion is just a way to work with workload that are not in Kubernetes, but you want this to be part of the self-image. So, you can enable this. You have some tools to some service mesh that implement this. This was not so important, but it's something that we would like to check. This is a basic draw about the application that we wrote is a microservices application. We have the employee data connecting to employee proxy, reusing RESTful services, then connected to two other services, employee test A, test B, which connects to a NailedApp database. At the great point here is that we would like to test if it was possible to put rules in that connection points. That, of course, we will solve the problem that we had. So, the Istio service mesh Pock, Istio access control works, policy works, with authentication authorization using MTLS. It have the sidecar envoy intercepting all the traffic to apply effectively, apply the policies. As I said before, we were worried about resource consumption because of that sidecar. It's just to show it's good performance, with no major problems. It could detect a little bit more resource usage, but that was not an issue for us. We thought it would be worse. There was an issue at that time with mesh expansion. It would like to test. The point important here I would like to mention is that information is about the end of 2020 and 2021. Some of the subs mesh may have bugs which already solved it, or functionalities in them could be changed. I don't know, but this is what we've run at that time. So, Kuma was a very, very young project back in 2020. We had several problems that made it impossible to really do the tests. When we applied the policy to validate the access, the probes didn't work, so the applications never could get up and running. And unfortunately, we could not be able to finish the test. This issue was solved by the end of 2020, but we already finished the proof of concepts. Well, console connection. Here is a very weird one. Sorry about that. Console connection versus policies by unicycle cars and containers at that time, just like Istio. But we found very unusual configuration to name that. An application to be protected needs to be bind only local host, which is we basically had to recreate all our images to applications by only local host if they want to be protected in the mesh. And this is most the weird one. Applications that access another need to access local host. So, employee data in our example, employee data as accessing employee proxy, the configuration, the properties inside the employee data should point to local host and the port. If it connects to multiple services, you should still use local host and different by port. I'm not sure if this is still working like that, but if it, yes, I don't know, it's really complicated. And then you need to put some annotations like connect services upstreams and the service name and the port and the original data center is a concept on console agent should be installed on Kubernetes nodes. So, we can't do with all that. We want a little simple infrastructure. I'm not sure if this is the way they work now, but back then it was the way they work. So, oh, Celium here. And Celium work with Celium network policies to allow and deny access without the need of sidecars. This is great. Multicluster policies works perfectly with some requirements that pod IP CDR should not conflict. And, of course, services must be deployed on our clusters, which are part of the service mesh. This is if you want the services to be accessed from other clusters, of course. Services discover uses Kubernetes coordination with school. Cluster mesh observability using EBPF is also. So, we did a test using KubePROX replacement in its rock. You don't need to have KubePROX running in your environment. And you can do a fully EBPF thing. So, the performance showed very good results without a sidecar. So, there is an obvious answer to our question. Why we choose Celium as our same mesh. Clusters network policies. Crosscluster was exactly what we need to implement the security services. We don't need sidecars. We have the EBPF. And it was already used as a CNI. So, no need to install new software. When you enable cluster mesh, it will install new components from Celium, but not a different sort of software. Like, for example, if we use Istio, there is a bunch of Istio controllers and Istio software that needs to be deployed. Liz Rice gave a very good talk, I don't know, in the day, too, about cluster mesh, including the she did a demo. So, I will pass this very briefly because they have more information on that talk, if you want to take a look. Here is the requirements. Our clusters must be configured with the same data path mode. Celium installed may default to encapsulation, native routing. The native routing here is the fully EBPF thing that have very good performance. As I said before, earlier pod CDR ranges in all clusters, and now nodes must be non-conflict and unique. Nodes in all clusters must have IP connected between each other. The network between the clusters must allow the intercluster communication. This is what Celium cluster mesh really is, a very important part. When you enable cluster mesh, what really happens is that a component called the Celium cluster mesh is installed in your ecosystem namespace. What is this? It's basically an ETCD. And what is the point of this ETCD? It's to store the service identities for all your services, and the agents of Celium will connect on that ETCD to get that information. So, if you have two clusters, the agents from one cluster will connect in the other Celium cluster mesh to get that information and make things work. But again, Liz Rice did a very good explanation about that if you want to check out. Okay. Here's some important questions. Enable cluster mesh using Celium CLI or Helm installation? How do you connect the clusters? How do you distribute that key securely? And it's possible to automate our cluster mesh configuration. When you look at this Celium documentation about cluster mesh, they will use Celium CLI to do that. But we did a little different. And I think this is more appropriate when you're talking about an environment that is not a playing environment. That's a really production environment. So, Helm was easy to install CNI. So, let's use Helm to manage cluster mesh installation and configuration. We are using Terraform and Ansible to fully automate the cluster mesh process. Just mentioned that in case of Terraform, we're using Helm models behind the scenes. But that stack that I said earlier are changing if we're applying this on our premise infrastructure or in Amazon infrastructure. We are using HashiCorp Vault. Now, HashiCorp, fantastic job with Vault and Consul, but not with the mesh at that time. We are using HashiCorp Vault to store clusters connection information and its certificates and credentials. So, there is a, in Helm, in Celium Helm, there is an option called TLS Auto Method, which we choose Helm. This means that Helm will be generated, will be used to generate and store the certifications and credentials. These also have other options like Set Manager, if you want to do that job, or if you want to provide yourself by hand. We need to generate a new certification authority that will be used across our clusters to sync cluster mesh certificates and credentials and store it on Vault. For each environment, we create and have it sound a certification authority. All communication between the clusters are encrypted, so these certificates are very, very important to you make things work properly. So, here is a flow of the automation process. So, we basically have Terraform or Ansible, and there is a flag which is cluster mesh is enabled. Yes or no? If you know, okay, don't do the configuration for cluster mesh and finish. If yes, then we need to check if the certifications and the credentials are already on Vault. If the answer is no, then it means that there is a new cluster mesh enabled. So, there is Helm property, TLS Auto enabled and set to true. This is the default. So, what will happen here is we will generate all the certifications and credentials that you need and store this using the certification authority that we provided first and then we will store it on Vault. Then it will start the information about that cluster, specifically what are the endpoints and the keys to access them. It will get the name of the clusters to connect from a list. It will get all the information in respect of that cluster, like the connection, the port, and the credentials started on Vault, then they will apply on Helm. If there are already certifications in Vault, this means that this service mesh is already enabled and maybe up and running. So, we set the TLS Auto enabled to false. That's pretty important because if we don't set this, what will happen is it will generate new certificates. If you have other clusters connected to it, it will basically block all the clusters from Connect because we generate new keys. So, your service mesh will stop working. Just a short example, we have this Terraform. This is a main.tf from one stack of a specific cluster. And we have an iteration about the clusters list. And then we will get on Vault and get all the information about the clusters that we want to connect. Down there, we have the check if these certificates already exist in Vault. If not, it will set the TLS Auto enable to true and will generate certificates. If not, it will get the certificates from Vault. What we have in here is the local tf of that specific cluster. So, we have this by environment, the flag, enable cluster mesh, which we can control if some environment could be false or true. And the connect cluster mesh, which is the list of that cluster will connect. The other clusters that will connect. So, for example, if this is the stack for the cluster D, right, then in develop an environment, it will connect to cluster A and cluster B. In production, we will connect to cluster A, B, and C. And this is a major difference because if you run what is in documentation using CLI, when you connect the cluster A to cluster B, the CLI do both way. They do cluster A to B and B to A. Helm doesn't work like that. So, when you put in your cluster D, for example, to connect to cluster A, you need to go into cluster A stack and put the cluster B. So, this is pretty important. Okay. How users control Selium network policies in our case with Dev Console? So, what is Dev Console? Dev Console is an internal developer platform solution for offering infrastructure resource from all platform product teams. So, a developer, when they need to start a new project, he accesses the Dev Console and asks for a new project, give a name to the project. This will trigger GitLab repository. And he can choose the template. He wants to develop using Java and Spring Boot. It creates a template with Java and Spring Boot with a sample of GitLab CI that will do build, test, and deploy. And he can also configure the resource usage for that specific component, the HPA, and stuff like that. So, here is a part of the screen. It's in Portuguese, but I will try to mention the most important point here. So, this is a screen for a service called Demo Dev Services KO2-6. So, here is the Ingress and Negress rules. So, when the developer needs to access some other API, other application, you have a button here, which is called, which has a new access, new request. Then he types the name of the service and he won't connect. This will trigger like, for the other team, some kind of improvement rules if the user is the developer of that API approved. So, the access will be okay. What this really does behind the scenes is creating ceiling network policies. So, we have that ceiling network policy for Demo Dev Services KO2-6 having access from Demo Dev Services 2-6 and the other way around. So, this is basically what Dev Console do is create those policies in all clusters. And this policies applies to all clusters that we have in ATS, in a no-premise infrastructure. The challenges that we face, this was the most difficult one. Palo Alto Fire was generating VX land drops due to a bug. So, if you have infrastructure using Palo Alto, you should check this document because we spend a lot of months on that. We were thinking that maybe the problem was in that Amazon components or gateway and we tried to do, but the problem was on the Palo Alto, which was dropping the excellent package. The solution was basically some kind of profile that they write there. I don't know really about Palo Alto, but our security team did that and it works. At that time, we have just our clusters in our own premise data center talking to each other through cluster mesh really well and ATS talking to each other really well, but when they need to connect to infrastructure that are on different data centers, it was not working. Ceiling agent memory leak bug, this was pretty complicated. When you have the cluster mesh enabled and clusters are connected, the Ceiling agent basically increases memories a lot and we don't have limits to Ceiling agent, so they start to stop working our nodes because of the lack of the memory. Cluster connect and enable cluster mesh with Ceiling CLI and Helm was a bit confused because of the lack of interparability between the two. So, they have an issue about that, but basically, in the past, we connected our clusters using CLI. So, okay, now we will do automation. So, when we started working with Helm, things become weird. And then we saw that they are conflicted, not work. If you are working with CLI, you need to keep going with CLI until the end. And if you work with Helm, you just have to work with Helm to control that configuration. There was an issue on Celium. I don't know if it fixed it, because we just worked with Helm and no use for Celium CLI anymore. One important point, we had to disconnect everything and disable cluster mesh with CLI to get it working with Helm, so it was really complicated. With cluster mesh and self-discovering mechanism working, we are now able to deploy the same application on both cloud infrastructure. What happens now is when a developer creates a project, it will be in one cluster, or in on-premise, or in Amazon. And you can migrate this from one site to another and from clusters to clusters. This is possible with Dev Console, but we cannot have the ability to run both in the same data centers, which is a very cool feature that we have with cluster mesh, and we would like to use. So, working will be done in Dev Console to enable this, because we do the deploying in the pipeline using Dev Console APIs. Extend observability with Hubbo and Grafana tools, because of the eBPF, we would like to use this a lot, but we use Dynatrace as a main observability software right now. So, this is something that we are discussing internally. We would like to use Gateway API to split traffic and enable counter-deployment. We need to see how this works in cluster mesh. So, we are studying and doing some tests, but it's all that. I'm here for your questions. Thank you for your time. If you don't have enough time, my LinkedIn is on the schedule, and you can reach me to more detail. There's a lot of things that we did, but I could not be able to speak just in 30 minutes here, but you can reach me to find more things. Just one quick question regarding the, I'm here. Regarding the on-premises CDM cluster mesh config, are you using no port or low balancer for that? If you are using low balancer to connect both cluster desks external or using metal LB for that? We are using, in OpenStack infrastructure, we are using the OpenStack LB, and on Amazon AKS, we are using Amazon LB. Okay, cool. Thank you. Thank you. I have one question. Thanks for the presentation. Thank you. I just wanted to ask what component uses the VXLAN. So is it from the Amazon Direct Connect? I'm not sure because that network part of the intercommunication between our on-premise data center and on-premise and now Amazon is done by other team. So I really don't know the protocol, but I think there is a lot of protocol because I get a low balancer work on Geneve protocol, but we could be able to connect VXLAN directly from one site to another. You can reach me out after I can ask this question for our network team. Thanks. And did you have any challenges with the IP management over the sites? Yeah, you need to control the cluster IP CDR, and when you control that, you will not have problems, but if you don't control and get the same IPs from one cluster another, this will give you problems, but we don't have face at any of them. Have you done a lot of traffic management between your clusters? So the documentation states every service needs to exist for that traffic to do, I think essentially it's defaulted to around Rob and then trying to do any kind of, what's your experience on that? Because at the moment I'm trying to be very declarative in my traffic management, and I was speaking with myself from my surveillance, we're thinking that you could just not create a service in a cluster and it won't and it'll go to the other clusters, I'm just curious. We don't have any problem in traffic management, but this is something that we need to do. All clusters should have the services, so Dev Console basically deploy all services in our clusters. Our link with Amazon is very good, so I don't know of any problems of traffic management between these connection points, but again, I'm not in the network team, but I think if you had something, they had reported me. Hello Matheus. Something we are dealing with is ClusterMesh right now. Brazilian company choose, Tony. Okay, we understand that pod IP must not conflict, but services, how you deal with services? Because in the cluster mesh, if we have the same IP for services in our entire environment? Okay, there is no problem with services. You can have the overlapping IP address for IP for services. I mean, you can have the same services with the same IP on different clusters, that's not a problem. It uses the services just for reference. What it cannot be overlapping is the pod IP, which is the problem. If you had overlapping or same IP in both clusters or in your clusters in the cluster mesh, you have problems, but you don't need to care about services IPs from our perspective. Okay. Thank you. Hello and thank you for the talk. I have a question if you have one node or even a cluster of Kubernetes running behind nodes one or two or more. Can you connect those with SeriumMesh? I think you can, because I think there is a lot of net between our connectivity with Amazon, but the point is that VxLan traffic should be enabled in all these components. You should guarantee that you have VxLan working on both, but I think it's working well with net behind the clusters, because this is also an area. We have nets in some parts in the communication between our own premise and Amazon, and it works well. All right. Thank you very much. So you need to have non-conflicting networks for pods, but if you initially had, do you have experience changing the network, pod network on the cluster, on the kind of fly without destroying and creating the cluster? Yeah. Yeah. That's right. We had, when you enable this, when you put some pod CDR, you need, and you want to change. I mean, if you start with some CDR and you want to change, this is very complicated. We have to restart and destroy every node to get a new one. So we passed this in our test environments and it's kind of annoying. But when the cluster is born with the right CDR, then you have no problems in our experience. Okay. Thank you very much for your time. I appreciate having you here, guys. Thank you.