 Hello, everyone. I hope you had so far a really great and awesome KubeCon and you have met some new technologies, some awesome cool projects, have learned something new. I'm Max and I will talk today about multi cluster communication with Link at D as well as the really nicely integrated observability you get with it. I'm co-founder of Liquid reply. Liquid reply is a part of the consulting network from reply and we are focusing on cloud native engineering, Kubernetes and yeah, basically bring container orchestration to multi cloud and hyper cloud environments. In the evening times, I'm actually working on the Kubernetes sick release team since Kubernetes 117 and on the weekends, I'm more on flying with my drones or crafting the native cloud deaf newsletter. As KubeCon is about staying connected, making new connections, feel free to reach out to me on LinkedIn, Twitter, or GitHub. I'm very glad to talk and discuss with you about the topics which I present today or anything else which could be interesting for you in the Kubernetes universe. The big question which we actually ask us several months ago or actually more than a year ago in the project which I will show you today is how many clusters do you like to have and how many clusters make sense? And this is actually a big discussion as I've understood on the market. They have the two extremes like one cluster for all applications as well as one cluster per one application. And you have somewhere the medium crown floor in all kinds of color shapes I would say, but most commonly I have seen on the one side the one per domain topic which is more like following the organizational structure as well as more on the cohesion or subject topics like all sales application or every application which is really chatty with other applications on one cluster together. We'll take a look on this question later on, but what is also really interesting to see in this context is what the CNCF Survey 2020 has shown to it. And these days that the, not only the amount of production clusters are increasing but also the number per client itself on the clusters increasing. So they're not just adopt in production but they adopt multiple clusters into production. On the other hand side where multiple teams work together they think a lot about how to isolate the names basis from each other or either do it by cluster separation. Overall, what they all have in common is that they struggle with topics like complexity, security, monitoring, logging, networking is always a really hot topic. And also sometimes just the lack of simple trainings is there is real. Our project which I want to show you today actually has the same starting point, new to cloud, new to cloud native. And as we have seen we were not the only ones started on this point. This list of requirements you're given is actually a short drafted version of what was given to us, but it highlights the most important points which we can see really outstanding everything must be encrypted. And I mean, every everything whatever somehow appears somewhere and the system landscape must be immediately encrypted and transit when the data is addressed and so on. Everything also must be isolated. The front end from the backend from the data management from the databases there must be really heavy isolation for smartphone network layers in between. On this other hand side we need to publish a kind of internal public API because the system is heavily used by many other systems itself. And then we had funny restrictions like if you want to have a communication from on-prem versus environment it's just a lot on the specific time window and you have to request it upfront. And of course, frequency changes. New components leads to new isolation, new network stuff which is good, but it's something which wasn't really planned from the client side and this often leads to a lot of increasing complexity. In the end we had multiple Kubernetes cluster with a lot of networking around virtual private clouds different accounts, different regions. We also currently plan for different cloud providers to isolate them. But why all of this stress? Really, why are you doing this? And the answer is we are dealing here with a lot of personal identifiable information which means in Europe it must be GDPR conform. In addition, as a telco provider you're also talking about an internet provider. And for these, you have tons of different regulations. It's critical infrastructure. There are different frameworks around it like critters and other requirements which you have to fulfill. And I said, we start here with a client-to-cloud journey which is awesome, but it also means there's no idea and full understanding what means cloud native. And so we have basically unchangeable rules and processes standing in an environment which moves really fast, which is evolving continuously. My favorite example here is that we are working in a cloud environment to change a port or add the IP addresses should be a thing of a couple of minutes. We need weeks and a good time, but more because of the process. It was brought to someone who planned or given your IP address. We got to approve it, bring it back and so on. Nevertheless, things are changing and this is actually quite good. First, I want to show you actually our landscape. I need to make it way more simpler on the one hand side because it's otherwise difficult to understand. On the other hand side, I cannot show you all of it. I said critical infrastructure and therefore we have several restrictions on this. We have the corporate data center on the one hand side and this communicates through a networking account with all of the other accounts available. You also call it often happens bulk architecture. The accounts listed here down are just like representing that there's a couple of other clusters and components in this whole context from the application we're talking about. But to keep it easy and simple, we will focus on the central part. And here you can see we have a front end running with some network layers. We have a backend also with some network layers and the data part or the data persistency part is isolated actually in another account. So when you have a front end request going to your application logic, it needs to go all the way around and then to the data management plane and then to the database. Why we do this? Well, we have shown you a couple of requirements. Some others are standing on the side of these. One thing for example is that in a really heavily managed network IP addresses are on the ones that deal like diamonds. On the other side, you want to have this heavy isolation. So you can be sure that not accidentally the one thing with the other thing can communicate. So this is a little bit like you need to consider when to choose which approach on this point. What we face with this architecture also is that the change of the application or adding new components can lead to that you need to introduce a new, maybe account, maybe network fear, how you want to call it and it's increased over all the complexity. And this also increase the amount of failure points or break points for your communication and to find then the right spot where something is failing is really difficult. So overall, this architecture is actually pretty stable for cloud architecture. There's some things missing, but overall it's okay. But it's not really cloud native. And this is what have bothered my team and me massively. So we did our research. We know what service measures around. We have used it here and there. But I said we are in an environment which just adopt to cloud native and to approaches which are on the cloud native field. However, we put some plans together and proposed in the end, link at D. We have also validated as a service measures, but we thought this is like the good mix of easy to introduce, not too heavy to manage and we can slowly get it adopted over the whole environment. We had also some really easy hopes with it. The first hope, obviously encryption for lazy people, everything, everything in the communication gets encrypted via MTLS automatically. We get the visibility on the communication side so that it is easier for us to see on which point of the communication in the whole system, we maybe lose something or something is broke. And in the end, last year introduced the service mirroring. We really had the hope that just communicating or let the systems communicating between the clusters natively without having all this network stuff around it. We reduce the amount of needs for API gateways and load balancers and increase services and routes and all these beautiful things which keeps you quite busy. As I cannot show you our life environment or environment which you have built for the client, I have basically refactored what we have more or less set up there also. We have a local cluster running on Monopoke was kind where I have a front end application running and then we have two different clusters running in two different regions on AWS, one west, one east running the backend service and the database or data management service. So let's have a look how actually it looks like when we set up link ID in the multicluster service. For this, we can also directly have a check here. Have a check here on our application running and maybe it makes sense to go from bottom to top. We have here the standard link ID components, the link ID control plane. One thing that you also have to learn was like this thing and the way how it's installed is not high available. So if you introduce link ID and it should go somewhere where it's serious, you need to give the dash dash HA flag to really set it up in a high available mode. And then with some of the latest changes of link ID, they have excluded some of the components to make it more extension driven like many things in the cloud native environment nowadays, I would say, to reduce over all the footprint of link ID initially, but on the other side to allow to go for the next and the next and the next service, whatever you need or suits to you. So what we have installed here is the WIS which basically just crawls metrics from the Kubernetes offers a Krofana service and allows us to basically go from the link ID dashboard directly to the Krofana dashboard and see what's going on there. And then we have the component which is really interesting for us. This is the link ID gateway and what you can see here also is the link ID service mirror West. So the link ID gateway actually is responsible for getting and keeping this configuration between one and the other cluster. I have a look in a few seconds how this works. While the link ID service mirror already tells us like, oh, there has happened something. And what basically happened is that we have on our West cluster on the other cluster which is running on the cloud, running a Yelp app server. And this one has exposed to service and the link ID multicluster identified through an annotation, this thing must be mirrored to the local cluster. And this is actually what happens exactly on this point. So the Yelp app server West was a dash West intends us the service is actually running on another environment. We can also check this. We have a look onto what's going on in the West cluster. And here you can see we have the Redis running, we have the Yelp application server running and I deployed an engineering service for showing you something in a few seconds. So we have our front end running on my local machine. We have the application server running in the cloud and if the database also somewhere flying there around. What we now can do with this is if we go to the web server here, it's a simple application where you can vote a little bit around. And what you can see here is also which web server basically surfed. So it's a J8 D5D, J8 D5D, which is actually exactly this service. So having a look here on the link ID dashboard, you can see here the UI has a good success rate. Everything so far is fine. There were a few reads, a few writes. And we can also go directly to the Grafana dashboard to see a little bit more, okay, what's going on. And you can see here, for example, our request which we have taken on the latency side. Now for the UI itself, it's quite boring to basically observe it. So what we also can do is to have a look on the other dashboard. I'm just right now thinking, I think it's runs already. If we go to the local hosts, 3,009, definitely we have something here. I have perfect, and it's for the application server. And also here we can go directly to the Grafana dashboard and see what's going on there. So we see here the spike just from our request where we have clicked a few times on the word buttons. So if you want, you can escalate a little bit more here on the web UI. It will take a few seconds until we see the request on the application server. And there you can see already the spike, perfect. So this is how the things are somehow tangled up. Now we have the flow from one side to the other side going but you will maybe ask like, okay, what happens if I'm on this application servers and want to call, for example, the service on the other hand side? Well, this we can also have a look on. So if you check out the services available on the local cluster, we have our UI and the app service. If we check out the services on the West cluster, we see that we have our database available here, the app server and the Redis. Now what we can do, we can shortly access into the engine X, which I've deployed on the West cluster. I can add directly this one, then we need to tell also the container which we are running on the context. And we need to tell what it should start in the container. Okay, perfect. So the most simplest way, obviously how we can check if you can reach anything of the other services, we can check out the application server and do the four, five, six, seven. Okay, looks good. You can do get nothing comes back, but this is fine. If you now do a curl on, let's say, the Yelp database East on the five, four, three, two, I have a Muspel here, why? East empty reply from server. It means at least we can reach it, but it cannot reply to the current. Obviously it's a database server, it needs another request. Now, if we are looking for the Yelp UI, which is running on the port 80, we cannot resolve the UI, obviously. First, this what we see here is on the local machine and what we see here, we don't see that there's a service published from the local machine in this direction. And this is actually what suits us very well because as you said, we have several security restrictions and requirements here, which actually wants to prevent that we go from or are able to go from more from the part where we have too many logic back to some other of the directions. So we have always one direction communication, if you want to. But for sure, it is possible to connect both clusters and have a cross communication. As you have seen here on the context of the West cluster, you can see we have the database East service published. So this is what we want to have. And this is also what we use in our scenario that we can reach from the application, the database. Now, I have also created here a link in the East cluster so that you have actually also the application server West in the database area available. So theoretically, we could now start the front end for whatever reason ever in the data part and request from there the application server and then it would basically make a round trip back to the database to store the data here. So it is possible to allow cross communication for the clusters. And our scenario just makes sense that it's going to one direction to not allow too many communication paths basically. So how this works when you configure the link ID multi cluster, it's set up your gateway. This is what you have seen already as a deployment. And when you run then these second command link ID multi cluster link, as you can see it actually craps or runs on the West environment and uses or bring together a service account cluster role and everything else what's needed for the cube conflict like the certificates bundle this together and apply it in the local cluster. With this command here. So then the gateway on the local cluster knows where it can find the endpoint of the Kubernetes API endpoint of the best cluster and allows it to crawl it for some updates when you now have an application running until like, okay, I would like to have not just my service for it available, but also to export the service to the other cluster, which is linked. You annotate the service with mirror link at the IO exported through the gateway will identify this annotation and will create a new service with the dash West and pointing to the application into the other cluster. And then your application here is able to communicate through the service mirror was an application on another cluster. One remark here, you need to add in your application the new service endpoint. So the service endpoint obviously will change if you have both components before running in one cluster you can call the service my service easily within the same cluster. But if you move the component out to another cluster you need to also adjust the service name because it's then called my service minus West. This is something which we also need to adapt it in the application because not every application took the service endpoints through some variables. Also one nice feature, I do not want to go too deep into it but it's a possibility of doing traffic splitting. And this will allow you that you have for example two times the same service running in two different clusters and you just like shift the traffic 50% to the be here and 50% to the application on the other cluster. This makes sense when you maybe play more round in this field of like you have one cluster application or you have a cluster with a high cohesion C and you prefer want to run for example an update but you want to ensure that your system stays available. So this way you can run two clusters and have them mesh together, have the traffic split and if something happens then always the traffic will go to the other cluster. So you have kind of fall tolerance here is maybe kind of progressive delivery actually but as a really nice feature. Okay, so what I've shown you and shortly described is actually what we have done also as a solution for our cluster. We have basically brought together the front and back and data management cluster, applied everywhere the gateways have linked them up into that the service from front end, the service from backend is just visible in the front end and the service from data management is just visible in the backend. So from the communication pass, the front end service can talk with the backend and the backend can communicate with the data management API. The lifesaver for us here was the API server address because as I said, we have really high network isolation. This means even this Kubernetes could not find the other Kubernetes in the beginning. So we basically given you API address between any our head due to the given architecture and this makes it really easy to set up these network communication also to allow the communication over to the other cluster. So basically that doesn't change so much in the infrastructure in the network communication but when you have it once set it up correctly, you don't need to touch it anymore again and this makes it actually quite nice and sweet. What I also mentioned, you need to annotate the service with the exported equals true to basically copy over the service or mirror the service from one cluster to the other. And this nice image down here from link at D explains again, if you create this annotation for the mirror, what your cluster on the right hand side is does is it takes the cluster role, the role binding and the service account, wrap it up into a cube config, copy it over to the other cluster, deploy it there and the service mirror gets created which then can point to the API server back to the right hand side cluster. Sounds complicated from the setup until you can fly it's actually quite easy and fast to me to say. What link at D also brought to us is the insights which you directly get over it. As you have seen, we had a lot of moving components between different clusters and it is actually a problem when you have services which maybe start stumbling around or even die because they somehow suddenly cannot communicate and you don't know at which point actually this happens from the operations point of view, we had the whole cluster as one, but then we saw something comes in, we saw enough goes out of the other side, everything is fine, but sometimes what happens inside was not so much on track. So this was actually our starting point to get really deeper down into our cluster visibility. And I actually can directly go here to the dashboard. Maybe we have here service also. Another example which shows that not every time the success rate is for example at 100%. So the success rate tells how many of the requests were successful was in a given time period. And in link at D, this is normally set to one minute, you can extend it by default if you want. You have the requests per second and then the latency. You have the latency for the presential of 50, 95 and 99. What means is that the P50 gives you more the average value while the top 95 and top 99 gives you more like the exceptions which you potentially have in the requests. And down here on the TCP, so really more on the protocol layer, you also can see how many connections do you have and how many read and write bytes per seconds you have. So this gives you a pretty nice insight into the whole service actually. And as usual was a very nice integration to Grafana. You have actually one place to go down and find the information which you need or identify where maybe from at least the magic point of view have failed something. I wish there would be a little bit more also integration with the logs or log files with Grafana, Loki maybe in the future but maybe this will come. So yeah, this was a really nice starting point for us to get into the insights of our application. What we also need to say here is that actually we went then one more step further and say like, hey, if you have already all the things set up this link at E we have our metrics we have really easy the dashboards integrated we know which service maybe doesn't got any requests or cannot answer all of the requests. How about tracing? From link at E perspective implementing the tracing is really easy. It's just one extension as I explained earlier link at E has several extensions and to set up everything what you need on the infrastructure side can be done within theoretically minutes if you want to make it production ready a few days but the implementation itself is super easy. On the application side a totally different story for sure you need to integrate libraries into your application code to make the tracing actually fully functional. If you have done this, there comes a Yeager UI out of the or along with the link at E it directly integrate gets all the information and data out of the link at E component. And then you can directly go into the UI and find your traces here and see how long you request needs at which point in time and so on potentially also where it fails for which reason it fails and so on. And basically what you can see also is that this really tiny diagram down here is reflected also here on top. So this is how the sports things are integrated with each other. It's very nice, but it's really long story until you get to the point of view that your tracing works very well. So when multi cluster makes sense after playing a lot around with it, I would say for sure when you have this full fallback scenario you have two clusters running beside each other or you want to just have a blue-green deployment with let's say a little bit more legacy application which is really slow and shifting from one thing to the other thing, it is a lifesaver. It's easy to set up with two clusters. You don't need to worry too much about anything else. And if your network is not that complicated then also the integration and the linkage is really easy. You don't need to do a mess around with it. Our use case, which we actually have is the handful of clusters, but I need to say the more application you run per cluster and the more you want to mirror the service from one cluster to the other cluster the more you complicated it gets somehow. For sure everything is declarative so you can just write it down and basically like a former time the teacher in the school just mark what needs to be changed. But it can be a lot in the beginning especially when you're in the beginning of a cloud journey with the client and you bring one after another tool into the discussion it can be a little bit too much. Where I found it really valuable it's actually another use case the hybrid environment extension. So you have some services running on the local data center and some services running on the cloud. This is very common between many of our clients also that there are really some things sensitive data or that sensitive information. They don't want to move it into the cloud but they want to use everything else in the cloud. So they have a connection. They can process or filter some of the data on the remote on the local environment on premise and basically just take the extraction out into the cloud and process is here. So let's say with machine learning or image scanning, image recognition, big data analysis with BigQuery for example and so on. So there it's very nice and very interesting. Coming back to our hopes what MultiCluster actually gives us if you consider from the beginning MultiCluster setup with Link ID you definitely need to just once set up the infrastructure and maybe do minor adjustments but overall you then can forget about the networking part. I nearly haven't talked about it MTLS communication encryption. I mean this is what service measures actually always raise the flag for. Everything is declaratively and therefore transparent. Yes, it can be a lot, it can be complicated but on the other hand side it's literally written down what you have and what you want to have. And this is a good starting point also to communicate what you maybe want to change or which things are not a good idea maybe. In some cases you do not have anymore a single point of failure. Especially in the case one that you run two clusters which are linked together. You have a backup or you have a fail backup system therefore this can really relieve the amount of stress if you have a really old application with a really old Kubernetes version and you need to upgrade now. This is a good way how you can balance a little bit away the problems. And my favorite Kubernetes first. What happens in Kubernetes stays in Kubernetes so it feels really natively the communication feels natively you don't need to suddenly look in your service for an IP address or an external domain name and then reach out to this one or allow the communication from there. It just like happens within your cluster which is awesome. A few lessons we have learned. Yeah, the cluster names get extended so the service which you expose get an additional cluster name. There's nothing to be complicated on it but you need to adjust your applications for it to be able to adapt now to the new name of it. Link ID upgrades can be a pain in the ass. I'm sorry to say that. So please have a look on the page of Link ID there are some upgrade guides. You definitely should read it two or three times before you do something like this. Sometimes what we have observed is that the services didn't got updated that fast and therefore some communication broke some services went unreachable and we have to restart for example once an application but we never could really replicate this. Service measures itself are a rabbit hole. Think twice before you use it. A lot of service measures I think they can bring a lot of benefits but they allow also so many additional features and functionalities to bring in. You never get out of it. And don't let get your service names too long. Organization tend to name their things with some of the hierarchical structures. The name given here it's like a representation of one of our names which we are using. It is still okay but if you now come back to the point number one that you get also a cluster name in addition to it you suddenly end up at 180 maybe 200 characters and this is getting too long. Also don't forget that in Kubernetes you have the limit of 253 characters or something like that. Don't crowd too big. Wrapping the things up. Don't forget the HAA flag. Linked is by default not HAA deployed. The custom regus chief leg is life saver. It's awesome. You don't need to do something with some weird proxies or do some mutating webhooks whatsoever. The admin part is great if you want to isolate the administration of the cluster and some certain components in it. As I said we have a highly regulated environment so have a certain or dedicated admin access to some of the tools is actually requested. So you cannot just expose it to some other port 8080 or something. For the cluster Iqini use the proxy CPU on the memory request and limit. And don't forget to do the link at EIO inject enable it. Our approach in the end was that in pre-production production we have it actually for all the namespaces directly set while in the development test we did it per deployment. Again, take care about a lot of other things. Link at the upgrades or service match upgrades itself are not that easy or can break a lot of stuff. From a security point of view, link at the requires net raw and net admin capabilities that's difficult to harden. And therefore link at the comes with our box and PSPs but it's something you need to discuss with security. Hoo, therefore, thank you very much. Thank you for your time. I'm happy to answer your questions. And I wish you a very great KubeCon.