 Hi, folks. Today, we're going to talk about Istio Service Mesh simplified beyond a single cluster. My name is Linsang. I'm a senior technical staff member at IBM. I'm also a contributor to the Istio project. Sivan. You want to introduce yourself? Hi, I'm Sivan Mossen. I'm a, yes, sorry, a little lag there. I'm Sivan Mossen. I'm a principal engineer at Google working on Istio and related service mesh technologies at Google. So we're going to talk about Istio simplified single cluster, a lot of work we have done in this area. And then we're going to dive into the background multi-cluster. And then we're going to show you some of the work we are working on together in the community, how we simplify multi-cluster experience for our user. And we're going to give you a really interesting demo. And then we're going to pass on to talk about what if you are running on VMs, running your service on VMs, how we simplify your experience in that scenario. So I don't know if you guys listen to our talk from Louie and Steve about Istio simplified on Service Mesh Con Europe early this year. So essentially, to summarize that up, just give you guys a background. We have done a lot of work in the Istio community to simplify Istio itself. So first, the mixer goes away. Well, the mixer functionality, you fold it into the psychoproxy. We have the injection system, Gailey, the config system goes away, and that function folded into pilot. We have the psychotinjector is also folded into pilot. And the last, we also have the node agent folded into pilot agent so that you don't need set apart security policies. We also have a Citadel function folded into pilot. So pilot is becoming IstioD, where the D stands for demon. And it really helps our operators who runs Istio control plane because they only have to worry about one single components. So great simplification work done by the community for the single cluster. Some of you might have already set up multi-cluster with Istio Mesh. And if you do, I guarantee you, you're probably very confused. Why are we having two models? We have this model called replicate control plane, which is what's on the left side. A lot of yous are actually very interesting that model because this model provides higher availability on the control plane. So you could have Istio control plane running in cluster one and cluster two, and you could selectively expose what are the services you want to expose to the other cluster instead of everything together. We also have another interesting model on the right side called shell control plane model, which means two of the cluster are actually using one Istio control plane. And in this case, all the services are shelled among the clusters. So there's a lot of confusing to our user, which one should I use, why we have to, why some requires core DNS setup, the other one doesn't really require, why the other one doesn't allow users to selectively choose what they are exposing to the other cluster. So a lot of confusing in this, the good news is we are taking a lot of effort in the community to unify the Istio multi-cluster model. So with this new model, if you have two cluster, like what's the diagram shows, you will be thinking about based on your requirement, whether you are going to need to run Istio D on just one cluster or whether you run Istio D also for high availability of your control plane on the second cluster. So it's up to you as a choice. You can also choose to run, whether you're on Ingress Gateway, both cluster or just running it on your primary cluster. So that's also a choice to you based on your control plane model and also based on your network model topologies. And then finally, you can also choose for the services on your data plane, you can choose what services are you exposing to the other clusters and you can also based on your needs to create the mirror for the services as needed. Saban, can you take over this? Yeah, so Lynn talked about some of these choices already, but basically we're trying to have a unified model here where you as the user are in charge of what control planes you run. So for each cluster, do you want to run a control plane in it for availability and redundancy or do you want to use fewer control planes so you have less to manage and use remote control planes? The network choices usually aren't really up to you. It depends on your network topology, but Istio works with kind of whatever network topology you have, whether everything is on one network and is reachable, all the pods can reach each other or if you need to go through some gateways. And as Lynn mentioned, you can also choose which services become visible to other clusters. And then there's also the sort of notions of identity and trust within the mesh where the different CAs in the mesh with their different routes all come together and trust each other. And so within the mesh, you can talk from one pod to another and everything works. You can also set up federated trust between meshes so you can have multiple meshes that talk to each other. We also talk in our multi-cluster model a lot about tenancy models and Istio really supports two different models here. One is kind of a namespace tenancy model where you don't worry as much about clusters and it's really just the namespace name determines sort of the team name. And that's kind of the build-in model where names are treated the same if they're in the same namespace. But you can also use clusters as your tenancy model where maybe one team owns a particular cluster. We recommend even if you do that, that you do actually still split out namespaces and make sure namespaces are unique. So just a little bit more detail on kind of the first two things the control planes and the networks. So control planes, it's really your choice, right? Where do you want to install a control plane? You can install a control plane in each cluster and those are sort of known as primary clusters. You can also choose not to install a control plane and just have a cluster that is remote to some other control plane. And you just set that up as part of installing Istio as part of our guides. You choose this cluster is remote and it's remote to this particular primary. So it's really your choice. You recommend that you have enough primary clusters that you have the availability that you need. So usually at least one per region. There's another model that actually recently has been coming out of the Istio community which is this notion of external control planes. And so when you have that remote cluster it's remote to some control plane. That control plane doesn't actually need to run in one of your clusters. It can run outside. It can be a vendor provided control plane. It can be one that a platform team is running for teams within your organization. However you want to set it up. But this lets you separate the management of the mesh from the management of the control plane and basically all your clusters become remote clusters to these external control planes. The network side as I mentioned Istio supports kind of whatever network setup you have. And the way we do this is when you're within the same network where pods can just talk to each other they'll just talk to each other and they'll call directly. And again you can run multiple control planes. You can run a primary or remote however you want to set it up. But we also support multiple networks where you want to run Istio on a different network in a cluster on a different network. But you want those all to be one mesh. And the way we do this is we use gateways. So you have gateways that can talk to each other and we tunnel the traffic through those gateways. So to the applications they don't need to worry or care where they're running if they're in a different network than some service they're calling. They just call the service and Istio makes it all work. So Lynn is actually going to talk through the sort of four base scenarios that we have documented on the Istio website. So go ahead, Lynn. Okay, great. Thank you, Savane. So let's talk about different topology models we have. So in this one it's multi primary, same network. So the first step you do is install Istio onto each of your cluster. And notice here on the second cluster you want to make sure you have the right cluster name and network it's the same. And the main thing is the cluster name. And then the second thing is create the remote secret so that it can do endpoint discovery for the remote cluster. And the way you do it is using create remote secret command. So that was it, the same network was really simple. If you do have a multiple network and you do want a multiple primary for higher availability of your control play. In this case, similarly as the previous you do install Istio, when you do remote create remote secret you want to do it on both of the clusters. So make sure the endpoints can watch the other clusters API server. And then you want to make sure you are setting up the East and West gateway to help you bridge the network so that the traffic can tunnel through the gateway and reach the other cluster. The last step is you want to also expose your user services to make sure service A can reach out to talk to service B. So we provide samples to allow you to by default I think it exposes all the services out on the gateway but you could potentially tune that to based on your business needs. The third model we have is you run primary and remote with the same network. In this case, similar as the previous one you would install Istio on the primary cluster. Notice here we just install Istio on only one of the clusters which we call primary cluster. And the second cluster we're just using the control plan from the first cluster. And then you would also the second step is set up the cluster to secret on the primary cluster one so that you can make sure the Istio decom discovery endpoints for endpoints on the cluster two. The third step is to set up the Istio West gateway to help traffic communication. In this case, you can see it also helps service B to reach back to Istio control plan on the primary cluster. The fourth step is expose Istio decom on the gateway. Sorry, I said a little bit early. Yeah, that was for the purpose of service B to reach out to Istio decom because it's in the same network. In this case, service A and service B are actually communicated to each other directly without the needing to hop through the gateway. The last but not the least model we have is primary remote or multi networks. In this case, you install Istio decom only on the primary cluster same as the previous one and same as previous one, you create the cluster two secret on cluster one for endpoint discoveries. And then you set up the Istio and West gateway same as the previous one. The first one step for expose Istio decom on the gateway it's also same, but you do have to do one additional step on step five to expose your user services. This is because your user service A and service B is not going to be able to talk to each other directly because you have multiple networks. So it has to go through the gateway. So you want to make sure your services are exposed on the gateway for services from the other cluster to consume. So what's really nice about these models as you can see is they're like building blocks, right? You can base on your requirements and your needs and you choose the building blocks that's needed to fix your requirements. So with that, we are going to be very excited to talk to you guys about our demo. Gosh, I've never set up something across different clouds. So we actually have four clusters on the left side. We have two clusters, cluster one and cluster two I on IBM cloud. In this setup on the left side, I believe they are primary, multiple primary and they are also multiple networks because we don't have flat networks on our cloud. So this is one of the model we talk about. On the right side, Saban has set up two clusters on Google cloud. And in this case, notice we only have is to be running on cluster one and there's no is to be on cluster two. So this is a primary remote and I believe this is the same network in his environment. So you can see where experimental different topologies across different clouds and we're setting up the single mesh among these four clusters. From an application point of view, it's the purposes to show you guys the multi cluster topology. So we decided to use really simple booking for examples. As you can see, we have product page and review version one running on cluster one I and then we have review version two running on two I and review version three have high availability running on Google cloud with cluster one and two. With that, I'm going to go ahead, share my screen and show you guys the demo. Okay, so we would like to show you our demo now. So is your IO go to multi cluster installation to set up multi clusters. The first thing we are looking at is config trust. So we want to make sure all the clusters are using the same root CA and each of the cluster would plug in the intermediate CA with this common root of trust. On my environment, I already have all the certificates and keys created for each of the cluster. Now I'm just creating the CA certs in Istio system namespace for two of the clusters. So the way Istio works is if you plug in your own key and certs, you need to make sure it's named the CA certs. So that essentially tells Istio not to generate self signed key and certificate. Now I'm installing Istio on this cluster now. And this is my configuration YAML. You can see I'm installing a primary cluster in this environment with my network and my cluster name. And on to the next cluster, two I similar plan except I'm having a different cluster name and a different network name. So let's go ahead and install that as well. So this basically installs the default profile of Istio, which you can see it's essentially just the ingress gateway that comes with the default and also the Istio D control plan. We talked about earlier, we're down to one single control plan components Istio D. So now we're looking at what next should we be setting up in this environment. So the model we're installing is a multi primary different networks. So we're going to need to set up endpoint discovery. We're going to need to set up Ist and west gateway. So we'll set up our finish on cluster two now. Let's check how the installation did. As you can see all the components reaches running within a minute. Now we're going to set up Ist and west gateway onto each of the two clusters on IBM cloud. The key thing on this configuration of the gateway YAML for Ist and west gateway is the cluster name and the network name and also the port number. What are the number of ports you are exposing on the gateway? So now we're going to install the Istio, Ist and west gateway onto the clusters. So Istio Cuddle install command, it used to be called Istio Cuddle manifest generate and then you apply Cuddle apply. So now we actually have a single command that takes Istio operator YAML and then you can apply things. So I really like that simplification. Now we're going to do the same thing on the cluster two to install the Ist and west gateway. Okay, we got on this install. Let's look at the next step. You would run create remote secrets and generate the secrets that YAML file so that your cluster two can consume that and also passing that YAML file to serve end. So he could use endpoint discovery on my cluster from Google cloud. So now we got remote secret generated for both of my two cluster. Now we're applying on the second cluster we're applying the first cluster secret which enables the second clusters to look up to query the API server on the first cluster to do endpoint discovery. Upon the secrets are applied, you would see the remote Istio remote secret with the cluster name as part of your secret in the Istio system. So as you can see on both of my cluster I have consistent secrets. The next thing we're looking at is to expose services so that the other cluster can consume it. So we're going to expose the services on the cross network gateway. Savane, over to you. Great. So I'm just going to show what we set up in a very similar way over on GCP. So I have two clusters. They're each running, well, actually one of them is running primary ones remote. So first let's look at the primary one. So the primary cluster has that same thing that Lynn just set up on IBM cloud. It has an east-west gateway, an ingress gateway and IstioD. And then on cluster two we just have the ingress gateway and IstioD. We don't have an east-west gateway because we're not running the full control plane there. IstioD in that cluster is actually just the CA. It's not running the XDS server. So it's a little bit confusing but it's actually not running the whole control plane and we're actually working on removing that out. And so you can see the same thing on the service level. There's this IstioD-remote service that is actually what the remote cluster is using to talk to the server. And the installation is very similar to what Lynn showed as well. We just have these Istio operator files that have cluster one and cluster two. On cluster two, the difference is this is actually set up as a remote cluster. The profile is remote and it has a remote pilot address. That is the address of the east-west gateway running in the primary cluster. And so basically that's it. We have two clusters on GCP all set up in Red Day Go. So back to you, Lynn. Okay, great. And now Savannah gave me his secrets. So I apply his secrets on both of my clusters so I can do endpoint discoveries. Now you can see all the secrets are applied. So I have each of my cluster, I have three of the remote clusters. So now we're going to look at deploy booking for. We talk about in the diagram, we're deploying booking for with version two, version one of the review on the first cluster in IBM Cloud. So we're going to need to expose the booking for product page on the gateway so we can reach out to booking for product page. So let's check it out. So the gateway configuration is very simple. Basically we're exposing booking for HTTP AD on these different URI passes, slash product page is our landing page. So I have the gateway information early on. So one, two, three is the IP address of my gateway. So as you can see, as a visit booking info, I should only see review version one, which is no star. So now we're going to deploy review version two onto cluster two I in IBM Cloud. This is just a review version two deployments and service looks like. Now if I ever hit the refresh button, you can see, now it's wrong robbing between two of my cluster. So certainly my product page on first cluster reaches out over to you, Savannah. Yeah, and so what Lynn was just showing, we have the product page working with just V1 and V2 and now we're actually gonna go install V3 over in Google Cloud. And this one has red stars instead. It's a big upgrade. And so really similar to what Lynn just showed, we have reviews deployment, the reviews service and actually the rating service because reviews actually calls back to ratings which is running back in IBM Cloud in cluster one. So the path is actually coming into ingress on the IBM Cloud side, going to reviews over in Google and then going back to IBM for ratings and then back out. So it's actually bouncing back and forth between clouds. But as the app owner, I don't actually care about any of that. I just deploy and now I have red stars. So this is actually a multi-cloud application that is running between the two. So it's pretty exciting. And as you see it round robins between, it round robins between the various options. I think that's it for our demo. Yeah, I'm gonna stop sharing and Savannah, what you do to finish up the rest of the slides. Yeah, let me talk a little bit about VMs now. So what we showed is how you can have a bunch of different Kubernetes clusters all connected but we know our users have a lot of services on VMs and they want all those same benefits. And so what Istio has this model that we've called VM expansion or master expansion and it's really, it's letting you attach VMs kind of the same way that you attach clusters. And that's actually how we kind of have a unified model here. So I can run what we call a workload group which is really just a group of VMs, similar to kind of like a deployment, but it's loosely defined as just any VMs that you wanna have as a group. And they can use that same East-West gateway. They can use IstioD as if they were a remote cluster and they get registered in the API server and then you can use those VMs as backends for your services and actually have them call into your other services in a mesh. So it all works as one giant, big mesh. So let's look at some of the improvements that we've made recently to this because we've actually had this for a while but it hasn't been the easiest to use. The big improvements coming in Istio 1.8 are around improving how you register those VMs and how you create them. So we're actually introducing two new commands I actually realized I left out the X for experimental and these are experimental commands but you can create a workload group really easily. It generates the M for you. And then based on that workload group, you can just ask to configure an individual VM and it'll actually spit out all the configuration files that we need to set that up. So you can just copy those over and run Istio on the VM and it'll be set up correctly. Some of the other improvements we have are we have DNS proxying, which before this you had to go and manually change your DNS configuration for your VM either changing it to point at one of the Kubernetes clusters, DNSers or setting up your own local proxy that forwards things for the mesh to the cube one and otherwise keeps them in whatever you have for your VMs. Basically all that is simplified. As part of running Istio, we actually run a DNS proxy that will automatically resolve mesh names to IPs and otherwise forward to your existing DNS infrastructure. So that greatly simplifies setup. We also have changed how identity bootstrapping happens. In previous versions, there was a very complicated setup with certificates and I believe the 1.71 you could only really have one VM attached to a cluster didn't really expand very well. This actually uses the existing Kubernetes token infrastructure to bootstrap them. So as part of that workload entry configure this will actually automatically be generated for you. We generate the token, you copy that to the VM and then that bootstrap certificate, which from then on uses just the normal Istio certificate infrastructure. We've also added auto registration so you don't have to worry about manually registering each VM as it comes up, which doesn't work well if you have auto scaling and we also have help trucking coming into you. So these all make VMs kind of as easy to use hopefully as pods once they're all in place. And let me give kind of a brief example of how this works. So just like in multi cluster you set up your primary cluster once you have that in place you can register a workload group. And so again, we can use these new experimental commands here at least I included the X like I should have. So this says I want to create a new thing called hello VMs in the sample namespace. Here's some labels and then that generates a YAML file. And then I can just apply that to stick that in the API server. And the reason I do that is when I want to generate config for the VM itself I can actually reuse that existing one. Actually you can call it by name here I'm actually using the file but if it's in the API server you can actually just refer to it by name. You call a workload entry configure with that name and namespace and it generates into the config all the artifacts that you need to run the VM. And basically that's it. You copy those to the VM and you run Istio and the VMs will then connect through that East West Gateway to Istio D they'll be processing DNS for you and all the requests between the entire mesh can now go to those VM endpoints and VM endpoints can call into the mesh. So all the networking is just set up and working. And that's basically it. So we've walked through a lot of different parts of Istio in this and here are kind of the key takeaways. So we've been working really hard to make it easy to build a mesh out of multiple clusters. It was kind of doable before it was definitely not easy. Now it's actually pretty simple. It was even amazing to me and maybe to Lynn too that we were able to get a multi-cloud mesh multi-cloud multi-cluster mesh with remotes and multiple primaries and all these regions and different networks, some things on flat network some in isolated mode and they're all connected. And to add an endpoint to a service was just a single YAML file on one of the clusters and instantly, product page was able to route to that. Istio took care of all the routing and management of that. You don't have to go set anything up. You don't have to set up all these kind of crazy rules and things that just it hands up for you. And it's all secure and with MTLS and authorization policies and all that good stuff. As part of this, you can choose where you want to install those control planes. Obviously it works across multiple networks. And then the other takeaway is that this is works great for Kubernetes but we're also making it really simple and easy for VMs as well. So those will also be able to join. So thank you very much. Lynn, do you have any other closing remarks? If not, I think we're demo other talk. So we appreciate it and thanks for listening. Yeah, same as you. I'm super pleased to actually set up the demo with you and everything runs just smoothly without hitting any major issues. It's amazing. Well, thanks you all. Hopefully to see you guys at the conference. I think we're live. Can you hear me, Lynn? Okay, so the observability work we haven't, okay, great. Yes, technical issues with live, not cute right here. Observability isn't totally multi-cluster aware yet. We're actually working on that. I think hopefully in the 1.9 release of Istio we'll be able to have some better stuff there. It's possible to set up but it's not easy sort of to the, we want to make things easy not just possible. Same here, it's not yet that easy, right? We don't have built-in multi-cluster dashboards. You kind of have to set that up yourself. You can actually have a single Grafana that is pulling from Prometheus in multiple clusters or you can have a single Prometheus pulling multiple cluster sources. There's a bunch of ways to do it. You don't yet have that all set up and I think that's something we want to go on. So I can maybe look at the next question. So there was a question about federation support and inter-accounts and intra-accounts. I'm not actually quite sure what that is meant to mean. Federation is something we want to continue also making easier. That's also not quite as easy as we'd like it yet. If you want to follow up on exactly what you mean about inter-accounts versus intra-accounts, then you want to answer the one about 1.8. Yeah, the hope had been that this 1.8 would actually be out before this talk but we had some last minute fixes that had to get it so it's not quite as yet. Yeah, in our demo, the East-West gateways were exposed publicly on the internet and they didn't rely on mutual TLS to authenticate and encrypt the connections. There are other choices. If you want to set up a private BPC or something like that, in our demo, there were a couple of questions noticing the problem with the demo. I think that, oh, there's a question, another question about the network setup in this case. So on the, I go into the internet publicly but protected with Istio security and then Lynn can talk about what's the IBM side of that looks like. The network setup on the IBM side, how's the network setup of that? Okay, should we talk about the failover case? So basically there's this feature called locality of where Lou balancing in Istio that solves this case for you. So the question is about how do I do failover? So if I have the same service in two different clusters, they're mirrored, how do I make sure it stays local when it's there but then fails over to the other cluster if needed and that's what locality where Lou balancing is actually for. So you can keep it in the local cluster and only send it over to the other cluster if it does rather than just blindly, ground-rope-ing between them. You can look at the Istio documentation for details on. Sorry, we're having some lag between our video streams. Then you wanna take the next question about upgrades. I think we're just about out of time there and we'll try to answer the rest of the questions offline.