 Hello, everyone. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Annie Telasto, and I'm a CNCF ambassador, as well as CMO at Vision, and I will be your host tonight. So every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer all of your questions. You can join us Tuesdays or Wednesdays to watch live. This week, we have a great session coming up with Lynn here with us to talk about LinkerD 2.14. I'm very excited for this one. As always, this is an official live stream of the CNCF, and as such, it is subject to the CNCF Code of Conduct. So please do not add anything to the chat or questions that would be in violation of that Code of Conduct. So basically, please be respectful of all of your fellow participants as well as presenters. But with that done, I'll hand it over to Lynn to kick off today's presentation. Okay, thanks. Glad to be here. We are gonna talk about LinkerD 2.14. I have some slides to go through at the beginning to explain what we're gonna talk about, but we are going to try to keep the slides quick and then go into the demo and then still have time for questions. That's me, that's how to reach me. Email's good. You can reach me as Flynn at basically any Slack associated with any CNCF stuff that as far as I can tell. So yeah, the agenda, we're gonna go over some of the headline LinkerD 2.14 features and then we're going to do a demo of multi-cluster phases as opposed to our usual phases demo. Hopefully that'll work. Okay, 2.14. 2.14 is very new. I wanna say it came out like three weeks ago, maybe, headlines. We have flat network multi-cluster, which we're gonna talk about, which also supports workload identity crossing clusters. We did some work with Gateway API conformance, which is kind of nice. We had a timeout support to HTTP routes and there's a lot of bugs being fixed mostly that we're not gonna go into today. Hopefully we fixed more than we introduced. So, you know, this is software, after all. Flat network multi-cluster. If you're familiar with multi-cluster from 2.13, the way that worked was that it would funnel traffic from the workload in one cluster through a Gateway in the other cluster to the workload in the other cluster, right? And this worked really well as long as you have IP connectivity between the cluster and the Gateway and such, but it does have two caveats. The first one is it adds some latency because it has to go from the workload to the Gateway and then from the Gateway to the other workload. Typically, this is not a major deal. Linkerty is really fast, but it's still there. The much more significant caveat was that the identity of workload one basically gets lost. Workload two sees the identity of the Gateway, not workload one. And that can make life really complex if you're trying to do authentication and authorization across clusters, which, you know, this is a good thing, right? People would like that. So the basic solution that we adopted with 2.14 was just get rid of the gateways. This is great because it preserves identity everywhere and all the MTLS goodness that you're used to and Linkerty just works and all the policy stuff just works and it lowers latency and it's really great. And there are a few things that you need to think about. I'm not actually gonna go over everything word for word on this slide. It mostly boils down to the things that you had to do to establish trust across the two clusters, you still have to do. And in addition to that, you have to be running in a network that allows a pod in one cluster to talk directly to a pod in another cluster using the pod's IP address. This implies that your clusters must use different cluster cider ranges because otherwise they'll step on each other and nothing will work. It also implies that it is often a good idea to give your clusters distinct trust domains, distinct cluster domains as well. Just because it can be easier to keep track of which identity comes from which cluster when you do that. If they don't all say food.service.cluster.local but instead say food.service.east or food.service.north or whatever. That is not necessary. And in fact, the demo I'm going to show we're not going to do that because the demo I'm gonna show is mostly talking about routing and not authorization and authentication. But you know, good thing to bear in mind if you're trying to do this in anger. Oh, I'm sorry. One other thing that I wanted to mention is the there is still a command that you will run called linkerd multicluster link that explicitly tells Linkerd hey, these two clusters should be joined. That command is still there with a tweak to it. So you'll see that in a little bit. Okay. By the way, let me also point out that at any point during here feel free to jump in with questions. You can put them in the chat and he will relay them. And otherwise I'm just gonna keep going and assume that everything is perfect. Sounds good. Even though everything is never perfect but we'll just assume. Okay. Gateway API. As of linkerd 2.12 we started using the Gateway API as sort of the core mechanism to describe classes of HTTP traffic very much including GRPC. And this is a thing that's kind of a big deal. We say this every time that we expect it to deprecate SMI and service policies and things like that. We still do not have a timetable for that. You are still free to continue using those mechanisms from older versions of Linkerd they will still work. But Linkerd 2.14 is actually conformant with Gateway API version 0.8.0's mesh profile. This is kind of a big deal. The whole conformance profile thing I think. Actually, you know what? Let's just talk a little bit about the conformance profile thing. The Gateway API when it first came out was very much built assuming that you were talking about Ingress traffic not service mesh traffic. So in particular early versions of Gateway API had this idea that if you were going to run certain conformance tests that you also had an Ingress controller. Linkerd is a service mesh. We do not ship with an Ingress controller. And this meant that there was no way for us to be compliant sorry for us to be conformant with Gateway API. In Gateway API 0.8, which came out in very late August. So about a month ago in 0.8, Gateway API introduced the concept of a conformance profile which describes a set of features within Gateway API that should be tested. And also gives you a way to talk about, oh yeah, we're conformant with this but not this other thing. And that finally gave Linkerd the ability to say, oh yeah, we'll go and run conformance tests against the mesh profile. We will test all the meshy things in Gateway API against Linkerd and that passes so we are in fact conformant with Gateway API as of Linkerd 2.14 at least with the mesh profile. Do not try to use Linkerd as an Ingress controller. It will not work. So a big part of that is the API group that you see in things like HTTP routes. In Linkerd 2.12 and 2.13, you would see HTTP routes only in the policy.linkerd.io API group. Whereas in 2.14, you can also use the official gateway.networking.kds.io slash v1 beta one APIs. And if you are doing new things, you should probably use the official one with one extremely important caveat that we will talk about in a minute. That said, policy.linkerd.io is not going anywhere and you can still use that HTTP route for the foreseeable future. It's also worth pointing out that if you are running an installation where you have the official, like suppose you've installed a Gateway API, Gateway controller as well as Linkerd, you will find that you have both HTTP routes to policy.linkerd.io and HTTP route.gateway.networking.kds.io. And unfortunately, you may have to explicitly spell out the whole thing when you're doing a kube control command because kube control can very, very easily get confused as to which one it should be talking about. We might actually end up trying to get a bug fix into kube control for that because it's kind of awkward right now. Okay, timeouts as I make sure. Okay, I see some hellos and a lovely Linkerd for the win. Thank you, but I do not see questions yet. Okay, in Linkerd 2.14, we have added support for timeouts and HTTP routes with the timeouts stanza here in the HTTP route. Very exciting. There are two kinds of things that you can see in the timeout stanza. You can see requests. There is a question immediately when we check that there's no questions. There comes one, which is good though. Thank you so much. So there's a, why is Movie Cluster so important? That is a lovely question. Let me finish talking about Gateway API and then come back to that before the demo because I think that will make a little bit more sense if we talk about it with the demo. If I talk about it and it doesn't make sense, then ask your question again if I don't answer it correctly. Okay, we were talking about two kinds of timeouts. There's request, which is an end-to-end timeout. There's also a backend request, which is just the part talking to the backend and we'll show this in a little bit more detail shortly. The syntax for the timeout is basically like a go time.duration, but you're not allowed to use floating point. So 1H30M is fine for an hour and a half, 1.5H, not fine. Don't do that. The gory details of all of this, and let me tell you some of the details are very gory indeed, are in Gateway Enhancement Proposals 1742 and 2257. 1742 talks about the syntax of the timeout stanza and 2257 talks about the details of the syntax of the thing that you put in as a timeout. And there are links there if you have a really unbreakable case of insomnia or some night and you really wanna go and read something that will put you to sleep. Okay, to explain a little bit more about what the two different things are, if we have this scenario where our client is talking to workload one and then workload one has to talk to workload two, then timeout.request covers the entire thing from the client all the way through to workload two and timeout.backend.request only covers the chunk where the workload one is talking to workload two, the backend services there. Usually, you would kind of only, like it doesn't really make sense to set backend request to something longer than request. In fact, that's not allowed by the validators. And in a lot of cases, it doesn't really make sense to use backend request unless you've also configured retries or something like that. So most people are probably just gonna set request and be done with it, but the functionality is there if you need it. Now, you remember I said that you should be using the official API group with one very important caveat. Yeah, this is the very important caveat. You must use the policy.linkardy.io HTTP route if you want to use timeouts in an HTTP route because GEP 1742 didn't quite make Gateway API 0.8. It was right down to the wire. It didn't quite get in. And so it will be a part of Gateway API version one which should be happening real soon now. But for now, if you want timeouts, you have to use the liquidity-specific version. Okay. It is now time to talk about the demo. The demo architecture we're gonna use is the typical faces thing that y'all have probably seen if you've seen me on any one of these where we have a GUI, which is a single-page web app that talks to a workload called Face. Face talks to a workload called Smiley and a workload called Color. The Smiley workload is supposed to return this grinning face and the Color workload is supposed to return the Color green. The Face workload puts the two together and hands it back to the GUI, which should show you a nice grid of grinning faces on green backgrounds. Unlike many iterations of, or many demos involving the Face demo, in this one, we actually expect to see that from the start because we're not demoing reliability stuff. We're gonna do a multi-cluster demo. It's also worth pointing out we do have a Smiley 2 workload, which returns hard-eyed Smilies and a Color 2 workload, which returns the Color blue. What we're gonna do that's different for this demo is we're splitting that into three separate clusters. So instead of having them all in one cluster, we have Face in the north cluster named because it's the one that's taking all the north-south traffic. We have Smiley in the east cluster and Color in the west cluster. We're actually gonna start off with a Color workload in the north cluster as well so that we can show some stuff that's kind of interesting. To get back to the question about why in the world would you want to do this? Sorry, I got very distracted by the comment that says wow and I'm wondering what the wow is for. Anyway, the reason you might want to do this is for this particular kind of demo, what we're doing here is showing something that we've been seeing a little more often as time goes on, which is that back in the, I would call them the bad old days except they're not really all that old. People tended to set up these gargantuan clusters, run everything in one gargantuan cluster and then they would just go through and throw in namespaces so that their developers could have a namespace to play in and the namespaces were providing isolation. Then people started figuring out, oh crap, if we have everything in one huge cluster and our huge cluster has a problem, that is very bad. So then they started thinking about maybe we should run multiple clusters for redundancy and almost as soon as you start thinking about multiple clusters for redundancy, you start realizing we could use clusters for isolation like we used to use namespaces where instead of having some gargantuan cluster, we can give each of our developer teams their own cluster and it will be much, much smaller and that way we can isolate the clusters from each other and if one of them goes down or if a developer screws something up in their cluster, then it doesn't affect the rest of the application. So that's the use of multi-cluster that we've been seeing recently, we've been seeing people talk about recently and so that's the reason why the phases multi-cluster demo is set up this way. And again, Sunil, if that does not answer your question and if I'm pronouncing your name well enough that you recognize it, I hope so. If that does not answer your question, then toss another comment in and we'll see if we can't clarify. Okay, so there is source code for this demo available in this repo in the sneak peak 214 directory. With a lot of demos that I do, I will start with an empty cluster and then build it up. I am not going to do that on this demo because this demo involves three K3D clusters that are set up on the same network and have their routing tables modified so they can all talk to each other and no, we are not gonna demo that because it's mostly just crazy absurd K3D black magic. So if you want to take a look at that and I would encourage that, especially if you play with K3D, you can find the setup scripts in here, create cluster.sh and setup demo.sh are the really relevant ones there. And they're kind of ugly, just be warned. All right, it is now demo time. Let's hope the demo gods are with us. Okay, and yes, I am in fact, recycling a demo from the service mesh Academy which is why it doesn't say welcome to cloud native live. So sorry. Once again, if you're trying to run this at home, make sure you have K3D working and then run create cluster to create clusters.sh and set up demo.sh. And you will end up with three clusters which have LinkerD installed and I think emissary and the faces demo and all the stuff that you need. And you should read over them and it'll be kind of horrifying but it'll hopefully be educational as well. Yeah, there's an audience question as well. And by the way, Sunil said, excellent, thank you. So we'll cover it there. And then Ashish asks, can we do that in managed cluster from cloud? It depends on your cluster provider. I chose not to do this in the cloud so that I didn't have to fight with a cluster provider. The question that you need to bring to your cluster provider is, is there a way that I can set up two clusters or more such that they are sharing a network and can route directly from pod to pod? I'm pretty confident, like I'm pretty confident Amazon can do that but I might be mixing up Amazon and Azure since they both start with A. I don't remember about Google. I've run across at least one of the smaller cluster map providers that just full stop cannot do it. So yeah, it kind of depends. Worth pointing out the older style of gateway based multi-cluster does still work with 2.14. So if you find that you just can't do that, then you can still go back and use the gateway mechanism too. All right. So let's take a look and see our Faces demo is running. This is good. It's actually giving us grinning faces and green backgrounds like it's supposed to. That's a rare sight. Okay, so let's go take a look under the hood and see what's going on here. I already said that we're doing this with three clusters rather than one. We have clusters named Northeast and West. We already talked about which one was which, but just to make that clear, North has the face workload and is where all the North-South traffic comes. It also has an instance of color. The East cluster is the only place that the Smiley workload lives and the West cluster only has color workloads. We can observe with kubectl cluster info, carefully managing the context that we're using here. We can observe that we have API servers in three different places. So we have North, East and West are all at different ports. And I have set these all up so that I have one kubectl file with three different contexts in it so that it's very easy for me to talk to whatever cluster I want to. I'm not going to tell you how many times developing this I typed a command with the wrong context and got very, very confused except that it was a large number. I'll tell you that. I've already done the cluster linking but let's take a look at how that works under the hood. This is an example of a command that you would use to link clusters together. Specifically, this is the one that links North to East. Pay very careful attention to the contexts on the commands in this pipeline. So we run linkerd multi-cluster link itself in the East contexts. Linkerd context, East multi-cluster link. That tells the East clusters linkerd installation to generate a set of resources that we can apply in some other cluster to link that other cluster to East. We also do gateway faults to tell it we are not doing gateway stuff. We're doing flat network stuff here. And of course we set the cluster name to East. And then we apply all of that in the North context. So because we want to tell the North cluster how to talk to the East cluster. Oh, did we lose audio? Yes, I don't think we can hear you. Yes, of course. No, not hearing anything still. Let's see, no, not either, no. But this is how the audience, you can tell that this is definitely live, by the way. So that's always great to see in that way. And I guess Ahmed confirmed, yes. Oh, no, I can hear you. Sergi, are you saying that you can hear me? Or do you can hear Flynn? I can't hear Flynn at least. You can hear me. No, I can hear you. Now I can hear you, perfect. Okay, this happened to me once before and I'm glad everybody was watching. Okay, okay. Perfect. Let's just hope that doesn't happen. Thankfully, that was not caused by a Kubernetes cluster that was caused by my USB interface disconnecting for some utterly bizarre reason. It happened. It shows that we're live and we are truly doing this as we go. Absolutely. Cloud native live, not cloud native recorded. Okay, where was I? I was saying that links are unidirectional. So this is a link that will permit north to talk to things in the east cluster, but it will not let east talk to north. For our demo right now, we don't really care. We don't need for east to be able to reach back into the north cluster. So this is the only way I did this. I have it set up so that north is linked to east and to west, but I didn't do the reverse links. It's good if I wanted to, but there's no need to. We can get a lower level look at all of this by, oh dear, I typed examining. That should be an examining, well, clearly there will be a push made to this repo shortly. We can go and actually look at the link resources that were created by the multicluster link command. And we see that in the north cluster, we have one for east and we have one for west. If we look at the resource itself, we get a lot more information. The interesting things in here are, for example, there's a secret that has a bunch of credentials for this cluster. You should perhaps make sure that those secrets are not readable by everybody that you don't want them to be readable by. This is also in the linkerty multicluster namespace. So if you protect that, then that's a good way to go. You can see that there's no gateway involved here. And you can also see that it's got a thing set up to look at labels in the other cluster. We'll talk about that in a minute. Another thing that we get to do here that's kind of fun is that we can use the same trick to look for links in the east context and the west context and see that there are no links in those clusters. They are just kind of doing their normal east and west things and not doing anything special with multicluster from their point of view. Again, we could link them back. It wouldn't help this demo so I didn't do it, but it works fine if you want to set up bi-directional stuff. All right, quick aside, if you are familiar with linkerty 2.13 multicluster with the gateways and everything, I can ask linkerty multicluster to show me all the gateways and it will come back and say, nope, no gateways, life is good. I'm pretty sure that 2.14 still has the interesting characteristic that if I ask the east cluster for its gateways, it will hang for 30 seconds because the infrastructure that permits it to know instantly that there are no gateways is not there. That will be fixed in a later linkerty version if it's not fixed in 2.14.1, actually. The way that this works without the gateways, just for the record, is that the control plane in the north cluster talks to the control plane in the east and west clusters and gets information about the services in the mesh in the east and west cluster and figures out which ones need to be mirrored across and we will look at all that pretty much right about now, actually, I believe. So let's take a quick look at what's running in the north cluster. If we look at the workloads that are running in the faces namespace, we can see that we've got some color workloads. We've got two color workloads, actually, two replicas. And we've got a couple of replicas of the face workload and we've got the GUI, but we do not have a smiley workload at all. However, if we flip back to the web browser, we are clearly getting smiley faces. So the way that this works is, let's take a look at the services and the interesting thing you might notice here is we have a smiley service and we also have a service called smiley east. So we can go look underneath at some of these things and we will find that smiley east is a mirror in the north cluster of the smiley service from the east cluster. This is why it's important not to name your clusters and services the same way. When I did this for service mesh Academy, I had clusters named face, smiley and color and then I ended up with services called smiley smiley which is very, very difficult to talk about with a straight face. But if we go and look in the east cluster and ask it for services in the faces namespace, we can see that there we do have the smiley service and we have smiley two and we also have the workloads. If we take a closer look at that smiley service, we will find that it is marked for exports using the remote discovery method. If you're again, if you're familiar with 2.13, you might be used to seeing mirror.linkardy.io slash exported enabled rather than remote discovery enabled is what you do if you want to use gateways. If you don't want to use gateways set it to remote discovery. So the fact that that was marked in the east cluster for remote discovery means that the mirror got created in the north cluster and yes, it's currently named as service name cluster name which is why you would get smiley east. And if we take a quick look at that at its end points specifically in the east cluster we can see that it has two end points. This makes sense. There are two replicas, there should be two end points. If we ask the end points, ask the north cluster for the end points of smiley east, we will find that there are no end points of smiley east which seems bizarre. The reason it works is that the linkardy control plane is actually keeping track of those end points with the remote discovery rather than going through and constantly updating at CD every time it changes we just keep track of it in the linkardy control plane but we can use the diagnostics end points command linkardy diagnostics end points to ask for the end points of that thing using its fully qualified name and port number. If we do that, we find the same two end points with the same IP addresses which is good because it's supposed to be direct pod to pod communication so they should have the same IP address. This is why you need to have different cluster ciders. Finally, I should point out that the smiley or sorry, the face application does not know how to talk to smiley east. It only knows how to talk to smiley when it's going and when the face workload is trying to fetch a smiley, it just goes to the smiley workload. And the way that this works out is we also have a service in here called smiley. If you take a very careful look at this, we'll put Annie on the spot. Hey, Annie, do you notice anything weird about this service? I'm always on the spot so I can't. You can just know all the audience if you can see if you can spot it. There's a little bit of a delay with the audience, but yeah, it'll be fun to see if anybody comes up with this. I'm gonna go ahead while we wait for the audience to chime in, but if you look very carefully at this service, you'll realize there's no selector in it. If there is no selector in a service, it can never match any pods and so it will never have any end points. And so we have here a cluster area service in our north cluster that is literally set up so that it's not possible for this service to talk to anything. And its only purpose in life is to have this HTTP route associated with it. This HTTP route says anything directed to smiley the service redirected immediately to smiley east. And that's the thing that permits the whole demo to work. We're capturing, we're basically doing just a forward from this smiley service to smiley east. Also worth noting that if you read very, very carefully, you would have seen that that HTTP route was using a policy that linkerd.io, HTTP route, not a gateway.networking.kates, gateway.networking.kats.io. That is very hard to say. We're not using that kind of HTTP route and the reason is that there's a timeout in there that we're going to show off in a bit. Also, I'm sorry, I should have mentioned this before. Only v1-beta-3 has timeout support. If you have older v1-beta-2s or something like that, then timeouts will not work until you switch them to v1-beta-3. Everything else is the same. You can literally just change the API version, it'll be real easy. Okay, we already talked about why you have to use policy.linkerd.io for timeouts and HTTP routes. All right, another important caveat. You cannot use, for example, an HTTP route in the north cluster to direct traffic to something in the east cluster, where there is also an HTTP route that will do further things with it. You also can't do that in the same cluster. You can't stack HTTP routes. The reason for that is that the first route it sees, linkerd.io uses that route, makes a decision and goes straight to the endpoint. And so the second and subsequent HTTP routes never have a chance to do anything. That should be true in every mesh that is compliant with the Gateway API mesh conformance profile. All right, let's mess with timeouts a little bit, shall we? If you flip back over to our web browser, you see how some of those cells are kind of fading out. The reason for that is that it's basically just taking too long for, it's taking too long for the face workload to get an answer back, or it's taking too long for the GUI to get an answer back. And so we want to try to use a timeout to make that a little bit quicker. If we come back over here, we can do that for the smileys anyway, just by scrolling down here and changing the timeout to 300 milliseconds. 300 milliseconds, because that worked pretty well the last time I tried this demo. If we flip back to the browser now, then you should see, there we go, fewer faded out cells, but you also see those counters appearing where the timeouts are firing and we're seeing some things going on. And exercise to the reader, why do we not get, why do we get any faded out cells at all? If anybody hanging out in the audience wants to comment on that, feel free. Another thing while y'all are thinking about why we're not getting, why we're still getting any fading cells at all, why we still get things that are taking too long. We're not limited to just directing all of the traffic over to the other cluster. We can also split traffic between clusters using HTTP routes. This is why we have a color workload in North and also the color workloads in West. So what we're gonna do right now is we're going to start by mirroring the Color 2 service from the West cluster into the North cluster. The clusters are already linked. I have not marked the Color 2 service for remote discovery yet, so I will do that now. And the moment we do that, if we come over here, we will now see that Color 2 West service. Oh, right, my mouse was in the wrong place so I couldn't click on that. Excellent. If we check out the endpoints for that, then we can see that we have endpoints for it as well. So we should be good to go. The Astute Observer also will note or possibly note that when we looked at the endpoints for Smiley East, we got endpoints in 10.23.1 slash 24. And for Color 2 West, we're getting endpoints in 10.23.2 slash 24. Again, different cluster-siders, very, very important. Ahad, no, that is not the cache. There's something else going on. But a valiant attempt. What was I doing that for? Oh, yes, sorry. I was also running the command to look directly at the endpoints in the West cluster and to note that, yes, they are the same as the ones that we get for Ligardy Diagnostics endpoints. All right. So if we do that, we can add an HTTP route that will split traffic 50-50 between the color service in the North cluster and Color 2 West also in the North cluster but pointing to the Color 2 workload in the West cluster. So if we apply that and then go back to the browser, you should see half of these things show up with a blue background because Color 2 returns blue rather than green. It's always nice when the demo actually works. Okay. We can do something that's kind of entertaining at this point too. So right now we're getting some, half of our color is coming from the North cluster and half of it is coming from the West cluster. If we edit the HTTP route and just delete that first backend ref, now 100% of our colors should be blue because they should all be coming from the West cluster. On the one hand, this is pretty pedestrian. All we did was we did a canary and we went, yeah, the canary's good. So we'll go ahead and flip everything over. But if you think about it, we actually just migrated the Color workload from one cluster to another one without doing anything with the applications or anything like that. So one of the things that's really, really cool about this, oh, and let's just go ahead and delete it to prove that it's gone. It is gone. It's still working over here. So one of the things that's really cool about some of this multi-cluster stuff is that you can actually migrate between clusters by deleting things. And if you can figure out how to get this set up so that you've got, say, clusters and different cluster providers, then one of the really fascinating things you can do, this actually holds true for the older gateway style too, right? One of the really fascinating things you can do is you could set up your application so that half the traffic is going to GKE or GKE and the other half is going to AKS. And then if you decide that Google is annoying you one day, just turn it off. Instead of making a big production out of the migration, which is kind of cool. Obviously there's a lot more going on beneath that, but once you have it all set up, it's pretty cool that you can just go through and do that. I'm gonna go ahead and show that, yes, those pods really did vanish by re-running that trend. So quick sampling of pod-to-pod multi-cluster with Linkerdue 2.14. We do have, I think, a certain amount of time for questions. In the meantime, I'm gonna point out that the reason we still get some fading cells here is that the face and smiley workloads still, we didn't put time-outs on them so they can still go through and take a really long time up to something like a second and a half. And that's the reason why we still get fading cells in here. So yeah, any other questions? Let's see if any comes in right now, no, but great that we got that mystery solved now. So that's always nice. But yes, we have time for questions. So let's see. I'll leave the link up there in case anybody wants to see it as well. Perfect, always good. And actually connected to that, while we wait to see if audiences any questions, what would be like good resource that everyone should jump on next if they want to learn more or would be this link here or what would you kind of advise people to go? I have several places that I could advise people to go. Perfect, even better. So Service Mesh Academy, Buoyant does this every month. The next one is on October 26th where I will be joined by Regina Scott from Argo CD and we will talk about Linkerdee and Argo and the Gateway API and it'll be great. The fact of the matter is I don't really know enough about Argo yet to teach this but that's okay. That's why it's a month away and I'll be talking with a lot with Regina and it'll be a lot of fun. KubeCrash is coming up. This is a virtual conference that happens before KubeCon. It is virtual, it is free, it is 100% focused on open source stuff and we will be talking about multicluster at scale using, we're actually gonna do a stateful demo for this which is new and different. We will be using Linkerdee, Emissary, CockroachDB, CertManager and Polaris, the policy engine. So that should be a lot of fun. Perfect, sounds good. It's like if you want that. And you can also go and join the Linkerdee forum which is nice because messages don't disappear like they do on Slack. And we also have a certification course now at learn.buyant.io. Perfect, sounds really good. So many good places to learn from for sure and to soak up a lot of info before KubeCon in particular so that everyone could go there with. How can one define and identify whether a Kube cluster is a suitable match for Linkerdee or Istio service mesh? I am biased. I think they're all suitable candidates for Linkerdee. Less facetiously, I think the balance is that there are things that Istio can do that Linkerdee cannot do but in general, Linkerdee requires much, much less in terms of operational complexity and resource consumption than Istio does. So for example, we say this a lot. As far as I know, there are no companies running Linkerdee in production that have a Linkerdee expert who is paid to only be a Linkerdee expert, except for Guyant. And they're paid to be Linkerdee experts to work on Linkerdee not so much because they have to be Linkerdee experts to keep it running, right? But it's very common running Istio that you will have at least one person whose full-time job is the care and feeding of Istio and it is very, very uncommon to see that with Linkerdee. Like I said, I don't think I personally have ever. We also find regularly that Linkerdee consumes much less resource and causes much less latency than Istio does. So those are the reasons that I would look at those two. If there's something that you, you know, if there's some bit of functionality that Istio provides that you absolutely must have, then yeah, you should go and use Istio to get that functionality but you should accept that there's gonna be a lot more operational complexity involved. Sounds good. And then there was also a question from Ahmad. Thank you. Can you explain how a service without any selector works in this way to redirect traffic? That is actually a much more complex question than you might think it is. But quickly, okay. If you're used to thinking about the world from the perspective of an application developer dealing with North-South traffic, you know, dealing with the Ingress problem, you are probably used to thinking of services as just this monolithic thing where you direct traffic to the service and it shows up at the end points and life is grand. In the service mesh, we have to make a distinction between the front end of the service, which has a cluster IP and the back ends of the service, which are all the end points, all the pods that provide the compute that actually causes that service, that workload to do something. In the mesh, we attach HTTP routes to the front end of the service with the cluster IP and then the mesh does all the routing. It discovers the end points and it does all the routing to the back ends. Having a service without a selector is basically asking Kubernetes, hey, please allocate a cluster IP for me, but don't give it any end points. And so you end up with a service with nothing but the front end of the service. Then you can use an HTTP route to direct that to the back end of whatever services you want. I would actually really, really like it if there was a way to have an HTTP route and then just ask Kubernetes, hey, please give me a cluster IP for this HTTP route. That is not possible right now and this is the subject of a lot of discussion. I mean, a lot of discussion in various Kubernetes working groups. So hopefully that answered the question. It's that being able to answer that question, I think involved about three months of discussion in the gamma working group. It was lovely. Nice. Yeah. Linkardee gateway on one cluster only. How will it survive if the zone with the cluster and gateway goes down? It will not. Whether you are using gateways or not, if you're using clusters for isolation in this fashion, you need to understand that the cluster itself becomes a single point of failure. So if you're trying to use clusters for isolation and you're concerned about the entire AZ crashing because, I don't know, the US East data center gets hit by a meteor or something, I would encourage you to replicate that workload in another cluster in another zone and then perhaps use HTTP routes to split the traffic across the two. One of the things we're gonna be talking about in Kube Crash actually is exactly this scenario of, oh, look, this whole zone just went away. What happens now? And it'll be interesting, I think, is the answer to, but we should be able to show how to survive that. Just crossed. And then there's a question. How do you plan to handle the transition of existing services to a Linkardee? So I may be misinterpreting the question. I think you're asking if you have a cluster that is running without Linkardee and you want to add Linkardee to your existing stuff, the short answer is that you end up doing a rollout restart of those workloads. There are a couple of different ways you can get the mesh, couple of different ways to handle actually injecting it in there. The certification course covers this, actually. I'm pretty sure we have a service mesh academy that covers it. If not, I suppose we should do a service mesh academy that covers it. But yeah, the simplest way to do it is to annotate the namespace for your workloads with linkardee.io slash inject colon enabled and then do a rollout restart of the workloads in that namespace and they will all be injected into the mesh once that happens. The longer, more detailed answer is that sorry, Linkardee provides a Kubernetes admission controller. So as new pods appear, it can look at their annotations and figure out if it should inject the proxies or not. So switching from not Linkardee to Linkardee does involve a restart. Anything else? Let's see if anything comes up. I guess this is then kind of last call for questions if there isn't like the zillion questions here on one go, which sometimes happens. It's a positive problem to have. We have too many at the end of the show in a way. I mean, given that we actually have time, that's a good problem for this time. Yes, exactly. So we're kind of hoping for that. But there's been a really good questions already so far. So. Yeah, thank you. Yeah, I guess I could have spent more time on the slides but that wouldn't have been nearly as fun as spending more time on the demo and questions. Exactly. Roger, if you're implementing a service mesh, including software and support costs, the right answer to that question, the best answer to that question is go to the Linkardee Slack and ping Gary Mick, Gary surname MYK, and he will be happy to talk to you about how to spend money on Linkardee. The much, the more detailed, well, no. He will have the more detailed question. I can give you the more detailed answer. I can give you the sort of broader answer. There are at least two routes to go for Linkardee. One route, which is the one that I, as an open source person, think is awesome, is you can just use open source Linkardee and not pay anybody any money. You will, of course, have to do your own Linkardee support. You will be able to ask questions on GitHub. You'll ask questions on Slack and the forum, but nobody is gonna be out there committing to provide support for you. Me, as an employee of Boyant, I will go, oh yeah, you should totally go and get a support contract from Boyant and do all the Boyant things and give us tons of money because that would make me very happy and permit me to keep working on open source stuff. But yeah, you can totally do it in open source. I don't, I don't think I know of anybody who has more than like half an FTE devoted to babysitting Linkardee, maybe less than that, I'm not, I'm not actually sure. But yeah, you know, by all means, feel free to ping me on, feel free to ping me on Slack and I can point you in the right direction. I can also point out that, hey, there's Enterprise Linkardee. You can talk to William on Slack and that'll be great. But there is how to find me on Slack or by email if you wanna ask more questions about that too. Great, I'm gonna get a question from Neil. Do you want to reserve, should I read it? Fairly new to Kubernetes is Linkardee on a cluster level. Yeah, okay, so HTTP, I'm gonna do this in kind of a different order. Linkardee does some things at the cluster level like CRDs and such. In general, it is fairly tricky somewhere between tricky and don't do that for trying to run two service meshes in the same cluster at the same time. I think it might be possible if they don't both try to interact with the same namespaces. So like the faces namespace, trying to have some of the pods running with Linkardee and some of the pods with Istio, I know for certain that that is not gonna work out, but you might be able to have Linkardee in one namespace and Istio in another, just I don't know anybody who's doing that, I would not recommend it. So effectively, service meshes are kind of a cluster level thing. HTTP route for pod to pod communication definitely works. You saw it in this demo where we used an HTTP route to control traffic from a face pod to the smiley pod and the color pod. The bit where I was canarying between the blue color and the green color, some of that traffic was crossing a cluster boundary, but some of it was all within the north cluster. And there's no, you can do exactly the same thing with color and color too, all both in the north cluster and it works great. HTTP route originally was specified as a thing you could only attach to a gateway resource within Gateway API. The work that the gamma working group has done over the last year or so allows also associating HTTP routes with services. We do not believe that that's the end state, we think we're gonna have to do some more work still. In that there are things where if you try to do, if you only support services, there are some things that get really hairy that we'd like to be able to do within gamma. So we think there's some other stuff coming as well. But for right now, that's how you use HTTP route for configuring a service mesh. And if you want to know more about that in particular, let me find the link to some of that stuff and put it in there. So it's good. And then we have at least one more question from the audience and we have still more time if anyone wants to ask a bit more questions. This was exactly what we were hoping to happen though, people coming in with the questions, which is always nice to see. Thank you so much everyone. And let me get the right link to everyone in the audience. We can then I guess go to the next question. Oh, there's one or two more as well. Yeah, I pasted Annie a link to some of the docs there about the way that Gateway API works with service mesh. And so that's a good place to start if you're curious about that. At least I think it's a good place to start. I hope it's a good place to start because I wrote a lot of that. Lincorny already supports canary deployments and AB testing and depending on exactly what you mean by blue greed deployments, it can do that as well. Or a better way to put that is, so Lincorny, you actually saw a canary deployment where we split traffic across two different workloads in a weighted fashion. So you can already do things like, oh, take 1% of my traffic and ship it over to my new workload and find out if it breaks. You can also do things like take all the traffic for the, take all of the traffic with a header that says Xfaces user test user and direct it to Smiley too, instead of Smiley. And in fact, I think I've done that demo for cloud native live, haven't I? Pretty sure I've done one on Lincorny dynamic request routing, which is the thing that we call that within Lincorny. It should be available then in the CNCF. Yeah, I think it is. Live stream app, yeah. And in fact, Parag just asked about recordings. Yes, the CNCF publishes recordings of all of these, which is pretty awesome. So the only thing for Suresh Kumar's question about blue-green deployments is depending on how people are thinking about blue-green deployments, sometimes they are talking about something that's pretty similar to an AB test and sometimes they're talking about something where they roll out the green deployment and then the system will automatically divert all of the traffic to the green deployment if it works and otherwise it'll roll back to blue. And for that, and this also kind of gets to Suresh Kumar's next question about third-party integration, plugins, et cetera. If you want that kind of functionality for blue-green, I would point you to the service mesh academy that we've already done on using flux and flagger with Lincorny or the one that we're about to do on using Argo CD and hopefully Argo rollouts with Lincorny. So yeah, there's a lot of ways to do that. The third-party integration and plugin thing, I tend to think of it more as integrations rather than plugins because, for example, with flux, actually these days, I don't even know that you need anything special. I think you just hook the two of them up and they work because flux has a mode where it can use HTTP routes to handle shifting traffic around and Lincorny does HTTP routes natively now. But previously, yeah, you'd use the Lincorny service mesh interface plugin and then tell flux to use SMI things and that worked great. Good, and we have a couple of minutes left if anyone is typing away a question and wants to ask it super quickly, but just to clarify for Parag as well, all of the recordings for all of the cloud native lives are available in YouTube, for example, from CNCF there. So you can just put CNCF cloud native computing foundation on YouTube. They are either in the video tab or in the live stream tab. I think they used to be in the video site. Now they're on the live stream side. Stream side. Yeah, yeah, and they should be a playlist as well so you can find this one as well as all the previous ones with Flynn and Lincorny so you can find all of those from there. And I think also the recording of this is available on the LinkedIn side as well, but that might get buried a bit. So if you want to find out the older ones, maybe YouTube works a bit better for that. Yeah, YouTube is. Typically when I've wanted to go through and find stuff, then actually typically what I've done is just Google cloud native live Lincorny or whatever. And that usually gets me someplace close enough that I can find it. And it's pretty much always been YouTube, yeah. Yeah, that sounds like a good option as well. And for the record, there are many cloud native lives that are not about Lincorny. And I'm sure those are fascinating as well. That is very true. And I think the final question for today is we have one minute left. Is it possible to integrate Argo CD with any prerequisites? I'm not even sure you need any prerequisites. I'm my understanding. And again, I am currently not an Argo expert, although that is likely to change somewhat in the next month. But my understanding is that as of Lincorny 2.14, then it's pretty straightforward to have Argo use Gateway API to do its traffic shifting and things pretty much just work. But October 26th in Service Mesh Academy, we will be diving deep into Argo CD and Lincorny. And it'll be kind of fascinating to find out if there are any gotchas we have to talk about. There are always gotchas we have to talk about, but yeah. That is true. It always depends on something. Perfect. But that's it, I think, for today. Amazing to have the screen share a few minutes to go, actually. No worries. It's good that they got all the contact info if they need to ask any questions in the future. But amazing that we had so many questions in the end, particularly. That's a lot of fun. I fully agree, yeah. But thank you, everyone, for joining the latest episode of Cloud Native Live. It was great to have a session about Lincorny 2.14. We really loved the interaction as well as the questions from the audience. And we bring you the latest Cloud Native Code every Wednesday or Tuesday. And in the coming weeks, we have more great sessions coming up, so to stay tuned for those, thank you for joining us today and see you all in the future. Thank you. Thank you so much.