 So here we are, the service mesh past, present, and future. I don't even remember submitting this talk, like the talk submissions for KubeCon are so far in the past that everything has changed in my life. But I also have to kind of apologize, but I have jammed three presentations together, and they all have totally different styles, and there's all sorts of weird animations. So this is just going to be a disaster. Visually, visually a disaster. The audio track is going to be incredible. OK, so let me give you a 30-second kind of overview of who this person is, who's up here. So my name is William Morgan. It's not on the slides, of course. I used to work at a company called Twitter, where I learned a lot of things. Those things turned into a lot of what we're building at Boyant. And we have these two service meshes. We have two now. One is called Linkerdee and one is called Conduit. I'm going to talk about some of the history and the rationale and my personal service mesh journey, and I guess how I got here. And in particular, a lot of it came out of this experience that I had and that we all had at Twitter, where we moved from circa 2010 when I started there from this monolithic Ruby on Rails application into fast forward five or six years into this big massive microservice kind of cloud native, I guess you could call it cloud native environment. We didn't have Docker back then. We didn't have Kubernetes. We had Mesos, and Mesos was a grad student project, so we had to make that thing production ready. Instead of Docker, we used like C groups and we were on the JVM, so we had kind of resource isolation and the packaging mechanism. And then we had this other kind of part of the stack called Finagle, which was this library, and the library was used for managing all service to service communication. So the way that you would build a service at Twitter, if you wanted to add a new microservice to the Twitter stack, you would build it using Finagle. And Finagle was this cool functional programming thing. We were all Scala programmers, so we were going to do functional programming on top of RPC calls. We used Thrift for whatever reasons, and Scala. And so we needed a cool way of doing functional programming on top of these Thrift calls we were making. And that turned into Finagle. And Finagle has kind of this long history at Twitter, where it became this, over time, it became this kind of very rich platform with all these features in it. And you as a service owner, you were treating Finagle as like this nice library that you could say, hey, I'm service A and I want to talk to service B. And Finagle, go make that call and give me back the result. And under the hood, Finagle was doing all these things. It was like load balancing and routing requests. And it was like, even the load balancing was based on application, instance, latencies, and trying to send traffic to the fastest thing. And it would do circuit breaking. And it had a consistent layer of telemetry and all sorts of cool stuff. So that's kind of the history of why I'm even here, why I'm talking about any of that stuff. So I'm going to take like a philosophical second for a couple of slides. And let's just talk about this idea of resilience. Because this is kind of what we're trying to build in these software systems. We're moving onto the cloud. The cloud's scary. Cloud's weird. We don't own a hardware. It's all different now. And so we still have to build these reliable systems. It's not OK to have scheduled maintenance anymore. You used to be able to do that. But you can't do that anymore. And you have to take into account the fact that when you're running in a cloud environment, 50% of your shit is broken at any point in time. So you want to build software that can still work under these conditions. So there's a material science definition of resilience. Basically, it's got to handle stuff without getting totally busted. It's a short story. Some people like to talk about anti-fragility. But I think that's just a nicer kind of marketing term. And so the basic idea in software is, OK, we've got these forms of stress. We're building this system. And we've got unpredictable load from the outside world. People are sending requests to us, presumably. We're getting traffic from the outside world. We've got flaky hardware. We've got buggy software. I think you could add a couple more things in the cloud environment. Like I'm running on this AWS instance, and suddenly some other AWS tenant gets all of the resources. Who knows why? And I get none. And that's just unpredictable. And then in response to all these sources of stress from inside the application, from the outside world, we want to be resilient. And how do we do that in software? Well, we can show it. We want to be able to shed load in a way that doesn't penalize the entire system. We want to be able to handle failures gracefully. And we want to be able to provision and scale and all that good stuff. And if you look at the history of software engineering, back in the early days of software, internet software, I guess, the way that you would build resilient systems or reliable systems is you'd buy the big iron. You'd buy two F5 things. And you'd over provision the heck out of everything. And this was all running in your data center, like in the closet in the back of the office or whatever. And this kind of worked. It was kind of, okay, it had a series of problems. But it was a way of building reliable software. And so now we advance 17 years to a world where it's not really feasible to do that for reasons one through 100, we're now moving into the cloud. And so how do we accomplish a similar thing? Well, we don't have any of the guarantees that we had from hardware. We don't have the guarantees where this is my machine and I'm gonna get 100% of its usage. And if it dies, then Sally from the hardware department will come here and replace it with another one. We don't have any of those guarantees. So those guarantee the guarantees around reliability that we lost to a hardware. We now have to replicate in software. And that's kind of the idea of this cloud native environment, okay? So what's cloud native? Well, we have a foundation now and we have a conference and everything is great. And kind of at the surface level, in order to be cloud native, you should be using containers and you should be running an orchestrator and should be building microservices. That's a pretty prescriptive definition. I think if you look behind the implementation details, what cloud native means and why I kind of like the term is the software that's designed to run in these cloud environments. In environments where you just don't have a lot of reliability guarantees from any of the underlying systems. Does it seem okay so far? Okay, so what's changed? So we went and we zoomed from 2000 to 2017. Back in 2000, we were thinking about virtual machines. Oh, maybe, I don't know. Yeah, I guess so. Data centers and like this hardware redundancy, we were thinking about individual servers, the IP addresses and we were monitoring these servers and whatever else. And now in this cloud native world, we've kind of abstracted a lot of that stuff away. We're talking about services now and services discovery, service monitoring, microservices. And we're not worrying about individual servers or individual IP addresses or any of that stuff. We've abstracted a bunch of that stuff away. We're not worrying about individual TCP connections. Like we kind of assume that that's gonna work 90, whatever percent of the time. Instead, we're thinking about, well, how are our REST APIs formed and how are we gonna use GRPC and are we doing flow control over the HTTP2, streaming semantics or whatever. So the abstractions have all changed. And the thing that we've introduced, the thing that we've introduced by doing this shift, especially with kind of the microservices aspect is now we've introduced this notion of runtime communication. That is not just this kind of one-off, the web server talks to the app server and the app server talks to the database. This is actually a pervasive part of our application now. We never had this before. And when you think about it, when you think about it in your head, well, it's kind of, it looks kind of like this. Service A talks to B and B talks to C and then the response comes back and, great. What could go wrong? And in reality, what happens is you end up with, you don't end up with like A and B and C. You end up with a big hairy mess like this. And this is the architecture of Twitter, circa 2013. I'm gonna continue using this diagram until the day I die because it's just so good. Like, it's just like who wants to be on call for this, right? But we were. I was. I had to wake up, you know? And you can see there's this poor little service down there called Gizmo Duck that was basically the user service that everyone talked to. And like, it's just this complicated topology of things. And so this is, the previous slide was like, how you imagine microservices are gonna be, and this is like how microservices really are. And so I'll give you one quick example about why, just to make it super concrete about why the introduction of this very pervasive service to service communication becomes complicated. And this is an example that I like because it's clear there's like 20 other things that fall in the same category. So here I've got this system, right? The very simple Twittery system. I've got a web service that talks to this timeline service which talks to the user service which talks to the database. And okay, each of these things I've got like, I gotta account for the fact that the, it might be down or it might be frozen or something. So I set a timeout and I set some number of retries. And you know, here we go, ready for action, right? Does anyone spot a problem with this? Yeah, overall latency might be too high, that's right. Yep, yep, any other, any other formulations? Yeah, the downstream service retries are useless. That's right, that's right. So, you know, if you look at what happens here, if the database starts failing, right? Let's say, not failing, let's say it's hanging. Cause failing is easy, right? Hanging is like the thing that's really painful. Database starts hanging because it's overloaded. While we start hitting this retry, you know, the timeout on the user service and user service starts retrying stuff, okay? But now we're hitting a timeout in the timeline service and the timeline starts retrying stuff. And then we're hitting a timeout in the web service and you know, pretty soon one tiny little, you know, slowdown in the database and it's just a little slow. Suddenly it gets compounded cause we're sending all these requests, right? And so this is a general class of like, it's really hard to manage this communication and the way that we parameterize it makes a big difference, right? So, we're used to kind of thinking about our telephone talking to the Twitter server, right? And what's it do? It makes a thing, it makes a request, or a web browser, talking to a web service. And you make a request and you wait 500 milliseconds and you make another one and if you're really fancy, you do exponential back off. But that pattern really starts breaking down when you apply the scale because this parameterization doesn't compose, right? And this gets even worse when you got a lot more services. Every service is owned by a different team. The teams are not talking to each other cause they all hate each other cause, you know, one's written in Java and one's written in Go or whatever. Yeah, okay. So, lots more examples of that but I think that's a fun one. Okay, so what's missing? Well, you know, now I have to do the selling bit. You know, we've got Docker, we've got Kubernetes, like surely at this point we're ready, we're ready for anything, right? Like we can scale, we've got containers and what's missing really is what was missing from the Twitter story up until the point where we got Fnagle. What's missing is this idea of the service mesh, some way of managing communication between these services consistently across the entire application. You know, and ideally this happens in a way where you don't have to, you know, get the developers to redeploy their services, right? That was kind of the problem on Twitter is you made a change of Fnagle, Fnagle had a bug and then you fixed it. Great and then it would take like six months to get everyone onto the latest version of Fnagle. Meanwhile, you had found another bug, right? And so that makes life difficult. Okay, so I think if you think about security being kind of this thing that you have to implement in layers, I think reliability is one of those things that you have to do in layers as well where Docker gives you a certain amount of reliability. Kubernetes gives you a certain amount of reliability, right, like the machine can go down and everything gets automatically restarted and rescheduled but there's still another layer on top of that which is the service-to-service communication. Okay, so the service mesh, finally, maybe I should have started with this slide. What is a service mesh? It's this dedicated software layer for managing the service communication, all right? So the idea is we wanna pull it out of the applications, right, and put it into the underlying infrastructure. Yeah, I apologize for these animations. So, you know, quick, this is a very overly simplified history of software architecture but we started with a lamp stack back when we were all children, like all that we had was a lamp stack. And you had service, you had communication, internal communication here, right? You had communication between Apache and PHP and between PHP and MySQL, right? But that communication was very specific, like these use cases were very, wasn't this general service-to-service thing, right? And Apache was really, really good at balancing load over a bunch of PHP instances. And the PHP MySQL clients got very, very good at talking to MySQL, you know? So you had these specialized clients, right? And you fast forward a little bit from that into the world of, this is kind of where Twitter was, you know, into this world of fat clients. So the early web-scale companies like Twitter and Facebook and Netflix and Google, well, you know, once they took some monolith and they broke it down into microservices, right? And now they had this consistent layer or pervasive layer of communication. So they had to fix it and they used libraries, right? So Histrix and Netflix and Google is stubby and kind of the associated libraries and finagle at Twitter, right? And this was a better approach, right? And they all kind of had to do this because there was no way of avoiding it. And the service mesh is really just taking that same logic and moving it into a separate layer, right? That's all it is. It's not new functionality. It's not stuff that we haven't thought of before. It's like, oh, we've never load balanced before. Well, we've been load balancing since load existed, you know? But it's moving it out of the developer and into the hands of the service operator into the underlying platform. So I have a little diagram of like what this actually looks like in practice. This is for Linkerd for conduit. I don't have a diagram, but then it looks a little different because it's side cards rather than per host. But, you know, so what does a service mesh look like in practice? Well, in practice, it's typically a bunch of proxies, right? And it's a bunch of proxies that you stick in between all the services one way or another. And then there's like a control plane that gets managed here, which, you know, that little green line leading off to the side is, you know, going off to the control plane. And when service A wants to talk to service B, one of those instances talks through its local service mesh proxy, which talks to the destination service mesh proxy, which talks to the destination instance. So you've got two hops in there. And the reason to put two hops in there is because then you can control both sides. So you can add TLS and then terminate it or you can like upgrade from HTTP one to HTTP two or whatever. Basically, you want to give the mesh, you know, you want to give it ownership over that communication layer. And the reason to do it as a separate proxy, well, it's because then you don't have the library problem, right? Then you don't have to, you don't have to A, like you can be polyglot, right? You don't have to like maintain the same library across N different languages. And B, especially for bigger companies, you don't have to convince the developers to redeploy their services every time you want to make a change. And this is all made possible. It's only made possible because we have things like Docker and Kubernetes and like make doing this sort of thing easy. You know, we could have done this at Twitter, we could have done this, you know, 15 years ago, but it was really, really hard. Right now we have things that make it easy. So it's enabled kind of this sort of architecture. Let's see. Yeah, service mesh does a whole bunch of stuff. Reliability, visibility, I'm not gonna, well, maybe I'll spend 30 seconds on this. So largely what I've been talking about has been the reliability semantics like retries and timeouts and how you might want to parameterize them differently and like, you know, the service mesh can do some of that for you, right? We, if it knows a request as item potent, it can retry it. If it knows, you know, that, you know, this application instance is returning, you know, application level errors, where we can just circuit break and kick it out of the load balancing pool, right? We can do a bunch of that reliability stuff. It's also the visibility and maybe visibility should have been first in the slide because that's really the thing that I think gets people hooked on linkerd on conduit on any service mesh first is the fact that you suddenly have this visibility into, you know, into these critical, like the top line service metrics. These are the things you've never had before, right? It's like success rate and request volume and service latency. Oh, you've probably, hopefully you've had them before, but now you've got a consistent, you know, consistent way of getting them across your entire application stack. That's independent of what language you're written in and independent of what libraries you're using independent of the frameworks. You suddenly have all this visibility and these are the top line service metrics. These are the things you want to wake up for at 3 a.m. There's a bunch of cool security stuff which I'm going to skip over governance. I don't even know what that means. That's, it's really expensive, whatever it is. Linkerd has some production users and has some things. Okay, I'm just kind of going to skip through this. Okay, the future. All right, so I guess what I've already done is a past in the present. All right, so I guess we're moving into the future. Okay, so what's the future of the service mesh? This is fairly linkerd-specific, so you'll have to forgive me. You know, so I think security is a pretty interesting, is a pretty interesting topic. This is certainly one that comes up a whole lot for us. As people are moving into these cloud-native environments you want to do things like isolation of services. You want to do things like having policy on top of which services can talk to which other services. Which calls are they allowed to make? How do we know which, who, how do we know whether a service is who it claims it is and how do we apply policy on top of that? And are we doing this like, what layer are we doing this? Is this per service, is it per request or do we have this notion of customers or tenants because we're like running on this multi-tenant application. And then there's the question of like, well what about the mesh itself? How secure is that thing? This thing is all your PII and all your HIPAA data is going through these proxies. Is that okay? Performance, this is a big one because what we're asking people to do is to install these proxies everywhere. And to just jam them in to every single application call. Not just once, but twice. So that means we've got to make this pretty fast. We've got to make it really lightweight. And we've got to make its behavior predictable. That almost might be more important than any of the other things. If you're introducing a constant time cost, well that's understandable. It's when you introduce something that has a high variance in the latency that terrible things start happening. So we care about the P99 or the P39 or whatever, way more than we care about the P50 when we're looking at the latency distribution. Okay, and then finally, and I gotta say, I'm trying to make this more about the service mesh in general, not too much about LinkerD, but LinkerD has been in production at this point for about 18 months that companies around the world and we've learned a lot of stuff about this. And a lot of the things that we learned are really around this kind of operability aspect. So when the service mesh goes wrong or when something is going wrong, how do you know? How do you know where the problem is? Is this a problem with my service? Is it a problem with the underlying network? Is it a problem with the service mesh? It's just hard, it's really hard to know when there's so many moving pieces. So how understandable can you make the service mesh? You have a sense, when you're relying on this thing, you have a sense that you can rely on it, that it's understandable and it's predictable. And then the other thing that's really interesting for us is like, I said here, separation concerns between dev and ops, I think maybe a better way would be saying that would be separation of concerns between the service owner and then kind of the platform owner, right? Because those are two engineering groups that have fairly different goals and fairly different requirements in kind of a larger organization. And I think a lot of the value of the service mesh is in allowing that separation and kind of increasing that separation of concerns between those two groups. Okay, great. So that is, that's the future. Oh, wait, no, the answer. Yeah, so these are all questions. Here's the answer, all right? Conn to it, Conn to it's the answer. Okay, end of talk. Done. Yeah, so I wanted to spend just a minute on Conn to it because we launched this thing yesterday and I guess a lot of the questions in the previous section were kind of segues into Conn to it because a lot of what we tried to do with Conn to it is address exactly those concerns based on the 18 months of production experience with Lincardie, you know, at a variety of companies around the globe, some who are very, very happy with Lincardie, some who have had problems with Lincardie and some who have not been able to adopt it for a variety of reasons, right? And so the goal with Conn to it was not only make something that's really specific to Kubernetes, right? And like just let's just absorb those idioms and make sure it works in conjunction with Kube-Cuddle and make sure it uses the Kubernetes like ontology and we're talking about deployments and pods and all those things, which Lincardie for all of its benefits was a very abstract generic layer that could talk to 20 different things and which made it very difficult to talk to one specific or to configure it in terms of one specific thing. So let's fix it to Kubernetes, right? And then let's just make it incredibly fast and incredibly lightweight. Lincardie, what was interesting about Lincardie is that all the building blocks that we built on top of things like Finagle, which in turn was built on top of Netty and Scala, which in turn was built on top of the JDM, like all those components are really good at scaling up. You give them enough memory and you give them enough CPU cores and like Lincardie can process tons of traffic. It's just crazy. Like just throw tens of thousands of requests through a single instance, but in the Kubernetes world, especially if you want to deploy as a sidecar, you kind of want the opposite, right? You want something that doesn't have to handle damage traffic. It handles a thousand requests a second or something because you're deploying it one per application instance. But it has to be really, really tiny. It has to be really lightweight and you want to have a really predictable P99 latency. And that was where Lincardie had trouble. So the production users of Lincardie, well, it kind of depends on the size of your services. We have many people who use it in production as a sidecar because their services are big. They're running these heavyweight JDM services that take a gig of heap. And so you spend 200 megs on a Lincardie container and whatever, it's not that bad. But then we have folks who are writing go microservices and then take the 50 megs and then it's a little silly to ask them to put Lincardie as a sidecar. So we do it as a demon said and that gets you a bunch of, that at least lets you amortize those resources. But it's not ideal, right? We wanted to do something that was right from the start. And the other big thing that we did was try and design this with security in mind, especially for our users who have PII or PCI compliance issues or HIPAA data or whatever. We really want to ensure that this data plane that's in there that's touching everything is as reliable and as secure as it possibly can be. So we built the data plane in Rust, which six months ago seemed totally crazy and risky. Right now, it seems awesome because it worked. And what Rust gives us is not only the ability to write native code, so it's about as fast as programming as code can be until you drop down to assembly. But it's also guaranteed to be memory safe. So we avoid a whole class of buffer overflow exploits and things like that. And it does not have a garbage collector. And that's important because that drastically reduces the variability of the latency. It means things are really, really predictable. And I'm not a Rust expert at all, but I know a little bit about what we've done. And it's super cool because the Rust language features allow you to very easily express these things where I want to tie all of the... I'm allocating memory as part of processing this request. And then when the request terminates, I want to just free all that memory at once. And when you do things like that, you can amortize the cost of memory allocation deallocation over every request and then things become very, very predictable. And then all sorts of cool, powerful features. I'm actually not going to have time for demo because I want to do a little bit of questions afterwards, but we'll do one at the SIG or the salon tomorrow. But powerful features, one of the things that Oliver showed off during the demo was this idea of TAP, where if I have services running in production, I should be able to just kind of TCP dump into those requests. Like just show me, just show me what's going on. And so we did a lot of work to make things like that powerful possible. Okay, I'm going to skip the demo as I said. And that is the end of the service mesh. That's the past, the present, and the future. And I'd like to do some questions and I'll ask that you do it with a mic so we can get on the recording. Ask me anything. Anyone have a question? Is that? Oh, I'm sorry. I see no questions. Yeah. Okay, I'll repeat it. There's two going forward and where does this evolution lead those of us who are running late for being caught kind of forgot to take that board? Yeah, yeah, great questions, great. So number one is kind of going to support Thrift in the future. I think the answer is possibly. The answer is possibly. That's all I needed to say. Yeah, we're not ruling it out. I think Thrift is not like the future of protocols but it's also really simple protocol so it might be easy to just do it. And then the next question was, okay, what about Lincardy? Are we throwing that thing away and like, sorry, Lincardy uses. And the answer is no. Lincardy is going to have a whole lot of use cases. We'll continue to have a whole lot of use cases that Conduit won't be able to address. And so Lincardy's here forever. Yeah, if you were deploying a service mesh in Kubernetes a month from now, let's say two months from now, we got to get production ready. Then yeah, I would start with Conduit. Yeah, unless you had some really specific requirements around I have existing Meso's infrastructure, I have existing Zookeeper, or I have Nomad over here and I need to make this stuff all work together because that's the kind of stuff that Lincardy is really, really good at that Conduit is just not going to address. Yeah, you got it. We'll Conduit and Lincardy speak to each other. I would like to make that happen. I'm not 100%, I don't have a great answer for you. I'd like to. Yeah, yes, that's sacrilege. The question was what happens when Kubernetes is replaced by whatever comes after Kubernetes, that's going to be the next, the new hotness. Well, that would of course never happen because Kubernetes is perfect in every way, but I think one of the funny things about the whole service mesh model is that it's not actually that tightly tied to the orchestrator, right? At the tech level, we integrate both Lincardy and Conduit integrate with Kubernetes as kind of a service discovery endpoint. We call a couple APIs, but that integration is not really that tight. The usage, the adoption is like super tight, but the tech integration is not that tight. I actually think that'd be fine. I don't think it would be that much work to make Lincardy and Conduit work with whatever the Kubernetes after next is. Yeah, yeah, that's right, that's right. Yeah, great question. Yeah, yeah, great question. So is Conduit a data plane or is it a control plane? Conduit has both. Actually, Lincardy has both too. We just did a bad job of talking about it. So Lincardy, we've almost always talked about this thing. So let me take one step back and kind of describe this distinction. Matt Klein actually has a really good, the author of Envoy has a really good blog post about data plane versus control plane. The idea is that in a service mesh, there actually are these two kind of logical components, at least. One is what's actually proxying their requests? What's handling those bits? What is seeing user data? What's exposed to PII and all that? And that's the data plane. And then the other component is, well, what are you using to kind of orchestrate? Oh, maybe that's a bad choice of word. What are you using to control all of those proxies? And that's the control plane. And with Lincardy, we almost always talk about it in data plane terms, but there is a control plane called Namerdy. With Conduit, one of the big lessons that we learned is we talked very explicitly about the Conduit data plane and the Conduit control plane. And those are two things, they're two separate code bases. The control plane's actually written in Go, which makes it easy to use all these Kubernetes libraries, which is nice for us. The data plane's written in Rust. And having that distinction has made us, it's been helpful for our own internal thinking as well. Because the two parts of that system have very different requirements. I think this may be on now. Excellent. All right. Oh yes, up front. I was just gonna ask how this compares to Envoy. Yeah, so how does this compare to Envoy? It's a very different beast because Envoy is, I think, much more of a data plane than Conduit. So Conduit is both control plane and a data plane. Envoy is pretty much a data plane, right? And the way Envoy's used it, Lyft is with a non-open source control plane. I'm gonna put this over here. With a non-open source control plane, the way that it's used with Istio is Istio is a control plane and Envoy is a data plane. With Conduit, we have both pieces. So it's not really an apples to apples comparison. I think a better comparison would be with Istio and Envoy, right? Because that is kind of the same thing, right? They're both service meshes. They have a control plane written in Go. I'm sorry, Istio and Conduit. Yeah, yeah, that's a more of an apples to apples comparison, that's right. Thank you. So here's the integration by setting a proxy on the data set for the proxy. What is the reason for having a sidebar proxy just for security? Yeah, great question. The question was, what are the security implications basically of using demon sets versus sidecars, right? And the big implication there and the reason why the sidecar model is so compelling is if you wanna do service, any kind of service auth or service kind of identity, right? Then doing that as a demon set doesn't make a lot of sense because you wanna have a cert, presumably you're doing this with TLS. There's other ways of doing it, but if you're doing this with TLS, then you'd have a cert per service, right? And then in demon set, you're just giving all the certs to this thing and you've lost all your security boundaries. So that's why the sidecar model is so interesting and so compelling is because if you wanna do any kind of service identity and build policy on top of that, you gotta be sidecars. Yeah, really good question. All right, let's do it in the back. Target for production stable early next year. It's gonna be in the feature set, it's gonna be super minimalist. That's the way we're slicing and dicing this is get something really small ready for production first and then start adding features on top of that. So the question was in the library case, if you make an update to the library, you then have to roll all your services. In the sidecar case, don't you have the exact same thing or if you have to update your sidecar, you then have to redeploy your services. And the answer is it's not quite the same because you don't have to get developers involved. You don't have to get the service owners involved. If you update the sidecar and I think Oliver basically did this during the live demo this morning, you update like the deployment spec to include the new thing and then you roll the deployment and that's a pure platform owner kind of operation. There's no recompilation. There's no like going through CI, CD, or unit testing or anything like that. Oh gosh, when I say it like that, it sounds really scary. There's definitely integration testing that you do before that. I'm sorry to say that again. You have to recompile. You don't have to recompile when you use it. A new version of a library. Okay, okay, yeah. Does work in a federated cluster? Yeah, definitely. I don't see why it shouldn't. I don't think the federation really affects any of this until you start doing really complicated cross cluster traffic things. Let's do just one more and then we'll let everyone go. Have fun, yeah, go ahead. Is LinkerD connecting with Istio? So we have a very preliminary integration that we did back when Istio was first announced to have LinkerD act as a data plane for Istio. So you could run the Istio as a control plane and LinkerD as a data plane. We didn't get a lot of interest in that. We did it and it worked. It only supported maybe 60% of the Istio API at the time, but we didn't really see any adoption. No one was really interested in using it, so it's possible to do, but no one wants it. All right, I'll stick around for a little bit. So please feel free to come up and ask me questions. Thank you very much. Thank you.