 All right, I think we're ready to get started. Hello and welcome, everyone. Thank you for coming. I know it's been kind of a long week, so I really appreciate you making it to my presentation. It's one of the last at KubeCon, so we're gonna make sure we wrap it up with some nice service meshing. Before we actually start, can I get a quick show of hints for people that I've heard about Linkerty before? Oh wow, all right, we're doing something well. And who hasn't ever used the service mesh before? Okay, no worries, we're gonna cover everything. Cool, so let's get started. Welcome to Overview and State of Linkerty. We run one of these presentations for every KubeCon. I'm gonna be going through what a service mesh is, what Linkerty is, and where it sort of fits into the CNCF landscape. And then we're gonna talk about what we've been doing and where we're going next, and what's planned for the roadmap. So, a little bit about myself. My name is Matthew David. I'm a Linkerty maintainer. I'm also a software engineer at Boynt, the creators of Linkerty. If you wanna get in touch with me after the presentation, you have all of my handles there. I'm on Twitter, GitHub, and a couple of Slack workspaces, not just Linkerty's. That's a picture of me before I started working for Linkerty. I didn't know what IP tables is, so I was very happy. Things have changed a little bit since then. But I basically started my contributor experience with Linkerty when I was still a student. It's been like two years and a half, almost three now, that I've been coding and pushing features for Linkerty, so I'm super excited to have a chance to talk about it with you all. Maybe you know, maybe you don't. There are a couple of people that don't know anything about service meshes, but Linkerty is a CNCF's first graduated service mesh, so that's a huge milestone for us. We've been running in productions for more than five years now. The slides are a bit old, so it's four plus years, still accurate. We have a bunch of Slack channel members. We have a very thriving community, both on GitHub and on Slack. We're committed to open governance, and we have been adopted in production by a bunch of cool companies ranging from enterprises all the way down to startups. So no matter how big or small your company is, how much traffic you have, Linkerty is definitely a good fit, and we're gonna see why. Let's talk a bit about service meshes and what Linkerty is supposed to be doing. So service meshes, I think, in the CNCF landscape have become a bit of a buzzword and have become a bit complicated. So they have this sort of history that predates them, I guess, and that makes them seem very confusing, so people will try and stay away from them. But service meshes are pretty simple. They're just platform level tools that give you reliability, observability, and security out of the box. So it's nothing complicated about that. We basically do this by injecting a bunch of proxies running as sidecar containers, next three applications. So what Linkerty does is it injects a proxy, next three application, it receives your traffic, it proxies it over to its target, and in the process, it achieves MTLS, so security. It pulls metrics from that, so you have observability, and it provides a bunch of useful features, such as retries and timeouts, to achieve reliability. It's not very complicated, so that's precisely why, in the Linkerty team, we like to preach about simplicity. Service meshes can be simple, if done right, and that's exactly what we're trying to do. Linkerty just works, and that's why we have, as you can see on the slide, the motto, less is more, and more is less. We have this strict philosophy, and we're kind of very opinionated when it comes to it, that when you have a platform level tool, it should just work. You shouldn't be concerned with learning how a tool works, and you shouldn't have to operate it too much on a daily basis. What we want you to do is to install Linkerty, and just run it. Take your hands off, focus on your day-to-day job. You're probably either a developer, or you're an operator, you have scripts to write, you have code to push, so you don't need to worry about how the proxy works, and all of the other platform level capabilities. So with Linkerty, we wanted to make sure that, with zero config, everything is possible, out of the box, for any Kubernetes app. We're also ultra light, we wanted to introduce the bare minimum overhead, and that's very important when you have a platform level tool, especially one that proxies traffic. We wanted to be simple, like I previously said, and we wanted to make security a basic feature. We didn't want it to be an option, we wanted to make it the default. For the data plane, Linkerty uses Rust. I'm not sure many of you are familiar with Rust. Can I get another quick show of hands? Yeah, all right, we're doing well there. So Rust is a programming language that's been sort of getting more momentum as a flate, but its main value proposition is that you get zero cost abstractions, and that is very performant. You can basically compile it to native code, so it's very portable, you can run it on any architecture that you want, and it makes sure that it has a strong type system so that most of the years are caught at compile time, and that's very important for us, because again, as a platform level tool, we don't want to bring down your prod deployment on a Friday, right? Your production environment's right on a Friday, because that would be pretty bad, it will affect poorly on us and on you for choosing Linkerty. Rust, like I said, compiles to native code, and that makes it ultra light and ultra fast, and it lets you avoid an entire class of memory vulnerabilities through its borrow checker. So Rust basically makes sure that you cannot commit any bad memory faults, such as double freeze or messing around with pointers, and if you've coded in C or C++, that probably happened to you at some point. It doesn't even have to be something that you don't know, just a simple negligence and a pull request can just lead to a bunch of disastrous cascading failures. We're also built on a state-of-the-art network stack, so there are a couple of libraries in the Rust ecosystem that have gained a lot of traction and that are used in production systems. So one of them is Tokyo, which is in a synchronous runtime, and the Linkerty maintainers and Linkerty contributors have contributed extensively to Tokyo. One of the core maintainers was actually Adboy and developing the Linkerty proxy. We're built on top of tower, we're built on top of Hyper and H2. So most of the modern Rust async networking stack that's now used in a bunch of libraries and a bunch of other production systems have started from Linkerty and we're still contributors to this day. Our philosophy when it comes to the proxy is that it should be simple to implement, sorry, that it should be simple to use and it should be in implementation detail. It should be sort of opaque to everything that you're doing in your Kubernetes environment. You shouldn't have to be a proxy expert to operate Linkerty. Our approach to security, so that's another one of our core ideologies. First, keep it simple. Second, keep it secure. Our approach to the security is very focused and follows this strict philosophy. We want to secure the foundations, so we're using Rust for the Linkerty to proxy so we don't get any CVSEs or memory vulnerabilities. We're built on top of Kubernetes actually and Linkerty as a whole we're leveraging almost entirely Kubernetes primitives. And in this case for security we use service accounts for pod identity. And we want to remove any barriers that you would have as an operator. So that's why we have MTLS is on by default. And as the recent versions you can't actually disable MTLS. We want it to be on because we're strict believers in security. This is a question that we often get and we kind of love to answer it. How does Linkerty compare to Istio? And it's a bit of a hard question because it really depends on your production environment and it depends on what you want to do. Istio is built for a feature parity. They add a bunch of features. You can do a lot of cool stuff but that comes at a cost. It comes at a cost of complexity. You have a bunch of CRDs to manage. You have a bunch of things to learn. And this is very hard when you have limited time to learn a tool and to adopt a tool. On the other hand, Linkerty uses the bare minimum to build a secure and reliable Kubernetes platform. So like I mentioned we leverage almost entirely Kubernetes primitives. So for example, for all of our network communications we're using endpoints, we're using services, we don't add anything new to the whole stack. So that makes us much easier to use but it also makes us smaller, lighter and faster. That's because Linkerty is built purposely, the Linkerty proxy is built purposely for Linkerty. And when we think about features, yes, we do miss out on feature parity but every feature that we add is carefully thought out. That's something that we want to do for our adopters. We want to provide a stable platform. We don't want to introduce any breaking changes. We don't want to surprise you. We want you to know that the tool is going to work for a very long time and it's going to have backwards compatibility. And we do that by carefully thinking through the roadmap and thinking through of what people need. Chances are if you have a problem, we can definitely fix it. But if we can fix it without adding more overhead, more complexity and more features we're probably going to do that as well. The cons is obviously for Linkerty that it only works on Kubernetes. So we were built for Kubernetes and we leverage the primitives. But this has an asterisk to it. And as we'll see on the roadmap, it's going to be changing in the future. That kind of covers the first part of the presentation. The introduction to Linkerty and service meshes. If you didn't really understand everything, you'll have some time to ask questions at the end. But now I want to talk a little bit about Linkerty's year so far. So since the last time I had this presentation for a KubeCon EU 2021, a bunch of things have changed. Shortly before last year, before KubeCon last year, we introduced Linkerty 2.9, MTLS for all. So prior to this, Linkerty only did MTLS for layer seven protocols. We did MTLS for HTTP, GRPC, and that was kind of it. But we wanted to introduce MTLS for TCP out of the box with no configuration and we managed to do that. We've also added ARM support. We're the only service mesh. As far as I know, please feel free to correct me. That compiles to ARM architecture. So if you have a home lab, if you have ARM instances, you can run Linkerty on it. We introduced a multi core proxy runtime so we can add more throughput and concurrency. This was sort of by community demand. People wanted to have some configuration options there and mess around with the proxy's FRED system. So we obliged. And then we also made Prometheus optional. So in Linkerty 2.9, Prometheus and a metric stack shipped with the core Linkerty installation. But we quickly realized that in a bunch of production environments, people already were running their own Prometheus instance. And we didn't want you to have to deal with whatever Linkerty was adding. So we just made it optional. But that gave us an idea for Linkerty 2.10. What if we just removed Prometheus and we make everything optional to begin with? What if instead of adding all of this stuff that you might not need, you can just have a default control plane. That's the bare minimum. It doesn't introduce an overhead. And then you can start composing as much as you want. So this idea of having composed sort of components and adding composition also comes in the proxy where we were inspired by the Finago library. I don't know if any of you had an experience with it and sort of follows a similar pattern of building middlewares to proxy traffic. So this is what we did for Linkerty 2.10. We split everything out into extensions. So we have a Viz extension, which is the metric stack. You have Prometheus, Grafana, Metrix API and the Linkerty dashboard. If you want to run with it, you can. If you don't, you don't have to. We introduced a Yeager extension for distributed tracing and a multicluster extension by pulling all of the multicluster-specific machinery into its own thing. Extensions can also come from third parties. So we wanted to give adopters the chance to run their own extensions and integrate with the CLI and the check system that we have. So right now, for example, in my spare time, I'm writing a Linkerty extension to take care of job objects. If you've used the sidecar pattern with job objects, you'll know that it's a bit hard to deal with. So I wanted to solve that problem for the community in my spare time and I can just do it in an extension. Besides all of this extension stuff, of course, we added more stuff such as opaque ports to proxy without protocol detection and we extended multicluster to all TCP connection and lots more, of course, that I didn't really remember to put on the slide. Today, we have Linkerty 2.11. So we released Linkerty 2.11, I think, in autumn and what we focused on was policy. So for Linkerty to truly be a security-conscious service mesh and to offer as much security as we can, we wanted to make sure that we can extend zero trust to your Kubernetes platform, but we cannot have zero trust without having authorization policies. So that's why in 2.11, we focused on server-side policies. We wanted to have a fine-grain access control system that you can use in a composable way, again, or big fans of composition, that works with Linkerty's secure identities and also that works at a network range. So right now, with authorization policies, you can describe basically a resource on your server that will allow you to accept traffic from specific clients, and this is all based on identity or based on network ranges. If you don't want certain subnets to connect to your server, you can just deny them. We also added some more stuff. We added GRPC retries, so this is more HTTP requests with bodies. If you have an HTTP post request that has a body and you want to retry it, I don't know if any of you have tried to implement this, but it turns out to be a pretty complex problem, especially if you want to keep the overhead minimal. What do you do with a lot of the bytes? Do you keep them in memory? Do you buffer them? It's pretty hard if you think about it, especially when you have tens of thousands of connections running, but we managed to do it, so we released GRPC retries, that's what we call it, and we also managed in the process to reduce the proxy footprint even more. So the control plane, for example, is also down to three deployments from four, I think, before. We also expanded a bit more on multi-cluster. We and the community had some requests to extend stateful sets in multi-cluster and be able to mirror them, but this is kind of a hard problem to do, and we solved it by the introduction of multi-cluster headless support. We also added fuss testing, some CLI tab completions to make the quality of life a bit better for contributors, and a few other goodies. So in the image here, you can kind of see the depiction of how server authorizations would work. Naturally, it's never that simple in real life, but at least it provides an example. So as of now, this is how Linkerty's architecture looks like. We have a control plane and a data plane. The data plane is made out of proxies. You just inject them next to your application, and this is done by the control plane's proxy injector component. The Linkerty proxy then talks to a couple of components that form the control plane. We have a destination service, and the destination service is used to serve service discovery. You have a target, you have an IP. What do we know about this IP? Can we find out more? So this is where we collect all of the metadata, where we do all of the current client side policies that we have in place, such as retries and timeouts, and where we scope the target out and build a load balancer. Next, we have the policy controller, and this is something that we've introduced in Linkerty 2.11, and the policy controller is pretty much like the service discovery mechanism, but for policy. And finally, we have the identity component. The identity component is super crucial for us because it allows the Linkerty proxies to acquire a certificate as soon as they start, and a certificate is refreshed every 24 hours. So when it comes to certificate management with Linkerty, all you have to do is install it with a root trust anchor, an issuer certificate, and then the issuer certificate will be used automatically to create a bunch of leaf certificates for your Linkerty proxy. So this is kind of how all of the magic works, and this is a default installation. However, you can make Linkerty as robust and as powerful as you want for the extensions. So I would have added a few more extensions, but I sort of ran out of space here, but I wanted to exemplify the two most used extensions I think that we have. The Vis extension introduces Prometheus instance, a Grafana instance, and the web dashboard. Prometheus is going to scrape all of the Linkerty proxies, the proxies themselves proxy the traffic and collect all of the metrics, and then we have automatic scrapes. Well, we have scrapes that ship with a control plane in the Vis extension, and Prometheus will use it to take all of the metrics out. We also have a tap component, and I'm not sure if any of you have used tap when you tried out Linkerty, but it's a really nice tool. It lets you basically have a bunch of streaming lines from the proxy that show you how the connections are being handled. And finally, we have the tap injector, which is kind of like the proxy injector, but for tap, it's pretty self-explanatory. Next up, with the multicluster extension, we have only two components, and that's a multicluster gateway that's used to proxy traffic across clusters. And we have the service mirror that's going to mirror all of the services into your own cluster. Linkerty's approach to multicluster is actually a little bit unique. We're not doing what most other projects are doing out there, and we wanted to design it in a way more simpler way. I can expand on that a little bit later if anyone has any follow-up questions. We're finally getting to my favorite slide. What's next for Linkerty? But before I sort of get into it, I want to first tell you about the Linkerty release process that we have. So Linkerty currently has two types of releases. We have weekly edge releases, and this is usually where most of our functionality and the code that we push on a weekly basis makes its way into, and then we have stable releases. Once we have kind of completed all of the milestone of a stable release, yeah, in weekly edge releases, we're going to create a release candidate that's going to turn into the actual stable release. So with our edge releases, what we really try is to have new features that kind of innovate as fast as we can and get the community input. And while I'm mentioning all of this is because while I have here the roadmap for the stable release, I want to talk a little bit about what we've been doing since 2.11 came out. So since 2.11 came out, we released two more stable versions, 2.11.1 and 2.11.2, and these are focused on backporting some of the issues that we fixed throughout the year in edge releases. Also, we shipped an Apache-style logging for the proxy, so you can have NGINX-like access logs, which is very useful and was kind of requested by the community. We're also working on having a log streaming endpoint for the proxy, so I don't know if any of you have had the opportunity to debug the proxy, but sometimes you have to change the log level, you have to get the logs from your Kubernetes pod, and we wanted to make that a little bit simpler, so we exposed an endpoint to stream all of the logs. You can put them into an aggregator, you can look at them in real time and do whatever you want. And we've also started improving the CNI plugin. So I know that the community had some issues with the CNI plugin and we were super quick to fix that and we already started working on it. Other than that, I think we've mostly had a bunch of fixes and a bunch of quality of life improvements that the community was super vocal about, and very importantly, we released a failover controller for multi-cluster. We wanted to have a way for multi-cluster communications to have like an active passive sort of scenario and have a way to failover and increase the reliability that liquidity brings. So all of this has been happening behind the scenes and edge releases, but then as soon as 2.11 came out, we also started working on the next set of features that the community wanted us to build. For 2.11.12, we have a very big focus on expanding on server authorizations. So right now, server authorizations work based on labels. You select identities or you select networks, but this is all happening on labels. You select basically a set of pods that you want the authorization to apply to. However, we wanted to stabilize in a different pattern. So we sort of took a hint from the Gateway API community and we started doing what's known as policy attachment. So policy attachment kind of allows us to target specific objects and extend the API as much as we want without introducing new types. It's super robust. We're super excited about it, but what it'll let us do is to cover GRPC and HTTP verbs and routes when it comes to server authorizations. So we're gonna have a much more fleshed out route system that's gonna work out way more nicely with the current authorization system. When it comes to CNI, I briefly sort of mentioned that we started doing some fixes there. We received some reports that the CNI plugin could handle different chaining scenarios a little bit better. So we're working on providing better integration and better tooling so LinkerD can work better with other CNI plugins. So if you wanna run LinkerD with Silium, with Calico or anything else out there, you can probably do that as of 2.12. Finally, on the roadmap for 2.14, we're super excited to bring client policies. So we've done server policies. You can now have zero trust in your cluster, but that's not enough. A lot of people want JWT authentication and authorization. A lot of people want circuit breaking. A lot of people want header-based routing. They want egress control. All of this stuff will come with client-side policies. Once these are fleshed out, all of the other systems will be much easier to implement and much easier to extend. Also, we heard that people want Spiffy support, so that's something that's on our roadmap when it comes to integration. And also, I mentioned at the start, there was an asterisk with LinkerD not working outside of Kubernetes. We're planning on doing mesh expansion. So this is sort of what the roadmap looks like. We're super excited to be working on all of this stuff. 2.12 is planned for, obviously, this year. And 2.14 will follow shortly after. If you haven't had a chance to look into LinkerD March at the conference, I know this is already a bit late because it's Friday. The conference is about to end. But I wanted you to be aware in case you wanted to watch some of the recordings. So we've done a couple of deep dives into how protocol detection works in LinkerD. We've also done a little lightning talk on logging and debugging failures with LinkerD and telepresence. And also, we had a couple of really cool case studies made by LinkerD adopters. So for example, if any of you have managed to catch the Xbox Cloud Gaming presentation, they basically talk on how LinkerD helped them secure 22K pods without a lot of effort and stress. Now, we also wanna talk a little bit about the community. So LinkerD is proud to have a very active community. We have a very thriving Slack community where people tend to help each other out. So I encourage all of you to join. chances are if you've joined, you've probably met me already. So that's the case. Hi, nice to meet you again in person. All of our development is done on GitHub. So if you have any questions, if you have any concerns, you can always reach us on GitHub. That's also where we track our milestones for releases. So this is very important because a question that we often get about the project is, oh, when will 2.12 be out? And the short answer is, well, when all of the milestones will be completed. And the best way to track this is by going on GitHub and looking at the issues. Finally, we have third party security audits and benchmarks that have been done. I can forward some links or you can Google them. They're pretty easy to come by. And there are two more things that I wanted to mention. We have a Lincority Hero program. So if you're thinking about contributing, this is super important. The Lincority Hero program is our way of celebrating our contributors because we really appreciate the work that you're doing. I myself started as a contributor, so I'm very fond of the program. If you want to be a Lincority Hero, I really encourage you to join up and put a PR in. And then we also have a Lincority Anchor program. The Lincority Anchor program is there to help people develop their writing and their presentation skills by sharing their Lincority journey. And without being said, I also wanted to say that the Lincority team is looking to grow. So if you want to work with me or if you want to work with any of my colleagues, we would love to have you. So head over to boyan.io slash I love Lincority. And yeah, let us know. That's it for the Lincority talk, but what I want to do now, so hold the claps for a little bit. What I want to do now is take questions because I think it's very important for us to engage a little bit and get to answer anything that I haven't really covered in the presentation. So any questions? All right, there we go. I have a mic, so I'm just gonna try and pass it. You get bonus points if you stump me, by the way. Hi, thanks for the talk. How is the upgrade process between the different versions? Are there many breaking changes? That is a very good question. So our personal recommendation is to always upgrade one version at a time. So for example, if you're running Lincority 2.8, we wouldn't really recommend that you go straight to 2.11. We recommend that you take it incremental. With that being said, we always have to guarantee that at least one version, we are not going to pretty much implement any breaking changes. As a matter of fact, I don't think we've had any major breaking changes up until this point, maybe once or twice when we change some CRD definitions, which again drives home the point that too many CRDs and too much configuration is bad for your system. But yeah, when it comes to upgrading, we generally recommend that you upgrade one version at a time because it's very unlikely that you'll have any breaking changes. Edge releases are a bit of a different thing there. We don't really guarantee that edge releases are not gonna contain breaking changes, but that's life when you live on the edge, I guess. I was about to ask the same question, pretty much so now I'm gonna, can we upgrade on the fly, on the go? Can we update our mesh service without interrupting? Yeah, so that's again a very good question. You can upgrade on the go without having any downtime for your own services. So naturally, if the Linkerty control plane has to be upgraded, the pods have to be re-rolled. So you're going to have a bit of downtime there depending on what the deployment strategy is. If you run Linkerty in HA mode, which is what we recommend people do in production systems, then you're not gonna have any downtime with Linkerty at all. You're also not gonna have any downtime with your services because once they acquire their target through service discovery, the connections are gonna be open for a very long time. So assuming the upgrade process is a success and in most cases it is, you're not going to experience any downtime as long as you run in HA mode. Any other questions? Yeah, there we go. Hi, as you said that a Linkerty works by injecting a sidecar pod, sorry, a container in your pod, is there any advice or best practice to work within a GitHub scenario where there's a tool that is constantly trying to set the state of the cluster to a certain... Yeah, that's a very good question. So Linkerty actually integrates very well with Flux and Argo CD. We have a bunch of guides on how to do that. We're fans of GitOps and we understand that deployment strategies can be a bit complicated, especially when you manage a big platform. Off the top of my head, I don't have any specific advice to give, but we do have guides that are meant to help guide you for the GitHub sort of workflow. But when it comes to proxy injection, one of the things that I do want to mention is that it's relatively simple. All you have to do is put an annotation on your workloads. So if you're using a deployment, that the annotation would go on the pod template, and that's responsible for actually telling the control plane to inject the sidecar proxy. So there isn't really a lot of configuration that you have to do, and there's nothing that would kind of set your GitOps flow apart, if that makes sense. Cool, thanks. Yeah, no problem. Good question. I have a reverse question, which is I just wanted to add a little color to the upgrade thing. So the LinkerD data plane can withstand loss of connectivity to the control plane. So the proxies will continue to operate. They won't get service discovery updates. So in single control plane mode, you actually can upgrade the control plane without having a loss or without having downtime of your application. As long as you're not launching new pods during that process, because if you launch a new pod, the proxy injector won't be able to add the proxy. So you can do it. And if you have control over the cluster, it's probably fine. HA mode, of course, I would recommend for production for a variety of reasons. Once you've upgraded the control plane, the data plane can typically remain at the previous version. So we'll allow one major version of data plane control plane compatibility. And then you can decide then to upgrade the data plane yourself immediately, or you can be lazy about it and just let it happen naturally. As soon as you start getting divergence between control plane data plane of more than one major version, no guarantees. So you kind of have to keep an eye on the version mismatch if you do that. Yeah, that's right. Thank you. Hi, firstly, absolutely amazing, really great stuff, really well presented as well. My question is about UDP. Do you currently have any known issues with UDP, specifically things like HTTP, DTLS? Yeah, so that's a good question. Actually, LinkerD does not support UDP. And it's sort of by design. We're mostly focused on TCP protocols. So LinkerD as a service mesh and LinkerD's proxy works mainly on top of TCP. Thanks. Any other questions? Thanks, very useful. I heard someone saying earlier today that using EBPF, for like as a service mesh has the potential to be faster than a sidecar proxy. I don't know, I don't want to sound like an aggressive question or something like that. I was curious like if you have any thoughts on that, if it's easy to, I don't know, support that as well. Maybe you disagree or, I don't know, I just want to learn. It's a very good question. And actually when you picked up the Mike House Run Ring, am I going to finally get an EBPF question? Because it's, I haven't had it yet. It's a very good question. And I'm going to tackle it into two separate answers. So first of all, the EBPF mode of running a service mesh when you run on a per host deployment, sort of a demon set, that's what we've tried with LinkerD1. So our first approach with LinkerD1 was to have a demon set that runs on a host and it was very hard for us to scale down. So that was one of the first points that I've noticed. The first version of LinkerD obviously predates me. I only started working there, working on a project full-time two and a half years ago. But another thing that I've noticed with it in my research was that it didn't really let us have a good separation of processes. So when we run as a sidecar container, we run next to the container and that gives us good isolation between the processes. When you run on the host, that's not entirely possible. So that's something to keep in mind. And finally, one of the reasons why we wanted to run the sidecar container and why we went with it for LinkerD2 is that it allowed us to scope identity to the lowest sort of unit of work in the cluster. And that's the pod. The only way to really achieve zero trust in Kubernetes is to tie identity to the lowest unit of work. And I think that would be much more difficult to do when you run per host. So the kind of TLDR here is that we tried it before. It didn't really work out and we stuck with the sidecar container for good reasons. It allows us to scale better. It allows us to achieve a better separation and it allows us to do zero trust. So I think for now it's the better model to go with for sure. And then when it comes to EBPF, a lot of the EBPF technology used right now is proprietary. So one use of EBPF for LinkerD2, for example, would be in IP tables. But the kernel modules for EBPF haven't really been contributed or developed enough for us to use it as a backend for IP tables. We're going to look into it when the time will come and when it'll be present as a module in the kernel. But until then, we're going to stick with the IP tables implementation and when it comes to it, another thing is that when you run in kernel space, you're sort of inviting trouble. So that's also something to keep in mind. Good question though. Thank you. Sorry, Matej. I have another reverse question about EBPF, which is, like Matej said, there's kind of two components. There's like EBPF, what can I do at layer four to like speed up how the kernel is processing stuff? And you kind of get that with Cilium as a CNI layer and that works fine for the most part with LinkerD. There's also the idea of the EBPF service mesh. And what happens there is they have EBPF in the kernel, but they also have a single envoy proxy per host. You know, that's a multi-tenant that all traffic goes through. And that's what I really don't like about that model because operationally, you're in a very difficult spot. You've got one point of failure for pods on the node. You've got one place where if something goes wrong, a random selection of pods from your services start going around. That was what we saw with LinkerD 1.X. Is that any kind of like operational task you had to do on a per node basis when the rest of your architecture is set up like service level gets very, very difficult. And Envoy is also, I've got a lot of opinions about Envoy, but it's not designed for this kind of contested multi-tenancy, right? It's just not, it's not. And designing it, designing any software for contested multi-tenancy is really, really hard. It would be possible to rewrite Envoy to do that, but it would be very non-trivial. And it would be, you know, I almost don't know what the point would be since we already have this very nice model, you know, involving containers and isolation, FIOC groups and stuff like that. So you should try it. You know, welcome you to try it, but I don't think it's the right model and I think sidecars are actually really nice. If anyone was at Service MeshCon, I was on a, you know, I talked about this a little bit at the panel at the end. I think sidecars are actually a really, really nice model. If you can make the sidecars fast, if you can make them small, you have really nice operational semantics, you have really nice security semantics. And there are problems with sidecars and there's stuff we can do to improve them in Kubernetes, certainly, when you get into like pod container ordering and stuff like that. It has an operational model and has a security model. They're really, really nice. So we're gonna stick with it. Sorry for the long reverse answer. All right, well, thank you all. That's time now. So if you have more questions, you know where to reach me. Thank you so much for joining. And yeah, give Linkerty a try.