 Welcome everybody. My name is Thomas Graf. I'm CTO and co-founder of Isovalent. Isovalent initially created Cilin before we donated it to the CNCF. Long-time kernel developers. I'm coming from the lower levels and also currently chairing the EBPF Foundation. And I'm Liz Rice. I'm Chief Open Source Officer at Isovalent. You may, if you've been to Kubecons in the past, a long, long time ago, I was a chair of Kubecon and I used to be chair of the Technical Oversight Committee, which gave me this amazing privilege of seeing a lot across the whole cloud native landscape. And when I chose to come and work with Isovalent, the capabilities of Cilin and EBPF had really kind of stuck out to me. So I'm very excited to be working in this technology. So we're going to talk a little bit about how many proxies you need to run a service mesh. I think in this room, we don't really need to tell you what the service mesh is there for. And we probably don't need to tell you the history of how service mesh allows us to abstract functionality that is common across multiple applications. We no longer need to write that directly into an application or in a library. We want to abstract that out into independent service mesh functionality that sits behind the application or in front, depending on your perspective. And for a long time, the model was to use sidecars. So you would have a proxy running in a sidecar container outside of the application, providing all these networking and resiliency and security and observability capabilities that we want to get from a service mesh. But sidecars come with downsides. They're complex to operate. They're complex to inject. They have additional resource needs. We'll dive into this in a little bit more detail in a moment. And they also require network packets to take a much more convoluted path through the stack. When we run in containers, our containers typically have their own network namespace. And we have to run a network stack inside the container, or inside the pod rather, and another network on the host. And if we have the proxy running inside the network namespace for the pod, we've got to kind of connect through that network stack twice before we even get as far as the host network. So there's a lot of path, a lot of processing for packets to go through before they've even left the pod. People have also found configuring sidecars pretty complex. And I love this quote. I mean, it's a couple of years old now, three years old now from Kelsey, but complexity is a real issue with ServiceMesh. So when I joined Isovelin, I remember Thomas telling me that Syllium was already kind of 80% ServiceMesh. We were already providing a lot of the functionality that we expect from a ServiceMesh in Syllium that is traditionally known as a CNI. So could we use this as a basis to remove sidecars and provide a sidecar-free ServiceMesh? And long story short, we did. Around this time last year, just about a year ago, we announced the Syllium ServiceMesh beta. Did anybody here try the ServiceMesh beta? We have a couple of hands up. Great. Excellent. This was a very kind of common comment that we got from people who signed up for that beta. They really liked Envoy, but they'd found the overhead, the resource usage, the complexity of configuring sidecars, the complexity of making sure that all their apps were instrumented, the additional network overhead, the latency involved. All these things were just putting people off from using ServiceMesh. And so they were excited at the idea of Syllium providing a ServiceMesh that didn't require sidecars. But I think something that got a little bit lost when we announced that was that we weren't saying we were completely removing proxies. There's still some proxy in there somewhere, as we'll see. And now, of course, we're not the only people talking about sidecar-free ServiceMesh. Istio now also support a sidecar-free model. And I think a lot of thought that actually getting away from the sidecar model can give us a much simpler, more beautiful implementation. There's a really great podcast about that Ambient Mesh, the Kubernetes podcast. Totally recommend it if you are interested in Ambient Mesh, taking a listen to that, because it's really interesting to hear how those engineers, Ethan and Justin, thought about that model. And one of the quotes from that was they thought for a long time that sidecars were inefficient and that users have a tendency to allocate resources for the worst case. So you can end up in this worst case scenario of resource allocation. When we move to the Syllium model, one of the advantages we have is EBPF, where we can push lots of functionality into the kernel. We can share resources when we're using EBPF much more easily, partly because we're not constrained by this idea of having isolated pods. When you have a sidecar in a pod, by design, that sidecar is isolated from all the other sidecars. It's in a pod deliberately isolated from everything else. When we push as much as we can of the ServiceMesh functionality into the kernel, it is shared, and that can be a really good thing. The kernel is there to be shared amongst all the applications that run on it. I talked a bit about the convoluted network path that packets take through a sidecar. And when we move to a sidecar-free model that shortens the path even through a proxy where that proxy is sitting on a node somewhere. So the ability to remove sidecars solves a lot of problems, but Thomas, where should the proxy live? And given my intro, maybe you already have a slight hint, given I've been doing kernel development for the majority of my career, I think proxy functionality, ServiceMesh functionality should be as transparent as invisible as possible, and the kernel is actually a great place for this. But this does not mean that then the kernels need to solve all aspects. The kernel has always been delegating functionality from kernel space to user space, and we'll see a couple of examples. In the case of ServiceMesh or ServiceMesh in general, there are aspects that are not great matches to be solved in the kernel, such as TLS termination, HTTP authorization or complex HTTP parsing, and of course identity management, TLS control handshakes, and so on. We'll get into the details here. This means that ServiceMesh has this advantage that we can be EBPF native when possible and benefit from the performance gains of EBPF and then fall back to an envoy proxy when EBPF is not the ideal instrument to implement. Other examples I mentioned that this has actually been the case for kernel development for ages. The kernel has always delegated some functionality to user space. Very good example is IDS. So Riccardo running in user space and doing deep packet inspection using IP tables, NFQ to punch packets user space, or even just kernel module loading when your laptop loads the Wi-Fi driver. It actually, the kernel module is getting loaded through a so-called user space mode helper. So let's look at what functionality can be done in EBPF native and where do we still need a proxy? I mean, then we'll talk about the number of proxies afterward. There's a lot of functionality that's already possible EBPF native without any proxy at all, purely in kernel space. Of course, any sort of L3DL4 forwarding, right? Any sort of L3DL4 load balancing, but also cannery rollouts to polygyro routing, multi cluster routing, network policy on layer three, layer four, and we'll talk about the details here, MTLS, as well as a variety of observability functions, HTTP, TLS, tracing, TCP, DNS, TCP, UDP, observability, and so on. When do we still need a proxy or for use cases such as layer seven load balancing, for example, path based load balancing, header based load balancing, retries layer seven rate limiting, as well as TLS termination and origination. Currently still in the proxy land is layer seven or policy. This is road map to be EBPF native, but currently not implemented yet. So this is the current split, which means that we can benefit from the EBPF performance and simplicity whenever possible. And if we cannot do it in EBPF, we essentially fall back to a proxy model. So what do these different possible models of where we put the proxy? What do they mean in terms of pros and cons? Because ever since we first discussed having Sidecarler service mesh, people have been debating the pros and cons of where that proxy lives. So if we think back to the sort of original Sidecar model, we had potentially what could be seen as a security advantage that the proxy shared all the same isolation that the application pod did. So it naturally has access to the application's secrets. It's isolated in the same way as the application is. So from a security model perspective, you don't need to think about it anymore because it's the same as the application. And Cilium has supported this model for Sidecarler for service mesh for a long time now. It has been possible and people are running Istio on top of Cilium as the networking layer. That's a completely supported model. But if we want to move away from that, we want to not have a proxy for every one of our application pods. If we put the proxy on the node, what does that mean? Well, it means on the one hand, it's co-located on the same node. So our network path can be pretty efficient. But it does mean that if we have multiple tenants that are not on the same node, they can't all share that same proxy. You don't need to have additional proxies elsewhere on the network. And you can have this very, you have a proxy right next to you, which I think is a good thing. This is the kind of default model for Cilium service mesh today, where we would have one proxy per node. But we are working on the idea of having multi-tenancy by having multiple proxies running on the node. So that can give that isolation that people are potentially looking for. Another model that we're seeing now is this idea of having the proxy not necessarily co-located on the same node, but somewhere in the network. And that pretty much guarantees that there will be some extra network hops. But it does mean you can share that one proxy amongst multiple instances that are in the same service. And this is, you know, I would say a relatively new idea. It's something that we're interested in and would like to understand whether Cilium users would like to see this model. It would certainly be possible to implement that Z tunnel approach in the Cilium layer. But I think most importantly, there isn't a single solution for, you know, you must have proxies, this number, this ratio of proxies to applications or this particular location of proxy relative to your applications for some require for some deployments for performance reasons, you will choose to have a proxy on every node. That will almost certainly give the highest performance and probably reduce complexity in other environments. You may choose to emphasize more multi tenant isolation. In that case, you may want to have more proxies, perhaps as much as one proxy per application. So it does depend somewhat on your risk model. And there is no single answer. What about encryption? What about encryption? So we talked about number of proxies and we figured it's actually interesting to look at the most common use cases of why you want to use or why you use a service mesh and encryption and observability are probably among the top asks that you have. So let's look at encryption first. First of all, if we look at encryption, there is two broad models to achieve encryption. We can achieve it at the network level using functionality such as IP sec or wire guards, where we essentially encrypt in higher network nodes to note. This can be fine for many use cases and has been the status quo in networking for a long while, but it does rely or it does, it does require that you trust your node. So if the node gets compromised, the key on that note gets compromised and then the network encryption gets compromised until the key gets rotated. So it uses node identity. That can be that can be great if that is your security assumption. What if we want deeper security, better security and actually take it one level further? Obviously MTLS comes into play. The downside of MTLS as implemented with side cars is that first of all, it is shared with the layer seven proxy, which means that if the layer seven proxy gets compromised, all certificates and secrets in that proxy get compromised as well. And the second big downside of using side cars for MTLS is that is pretty much limited to TCP. And in particular enterprise use cases will require multi-cast UDP, SCDP and a variety of other protocols that need to be supported. So we looked at this and asked ourselves, can we support an MTLS model, a mutual authentication model, while benefiting from the network level to support any network protocol? And this is what we came up with for all the details on how exactly this works. I've linked the blog post that goes into the details. We'll also see the poll request with the initial work later on. Essentially, this splits the authentications piece from the payload, from the data path. And this means that the authentication happens through a regular MTLS handshake where we can integrate with Spiffy, serve manager and so on and do a regular mutual authentication using certificates that are per service. And then instead of instead of carrying the actual payload and network data, the data in that MTLS connection we carried out through the network directly while using the keys to secrets that have been negotiated via the MTLS handshake using IP second wire guard. And we kind of gain the benefits of both models where we get a strong mutual authentication handshake with MTLS while we can support any network protocol without introducing any sort of proxies. So we can establish encryption and mutual authentication network wide without requiring proxies while supporting any protocol. Let's look at how this actually looks like because let's say you're not using this and you're going, you're, you're going down this road and want to use this. We're integrating this into a system that our policy, which is an integrated, which is an extension of Kubernetes network policy. And all some users have to do is essentially opt into this and say, for these connections, let's say front end talking to back end, you need to authenticate. And as soon as that intent has been defined, authentication is required before the connection can be established. The actual service specific certificates are in this case provided by Spiffy serve miniature integration is also on the roadmap. But currently the pull request and the issue that I've listed here on the slide is essentially enabling the Spiffy integration that allows not only to generate certificates for each service and thus give or make X five or nine certificates available to psyllium. It also allows us to actually tie network policy to Spiffy IDs, which means instead of selecting policy by our Kubernetes labels, we can actually use strong Spiffy IDs. So essentially we solve two problems and ideally because psyllium already has a concept of identity, it's simply or easily maps to Spiffy IDs and Spiffy service keys. What about observability list? Well, observability is a pretty hot topic for us this week. Um, I want to actually give a shout out to Adam who gave a talk earlier about how you can use EBPF to kind of gather metrics in a very efficient way because basically that sort of shows the sort of underpinnings of what we what we do here and how we're gathering some really good metrics and then making them available. So how am I going to do this? Okay, my actually, no, I'm going to show one more slide before I do that. We this week we're doing a lot of work with the team at Grafana to show off some really nice integrations between Grafana and psyllium. I'm just going to show you kind of the tip of the iceberg here, I think with this, this demo today, but there's, there's some really nice integrations here. So demo time. So I have a an open telemetry demo app. I'm going to guess that some people here will be familiar with the app. It looks like this. I'm going to just refresh it just so I've started getting some network traffic. Um, and it's sitting behind an ingress, which is a psyllium. It's a Kubernetes ingress with, um, of psyllium type. And as a result of that ingress being created, we should have actually, let me see if I can find, no, I'll show this first. This ingress causes a psyllium envoy configuration to be created, which is that one. And hopefully I'm trying to do this with one finger. That's why I'm um, so that envoy configuration was automatically created from the ingress configuration. And what I'm looking for is the bit that says here we go. It's basically going to send all traffic from, you know, slash anything to the demo front end. So this is a very, very simple demo, but we are going through an ingress layer seven routing here. So let's just do a little bit of, um, telescope shopping to generate a bit of traffic. Um, I think we can buy these. I've bought a lot of telescopes today. It's quite, quite good. Come on, let's place order. I don't know why that didn't, doesn't really matter. I'm just trying to generate some traffic. And when I generate some traffic, you can see here at the right hand side, those are my requests, my most recent requests and here is me just checking that the demo works a few minutes ago. Um, and we get this really nice layer seven metrics generated by psyllium through EBPF. A lot of these, these metrics are being gathered by using EBPF and, uh, they're generated in Prometheus format. So they're very easy to pull into Grafana dashboards like this, uh, getting things like the duration, the latency rather very nicely graphed out. We also get a service map. Now, if you're familiar with psyllium and Hubble, you will probably have seen the Hubble service map. This, this is something that, you know, we've had in Hubble for a while and you can see individual network flows as well as the way those flows go to build up an overall service map, but we can now also see that service map inside the Grafana dashboard. I hope there we go. Yeah. So we have this really nice ability to gather metrics. As Adam was talking about earlier, it's very efficient and it's incredibly useful debugging and observability tooling available through this. I'll leave it to you to explain the rest. Yeah. So I think we've seen nice demo. Obviously this also works for just pure network observability on the lower levels. We've seen the layer seven, obviously a free layer for all of this comes integrated into, um, kind of one set of dashboards. Very nice that hobble UI can now directly embed Grafana dashboards. We can also do tracings or is an example of a, um, integration of spans, uh, displayed in Grafana tempo generated by, by hobble, as well as the goal single dashboard that we've just seen. The last point, I think we've looked at observability. We looked at encryption and TLS observable. And the last part is kind of traffic management. So how is it possible to define and use traffic management in Selim service mesh? And we want to quickly show like what is the state of the individual configuration options that you have ingress, gateway API and Kubernetes services, as well as the envoy config. The rest of the stack, I think is hopefully pretty obvious. Obviously Kubernetes has the control plane itself. You don't really need anything beyond that. Um, two of the secrets as the current identity management integration we have an on the road map is spiffy and serve manager and then the data plane, EBPF and on the way. So let's look at these individual layer seven traffic management options ingress services, gateway API and on the way config ingress and eight and on the way config are already fully supported in the one 12, um, services with annotations and gateway API are coming in one 13, the code is actually complete. It's just not merged, but it's available in deaf branches for testing. If you are interested, I've linked the pull request with the branch links in there ingress, very simple. It's simply fully conformant. So it's a basic example of path based or path prefix based routing using ingress. And that's what you saw in the previous. That's what we saw in the demo. Exactly service annotations. Like some of our users said, well, that's great. Ingress is great, but we really want to have this most simple to use case possible for layer seven low balancing. So we simply added a bunch of annotations as a Kubernetes services to enable least weighted, um, least, uh, rated least request, g RPC low balancing where all you have to do is annotate your Kubernetes services with these two annotations and you get transparent g RPC low balancing. Very, very simple. And it gets better because Syllium has multi cluster capabilities built in at another, another annotation to mark this service as global and while a lot of the service low balancing works across clusters now. So this is the simple, the simplest option, right? Okay. Ingress services, if annotations and now gateway API, right? Initially or originally called ingress we do probably the way forward for most of you, how to define traffic management in the future. We have the code complete to support the latest gateway API spec. This is a typical example using the gateway and HB route resource also using a path, path prefix. And then the last option, and this is really kind of a low level building block, but it's also available to you as users. You can actually define and configure raw envoy configuration in a custom resource. This is how the ingress controller and the gateway implementation actually map or use a Syllium service mesh. So they essentially translate into Syllium service mesh. But if you want to use this lower level API, you're absolutely free to do so. This way you can essentially use the full envoy feature set in your own service mesh. Definitely, as you can even see in the example, definitely a lot harder to use, but with a lot more capabilities. And that is it from our side. I think we have a couple of minutes for questions. Thank you. For MTLS, you mentioned that there was an initial handshake that was MTLS and then the data was pushed somewhere separately. Does that mean that data is encrypted and confidential? Yes, correct. So MTLS is used for the handshake. The MTLS handshake generates the secrets, the keys, and they are handed down to WireGuard and IPsec where the symmetric encryption happens in the kernel. So HMAC and encryption is happening in the kernel. But the authentication handshake is done in user space using MTLS. Thank you. When you use mutual tiers, do you use that also for exchanging the certificates for the workload? And when you use both mutual tiers and WireGuard, how do you avoid the double encryption? Because by default, mutual tiers has an encryption right. So hopefully I got the question right. So we are using, in the example shown, either Kubernetes secrets or certificates created by Spiffy. So those are service-specific certificates for the MTLS model. And in our MTLS model, the data gets encrypted using IPsec and WireGuard. So there is no additional TLS-based encryption. There is no double encryption. Hopefully I got your question correct. Otherwise, please clarify. Any other questions? I didn't catch on that slide where you showed the service with the annotations. Was that a service of type load balancer? No. It is a cluster IP. It's a cluster IP service. Okay. Yes. But it would actually also work with of type load balancer. So Cilium also implements type load balancer. And then the cluster IP portion just comes after it. So it also works for node port and load balancer. How do you get your envoy between the load balancer and where it's going? We use EBPF to inject envoy into this picture. Cool. I got you. I got you. I got you. No worries. It's my responsibility. I'm not get paid for this, but you know, I take my responsibility seriously. Thanks, Thomas. You had one slide where you were showing alternative methods to SciCar proxy or even co-locating the proxy. And it was just a proxy on the network. Is that kind of just taking that simple load balancer concept that we know from yesterday year? That was, I guess, a nod towards our understanding of how ambient mesh is doing proxies that they're not specifically located on any given node. I mean, we've kind of shown it floating in space. I think it would be on a node somewhere, but the point was more it's not specifically associated with the node where an application is running. It could be in space. Is it fair to say it can be like a part of namespace or something like that kind of a little bit more abstracted? Yeah, I mean, I think for both models where it's on the node or on the network, any granularity could work. I think it's more just location versus ground. I think the benefit of the network model is that you can share that proxy for the same tenants that are not on the same nodes. Whereas if you're on that node, you'll share it only with tenants on that same node. That's the clear difference. And I think that's a great example of a trade-off where you might minimize network hops in order to but you may have to have multiple instances. So I think from a Selenium service mesh perspective, we're interested to support both models. So if you want us to also support like the proxy on the network model and support waypoints, file a GitHub issue or just talk to us on Slack and we're happy to add it. It's actually very compatible with how we think about service mesh as well. Hey, fantastic talk. With the like sidecar for every pod, those individual envoy instances are only going to have their tunnels to whatever the application happens to be talking to, right? Whereas on the node one, I know this came up in the Istio example as well. It's like on the node, you've now got a single envoy instance that's going to have to do all the wireguard or IP set tunnels from every application on that node to every application that that's talking to. Like, are there any concerns around overloading an instance? Like, I don't know. I presume there's been a lot of testing on this. I'm kind of curious if you can talk about how that's worked for like larger clusters. Yeah. So I think the benefit is that on the proxy on the node model to get a revive the second wire guard, we don't need any tunnels at all. It's purely on the network level and that works for 10 thousands of nodes. So that uses this, the standards to lay in that work layer encryption. So we're not actually creating tunnels like peer to peer tunnels at all on that, on that node. But maybe, maybe clarify your question because I think I got it wrong based on your, but we can also talk outside. Well, I think we have a time for one more question. And I think I have working for one last question. Hi, thank you. That was a great talk. I was wondering for requests that need to go through that envoy proxy. How do you scale that proxy as the requests increase? What mechanism do you have there? Yeah. So envoy has several mechanisms. First of all, you can define the number of listeners, like how many threats on what we can use. And then we also limit the resource per listener. So when we have a shared envoy instance, we actually configure a separate listener for every ingress and for every egress use case or a pod that actually needs, and we limit the number of HTTP requests or layer four requests or layer four connections and scale by the number of threats. And obviously it's also subject to the CPU memory limits that you configure. All right. So let's get around the four plus to Liz and Thomas.