 Hossiaeth yma, ddod ddod i ni eich cyfnod â y ddechrau fyddo o'n fwyngo. Mae'r byw yw'r ysgol yn gweithio. Mae yna, ddod. Ond, ddim yn ei wneud. Felly hynny'n Liz Rice. Roedd y cyfnod yw'r cyffredinol cyrcaf ymlaen ymlaen. Ymlaen ymlaen yn y cyfrifoedd mewn cyrcaf yma. A ydych yn ei wneud i'r cyfnod yma o'r cyrcaf ymlaen yr ysgol. Ac rwy'n gweithio Isovalent yn ystod o'r ysgol, ac rwy'n gweithio bod rwy'n gweithio EBPF. EBPF mae'r tectnologi sylwyr sylwyr. Rwy'n gweithio Isovalent yn gweithio EBPF, ond rwy'n gweithio Isovalent yn gweithio'r cyffredinol. A rwy'n gweithio'r cyffredinol, Thomas Graff, sydd rwy'n gweithio bod rwy'n gweithio. Rydyn ni'n gweithio'n gweithio sylwyr yn gweithio 80% o'r cyffredinol. Oedden nhw'n gweithio arall y rhai oherwydd eich cyffredinol. Mae'n gweithio allanol rhywbeth o'r cyffredinol o'r cyffredinol, yw'r ysgolwyr. Felly yw'r ysgolwyr yw'r ysgolwyr, fydd yw'r ysgolwyr yn ymgyrch, a'r ysgolwyr yn ysgolwyr, ac yn dweud y bydd y bydd yma yw'r ysgolwyr yn ddweud. A'r ysgolwyr yn ymgyrch yn ddweud. Felly mae'r wneud â'r ddweud yn ysgolwyr? Mae'r ddweud yn ddweud. Dwi'n rhai bod yna'r ddweud yn ysgolwyr? I see a hand back there, so thank you very much. Oh, a couple of hands, yeah, awesome. So when we, I don't know why that. So when we announced the Selim service mesh beta programme and our artist Vadim came up with this nice little bit of artwork about getting rid of sidecars and I think some of that has been a little bit controversial. So I hope I'm going to explain today why we're preferring a model that doesn't use sidecars and some of the advantages that that gives us and how we've been able to get there. So Selim as a CNI, it's based on EBPF that I mentioned, it's a very high performance network plug in for Kubernetes. You can also use it standalone, but I think the majority of users are using it in a Kubernetes environment. It implements connectivity, what else is networking for, then to connect our workloads to each other and to the outside world. It implements security features, we have network policy, we have network layer encryption and it provides load balancing, whether that's as a cube proxy replacement. So load balancing between the different pods that make up a service or also as a standalone load balancer. There's also a component called Hubble, which gives us visibility into network flows. So we can get detailed flow logs of every network packet and from that the Hubble UI can show you a service map showing which services are communicating with each other. We get lots of metrics that we can export to Prometheus and produce Grafana dashboards. We'll see a few of those in a moment. And we get visibility into the layer 34 and layer 7 traffic, so layer 7 being the application level protocols. Let's just step back a minute and talk about the history of Service Mesh. So before it was even called Service Mesh, applications that are potentially written in different programming languages needed to communicate with each other. They might be sending HTTP messages between themselves or GRPC or whatever protocol. Perhaps they're implementing TLS for security and each application back in the day would need to import its own library to implement those networking functions. If you wanted to do the things that we associate with Service Mesh today like canary rollout or retries or rate limiting, well, good luck, that's going to be a significant amount of work to implement in all of your different applications. So we move to the sidecar model. Now Kubernetes allows us to group containers into pods. Pods share a network name or all the containers in a pod share the same network namespace. So that allows us to have a proxy called a sidecar inside the pod with the application container providing the Service Mesh functionality. So, for example, your application doesn't need to worry about setting up TLS. You can leave that to the proxy to take care of. You can leave service discovery. You can leave retries. You can leave all the load balancing up to the sidecar proxy and it makes life far easier for the developer. So this is great. This is much more efficient from a development perspective. And also, because containers are containers, you don't have to have the same programming language. Those sidecar proxies can come from a third party. And indeed many of them do. A lot of people are using Envoy, for example. There's also LinkaD. The proxy can be the same in every application pod regardless of the language of that application. So the question comes to mind, can we take this Service Mesh functionality that started in the application and then moved into the pod? Could we bring it into the kernel? Why would we want to do that? Well, networking stacks used to, back in the day, when all of us were very small children, a networking stack used to be written in the application. You'd have to write your TCP IP probably as a library, but you'd have to have that in user space. Now we all expect our networking stack to be taken care of by the kernel. If we think of Kubernetes as a distributed operating system, Service Mesh is really the equivalent of the networking stack. Could we pull that into the kernel today? And the answer is mostly not quite everything yet, but mostly. Layer 7 is the part that we don't really have today. It will come, but we don't have that right now, this moment. But all of the Layer 4 functionality and beneath is already being handled. We've seen that in Cillium. Cillium handles Layer 3, 4 traffic, provides load balancing, provides security, provides observability. And yet Cillium already has visibility into Layer 7. Let's... Actually, I'll just skip to the demo first to show you what I mean about Layer 7 visibility. So, is that big enough to see at the back? Is that okay? Somebody wave if you can? Yeah, I see some thumbs. Good. Right, so what do I have in this namespace far, far away? Come on, network. There we go. Okay, I have an X winning a TIE fighter and a couple of death stars. And my... Let's suppose my TIE fighter wants to land on the death star. Well, first of all, I'm going to start off some observability. Run Hubble to watch traffic in that far, far away namespace. And if I exec into my TIE fighter, and... Actually, I'll find a previous... previous one of these. A bit easier. Let's not do that one yet. So, I could request from a TIE fighter that I want to land on the death star, and that's okay. I'm allowed to do that. In fact, I could find out all sorts of... Find out all sorts of things from this... the death star API. Let's see what's available from the death star API. Yeah, there's all sorts of things I can do with this API. Good to see the Empire has adopted Cloud Native. So, I don't know if you've been noticing these flows going up in the top half of the screen, but we have things like this where we've got visibility into the HTTP request. This is layer seven, and we can see the URL, the path that's been requested, and we can see the response here in this example. That was a 200. So, we're getting all this visibility into the traffic flowing within the namespace far, far away. And we can also layer seven provide security with network policies. So, suppose I'm the security officer on the death star, and I probably don't want people putting things into the exhaust port. This looks a little bit dangerous. We can just see what happens if we do that. I think I've got an example of that from back here. Yeah, that's a bad day on the death star. So, if we want to prevent that from happening, we can use a layer seven network policy to do that, and I have one already prepared. So, this is going to match, at layer three, four, it's going to match endpoints, no, it's going to match endpoints in the empire, in the death star, and this is what it applies to, and it's going to police ingress traffic. We're only going to allow traffic from other empire organisations. No landing from the rebel alliance, please. And we are only going to allow TIE fighters and other empire equipment to post to request landing. Everything else is not going to be permitted. I've also got, this will come into effect later on, I'm going to allow external traffic from world outside of the cluster. We'll also only be allowed to post to request landing. So, if I apply that rule, layer seven rule, I think, and that will create a psyllium network policy that should prevent the exhaust port from having anything put into it. Okay. And we can see, just scroll off the screen, we can see that Hubble was able to see that level seven request again, and we can see that it was dropped. It was out of policy. And if I take a look in my Grafana dashboards, we can see information about when requests have been placed, we can see things like latency, we can see DNS requests and responses and so on. All of this has been generated by Hubble and psyllium metrics. So we get a lot of visibility into what's happening. So we've got that layer seven visibility that we need for service mesh. How did we achieve that? We achieved that by having an envoy proxy. So built into psyllium, we use envoy. Psyllium is using EBPF to dynamically program the kernel with network endpoints and connections between them. And whenever it needs to terminate layer seven, be that for policy, be that for visibility, it's using envoy for that. So if we want to implement a service mesh, what do we need? We need observability. This is actually a graph of what our service mesh beta users asked us they wanted to see from service mesh features. By far the most popular feature was observability. We already have that. People are very interested in Kubernetes ingress. They're very interested in being able to do those kind of traffic management features like canary rollouts or circuit breaking or rate limiting. They're very interested in encrypting the traffic between services. We tend to assume that's going to be MTLS, but let's hold that thought. We are interested in authenticating services to each other and we're interested in encrypting the traffic. But we've split those two ideas out for a reason. So people wanted to see observability. They want to see ingress, which is going to require load balancing traffic that comes from outside the cluster sending it to the right backend services. In order to do that, we're going to have to be able to parse layer 7 protocols so that we can do path-based routing or routing depending on where you're from in the world. We need to be able to have those more detailed rules that we anticipate with service mesh. So to be able to load balance traffic to different pods within a service based on perhaps what version they are, and we want identity-based security. Something I didn't point out but you probably noticed in the Hubble flows is we can see the identities of the Kubernetes services that are involved. We can see the namespace, we can see the service, we can get all the details about which pods are sending and receiving traffic because Cillium understands the Kubernetes identities. So a whole bunch of this functionality already existed in Cillium before we did anything to call it a service mesh, and I think that's what Thomas meant when he said, it's already basically an 80% service mesh. So we have had to do some additional work and that is essentially to say we're not just going to use Envoy for the policies and observability, but we're also going to start programming Envoy listeners to implement those different rules. All those traffic management functionality that we associate with service mesh is implemented in Envoy and we're going to have Cillium program those listeners through a CRD that we call Cillium Envoy config, programming that in user space and programming EBPF for the network traffic within the kernel. Now let's have a little look at Ingress. I have an Ingress for the Death Star so that not only can we have traffic within the cluster but we can also allow tie fighters who may have been somewhere else in the university land on the Death Star. So let's have a look at what that's created. That has created Cillium Envoy config. This was automatically generated by the definition of the Ingress. So let's have a little look at what the Cillium Envoy config looks like. So this has a listener, the name of which was automatically generated and this is going to route traffic that comes from outside the cluster to request landing and it will route it to the Death Star service. I can also show you that the Ingress has created the service automatically here. So this load balancer matches the Ingress that I just showed you. So this very complicated address we're going to need and I should be able to curl, let's post to request a landing and the ship can land from outside the cluster. That was only permissible because I still have my network policy in place and I allowed for external traffic as well as internal Empire Mark traffic to be allowed to reach that destination but I can't do things like put to the exhaust port. Let's hope not because we don't want anybody blowing up the Death Star. That's not going to get there. In fact that has been dropped by the Ingress so we never even saw the traffic here. Again we're going to see the Grafana visibility or Grafana dashboard showing us things like the latency of these requests. We can see when things get dropped, why they get dropped so we should see somewhere, traffic being dropped because it was out of policy and so on. I showed you the underlying Cilium Envoy config that programmes the Envoy listeners. Those are pretty low level CRD and although if you take a look at the service mesh examples that we've created there are some examples of traffic management where you would manually programme those Cilium Envoy configs but that's not really the end game. The end game is more akin to what we've done with Kubernetes Ingress where by defining the Ingress there's an operator that automatically creates the corresponding Cilium Envoy config. So we will, as I'll come to later, we will have other abstractions for configuring Cilium Envoy configs for the other features that you expect from service mesh. We had beta testers telling us why they like this model. We asked for comments on why people were interested and this was a very typical comment. People who tried using service mesh and have found either the performance or the complexity of the sidecar model problematic. So I have a few examples of user comments about this. It's not just the overhead, it's also the management complexity that people found challenging and the idea of not having to manage such a complex set of resources people find very attractive. It's another example. Having a sidecarless service mesh with a C&I sounds like a perfect solution. If we looked at the sort of summary of those surveys and I've linked to this in the presentation which I haven't yet uploaded but will shortly, you'll find the survey data where a lot of people mentioned the operational complexity of service mesh, they mentioned the resource consumption of service mesh, they're looking for better performance and they've also had problems with the start-up of sidecar containers where you've got perhaps init containers, you've got your proxy, you've got your application containers that all need to start within a pod and can cause race conditions or otherwise have issues in the order in which those containers come up. Talked about reducing resources. If every application pod includes a sidecar and that sidecar has a bunch of routing table information, for example, if you have to duplicate that for every single application pod, that can be a significant amount of resources. If we only need one instance of that routing information per node, that's a significant reduction. And we're also making the network path, so the route that network packets need to take within the machine, making that much more efficient. This is an example of how a network packet needs to flow from an application through the networking stack in the pod to reach the service mesh proxy and then from there it needs to exit the pod through the networking stack to get to the host's networking stack which it then needs to traverse to get out of the physical machine. With EBPF, we can connect the socket layer from the application essentially directly to the networking endpoint, to the network card, the network interface. And if we don't, many packets do not need to be rooted through the proxy. We only need to route packets through the proxy when we need to get at that layer 7 information so lots of packets can go straight through EBPF to the network interface. But when we need to, we can route things to Envoy for layer 7 termination as necessary. And that does give us the performance improvements that you might expect. You still have a proxy on the node but you don't have to have two proxies if you want to go from one service to another service on the same node. There's a really great blog post that Thomas wrote that explores this in more detail. We've also published the details and the scripts of how we've made these performance measurements. We would love for other people to go out and replicate them and tell us if we've made mistakes because it's obviously important that we get this right. But I think it does intuitively make sense that if you're not having to go through such a long network path, you will have better latency. And the throughput is pretty similar, marginally better from what we tested, but pretty similar. One thing that was dramatically improved was pod start-up time. Don't have to wait for the proxy to become ready as well as your application that can make a very significant difference. So I talked a bit about the Cilium Envoy config being kind of low-level abstraction. There are many ways of configuring a service mesh today, whether you're using Istio or Linkedy or service mesh interface. What we've done with Cilium service mesh so far is really about making that data plane much more efficient. How we program that data plane, how we program those envoy rules, we're pretty agnostic to what the right control plane abstraction is. Whatever is the right control plane abstraction for you can be integrated. It's just a case of providing that mapping between the service mesh... I want to say service mesh interface, but I mean any of those abstractions, how their control plane configures those abstractions and mapping those to the underlying Cilium Envoy configs. So there's some work to be done, and if you want to get involved in contributing the integrations between these interfaces and the programming of the CEC, that is work that needs to be done in the project. As an example, what you saw with the Kubernetes ingress, there is an operator that is taking that ingress information and programming in the underlying CEC. So that's an example. There will be more. So there's one piece of the puzzle missing. We've seen connectivity. We've talked about how we can program the envoy proxy. We've talked about visibility. We have not talked about MTLS much. The plan that we have here is to separate the authentication part from the encryption part. And the reason why we want to do this is because we already have very efficient network layer encryption, be that IPsec or WyGuard. The encryption can happen within the kernel, but we need to get the certificates injected into the kernel in order to perform that encryption, and we need to validate those, authenticate those certificates, authenticate those identities when we're setting up a network connection. So on the roadmap intended for the next, so 1.13 release, will be the separation of authentication in the user space layer and using that to control network layer encryption. Again, we're pretty agnostic to where those certificates come from. It could be spiffy. It could be cert manager. There is work to be done here to integrate those with the authentication layer. But we see this as the future of providing those encrypted connections between services. And again, there's more detail in a blog post on the ISO surveillance side. So pretty much to sum up, the intention here with Cillium Service Mesh is that we can produce the most efficient data path in much the same way that Cillium uses EBPF to provide a much more efficient data path than IP tables-based networking solutions. We're extending that even further to Service Mesh. But we're still using Envoy where layer 7 termination is required. By using EBPF, we're getting much faster performance, almost as fast as just pure node-to-node network performance. And we can support any protocol. There is work to be done to bring the control plane of your choice to make it easy to configure the Service Mesh. And there is work to be done to bring the certificate management of your choice to encrypt the traffic. The observability integrations are essentially already there. So we can export those metrics. We have all that flow log data, all that telemetry goodness already there. And this makes Service Mesh part of the Cillium family of very efficient networking, observability and security enforcement. So I hope that's explained something of how we've gone from Cillium as a network interface to Cillium as a Service Mesh. And if you want to find out more, the Cillium website, the Cillium project on GitHub, Cillium on Twitter, we have an amazingly active Slack channel which you'll find linked from the Cillium site. And also there is, if you have any enthusiasm for, actually I'm not sure if the expo is still open, so it may be too late to say go and pick up a book, but you can get the PDFs of these two books that we've recently written from the ISOvenant website if you're interested in learning more about EBPF or about security observability written by my colleague Natalia and Jed. So with that, I think you're almost at the end of the day. I think maybe there's one more session to go. So well done for maintaining stamina for this long. Very much appreciate that you took the time to come here at the end of the week. And yeah, I don't know if we have a minute for questions. So yeah, if we have questions, let's do it. Thank you very much. I think there's some microphone action going on. Hi, there's one question from people who are virtually at the conference. So they ask about your opinion about GRPC Proxyless Service Mesh and do you see a different goal for that approach? So we've already got examples of GRPC, certainly for Ingress. It's essentially the same. We're agnostic to the protocol. Any other questions? The mic is there. So you didn't really move the data plane into the kernel and why it's still running in the user mode? Is there some thinking around moving it into the kernel? I think ultimately that will be the future, yes. And we already have some layer 7 passing being done in EBPF. I'm going to say more experimentally than in production, but so, for example, we do have an HTTP2 parser that runs in EBPF. So I do expect that eventually it will be possible to run the protocol parsing parts in the kernel. Hi, Liz. Can you talk a little bit about the trade-offs that there are of going from a per pod security context where Linux isolation features are being per proxy in the sidecar to trusting a single host proxy to work with multiple workloads? Yes. So this is, I think quite... There are various different schools of thought on this and I don't think we've completely got to the bottom of every possible argument here. I would say we have in the kernel namespaces and C groups that we use to isolate network stacks already. We don't worry about the fact that we're running pods using a shared kernel to run our network stack up to layer 4 anyway. So now what we're doing by running this in Envoy, obviously this is user space rather than in the kernel, but we're using that listener concept as a kind of isolation process. Now I would say if you're in a high security, you're very worried about the multi-tenant situation, perhaps you don't want to run those different tenants on the same node, but I don't think that... I think it's going to be on a case-by-case basis whether that's important. I've also heard people worrying about the potential for interference noisy neighbours, the resource constraints that are implemented using C groups in the kernel, but if we're in Envoy, how are we doing that? Definitely room for research there, but I would say if we're going to end up saving 40% of your resources, then perhaps that's going to solve the problem anyway. It's trade-offs, right? Absolutely. It's like C++ implemented parsers, and then it's like you have user data flowing through it. It makes sense. Thank you. Appreciate it. Hi, Linz. Hi. It took us more than a year to migrate more than 3,000 pods to a steel-based service mesh for us. I'm not sure if you want to do another migration. What is the... What would you recommend the easiest migration path from Istio to Siloom service? That is a great question. I think it... First of all, it is pretty early days. We're in the closing stages of the first release that will have ingress in GA, so we don't have a whole load of experience of people migrating yet. If there is a significant group of users who really want to take those Istio CRDs and keep those in place, but use those to programme Envoy, that could be one of the options. At the moment, we've seen a lot more interest in doing SMI, and also in doing annotations to Kubernetes. But I think if there is interest from the community, then making it easy to migrate from Istio could really involve just taking those existing Istio CRDs and having the integration to use that to programme CECs. You can't do that today, but I think that will be... Thank you so much. Hi, good. I'm interested in... Do you think the tooling... I think it goes well, this is great, but if things go sideways, like you're turning to debugging the kernel, basically, which is a whole different story than working on user space. So... I don't think it's different from what we already have in networking in the kernel with Silium. And we have some great tools for observing that. Hubbell gives us a lot of visibility into where there are various different hooks while we're hooking EBPF programmes. We can use that to get visibility. There's a lot of, you know, detailed information that gets generated by Silium when needed. We have a ton of metrics. I don't know how that maps to sort of what's happening in the on-voice space. I'm sure there will be learnings that need to be more tools, but I don't think the fact that it's moving to... or that you might be moving to a kernel-based networking layer, people love Hubbell for debugging networking issues. And we have some users who really run a massive scale. We had datas or yesterday talking about thousands of nodes. And I think they found the tools really useful. So I'm optimistic. So how about this tool then? I see one more question coming up. Great. Great talk first. And I have a question about debugging in this context. I mean, from my point of view, having a sidecar container, it provides isolation also in terms of debugging, meaning if there's an application not working, I can just isolate the problem to the pod and check the logs in the sidecar container. How does this look like in a demon-set setup? Because, again, that's what it is. I mean, I work with other applications which use the demon-set and, on average, it's harder to debug from my point of view. Yeah. So with Hubbell, you can filter your logs down to namespaces or pods or services or a variety of different layers at which you can filter your logs. You can also export, and when I say logs, they're network flow logs, so it's individual packets. It gives you a lot of visibility. And you can also export those to other tools like elastic or a SIM and use that to then query specific logs. Okay. And what about the envoy configuration itself? I mean, sometimes with this, it also goes to that. I mean, really, the bugging filters or listeners, whatever is created. Is that easily retrievable as well? I think there could be some work to be done there. For example, at the moment, if there is a problem in the CEC, you won't necessarily know about it when you create the CEC. You'll only see the sort of side effects for it. So I think there is work to be done to improve visibility into, yeah, actually, this configuration is invalid or it's not going to work. So, yeah, I think there is more tooling to build there. Thanks a lot. Last question. Hi, Liz. So the announcement about the one proxy, is it one proxy per cluster, one proxy per node? So it's one proxy per node because it runs in the Siliim agent, and we have one proxy per node. I think there may be, when we talk about this sort of security model, there may be reasons to explore doing one proxy per namespace, for example, and possibly to do with the right abstraction for configuration. I haven't seen anything yet that says this is compelling and you must do it per namespace, but we're certainly open minded to that being a possibility. OK, so my one question is, what's the guarantee that this is not a single point of failure if one worker threat of onward goes down? How's that not guaranteeing that, you know, you're not routing traffic to the rest of your services if one? I think that's a fair comment and that might be a good reason to have multiple instances. I think the argument that I've heard normally about that model is, well, you might be running custom, wasm filters or whatever in your envoy, and so they're custom, they might not be very tested or hardened, so things could go wrong. That could be a really good reason to not run that proxy across your whole node, perhaps have a use Cycas for those particular applications where you need those custom filters that you're not very confident in. There will be scope for applying a hybrid of these different models, but I think for the 90% use case where people don't need to do anything super custom, they're using pretty standard filters that are very field hardened, then we're talking about something pretty unlikely. Sure, I mean, CVEs are found in envoy orders. Absolutely, absolutely. CVEs are also found in the kernel, so we have single points of failure per machine that we have to deal with anyway. Thank you. Thank you for the questions, I appreciate it.