 Thank you very much for coming. So my name is Dan Finneran, I work at iSurveillance part of the Selim C&I. I'm a maintainer of some load balancers, projects and bare mail provisioning projects and I've been quite fortunate recently to get involved in some of the telco projects working with them, helping them kind of get some of their applications running and this is a session really about some of the problems that I face, some of the I guess innovative solutions that we've had to kind of come up with in order to actually get traffic in and out of Kubernetes clusters in a way that makes sense. I was fortunate to do an ECHO session with book recently where he kind of went through what telco networks actually look like in the real world and it's quite mind boggling really. So what are the problems with Kubernetes networking kind of out of the box? Well, for most people, if you've never really deployed it yourself, out of the box, Kubernetes doesn't actually come with any networking capabilities. Networking is a plugin that you need to apply on top of, largely born out of the cloud infrastructure. So that existing plumbing is usually already expected to exist. You know, kind of the cloud's already provided a lot of the networking capability. When you're trying to do it yourself, there's a lot of things that are going to be missing. There's a lot of predetermined behavior in terms of what traffic actually looks like coming into and out of a Kubernetes cluster, which again makes sense for very modern applications or brand new applications that you're writing, but porting existing applications or expecting to have certain behaviors again just simply isn't possible. And it's all recently a lot of the networking plugins largely just focused on internal traffic. So everything within the cluster that typically made connecting and interacting with existing network and physical infrastructure a bit of an additional task. Why is this actually becoming a problem? Well, as I mentioned, we're moving past, you know, your first application that is kind of simple like a web server and this and the other. People wanted to move existing applications that have different requirements. People are moving, as I mentioned, those critical applications on there, and they wanted to interact now with existing and physical infrastructure. So whilst we've extended the capabilities of Kubernetes cluster, we're now having to create new and innovative solutions in order to alter those default behaviors. So I'm going to kind of step through kind of couple of the quick scenarios and the sorts of things that we're having to do in order to get traffic in and out of Kubernetes clusters. So here we have a simple node. We have our pod that's actually running and we have the address of the node itself. So basically the most simplest way you can get traffic into your application is through a node port. And that is typically exposing your single application on the node itself. Very simple, but that is very inflexible. You basically have almost a once one mapping with a physical server, not very cloud native. So the alternative really there is to implement a load balance solution in the real world, you know, in physical environments, that typically means adding additional IP addresses, letting the infrastructure know that that new IP address actually works and then getting the traffic to that new IP address. We can do that pretty simply with kind of layer two and adding addresses and you can actually do it kind of manually through manual steps if you want to go down that route. But when we start talking about scaling and operating in a real world environment, we need to start thinking about a lot more physical infrastructure. So all of a sudden we need to integrate with that physical infrastructure and let it know that these changes have actually occurred. So we typically need to implement things like BGP, we can use projects like the go BGP project. So when we create these new addresses, we can then plummet the capabilities to speak to that physical infrastructure and tell it to get to this pod to this new IP address, routes to me and traffic will eventually then make its way through the kernel logic to the pod that's actually running. So you can see we're already now having to implement into Kubernetes the capabilities to start speaking to physical infrastructure. We're starting to bring all of that into the Kubernetes stack as it were. So that's getting traffic into a Kubernetes cluster. We have to inform the network one way or another that this is how you get traffic to me so that I can get traffic to the pod itself. Getting traffic out of a Kubernetes pod or a Kubernetes application is even more confusing. So we have a pod that wants to speak to an external database. What is the source address? So what is the database going to see when the pod tries to connect to it? Does anybody know? Yes, exactly. So as the gentleman said, they're going to see the address of the node where the pod is actually running. This is so that traffic can actually go back and forth between the two. Traffic can come out of the pod with its pod IP address, but traffic can't go back because that address is internal to the Kubernetes cluster. So hence why we need to do mapping to the node address. That's fine until we start to kind of protect our database. We may have a firewall in front of it. That's fine if we just had one worker and that pod was always going to be on that one worker. But in a realistic term, you're going to have multiple machines with pods running all over the place and it doesn't take long before you end up with whack-a-mole in your firewall because your pods moving around and then the source address is always changing to one of those new nodes. If you're continuously killing and implementing new nodes as well, at the end, your firewall is largely going to become redundant because you basically have to put everything in there at some point or another because you're going to need it to access that. So how do we fix this problem to ensure that we have stable addressing so that the program, the endpoint that we're trying to access always has stable addressing? Well, to do that, we need to do some level of stable egress and there's a couple of ways that we can implement that. One of which is to use the load balancer IP address. So create a load balancer. We get that one single IP address and that IP address then will move around wherever that pod sits and it allows us to have a stable address. So in this example, we don't need to put in our new load balancer address into the firewall. Should this node go away or that pod moves around, we will re-plumb things in with the kernel logic so that the sourcing always comes from that load balancer IP address. The alternative to this is to use a gateway pod and what this will do, almost kind of the same, this gateway pod will always attach to the same IP address and traffic from that pod will be routed through that gateway so that regardless of where the pod is actually running, traffic goes through that gateway and it always appears as though it's coming from the same address. So these are kind of the two main problems that we're starting to see in kind of the telco world at the moment is getting traffic to those telco applications and having stable traffic coming out of those applications. What is powering all of this at the moment? So the kernel logic at the moment, net filter is typically the way of doing things typically at the moment, that's actually changing. So IP tables is still largely powering most things inside a Kubernetes cluster. A lot of that is moving to NF tables. In fact, some of the distros are forcing that upon us in a pretty broken state at the moment, but that's out of my hands. But we're moving towards EBPF. A lot of the projects have adopted that. So projects like Sillium are doing a lot of the egress through EBPF. Some of the other projects that I'm being part of, we're going to start adopting EBPF to move things forward as well. So as we move to more modern kernels, we'll be able to have much more stable behaviors inside our clusters. So in conclusion, the flexibility of Kubernetes and the underlying operating systems is allowing us to move past these default behaviors which have made sense in the cloud, but are becoming a hindrance and a bit of burden in telco and on-prem environments. Depending on some design decisions, some of the hacks that we've had to implement almost fight against these existing components. For instance, egress, actually some of the things we've had to do with egress fight against the default behaviors of CNIs. And kind of a final thought to a certain degree, we're having to put infrastructure logic now into different areas of the Kubernetes stack. Should it go into Kubernetes itself or should it live in different kind of areas? So with that, that's kind of a quick overview of some of the issues that we're facing in terms of getting traffic in and out, some of the ways that we're kind of getting past those problems. Thank you very much for your time today and hope you have a good conference. Thank you.