 Yeah, hello and welcome to our talk, making on-prem bare-matter Kubernetes ready, our network stack tech already. And yeah, so Christopher and I will guide you of who we are as a company and also what our platform is. And in the end, there will be a tech-savvy thing, which we open sourced. So we guide you through that. Next thing would be our introduction, where we are. So we are located under, it's not working. Yeah, it's okay. So yeah, I'm located in Düsseldorf, Germany, and I'm drinking a lot of coffee and yeah, I'm cycling, doing stuff, and I'm also hacking. And yeah, I'm also now, since two years, amateur snowboarder. And if you want to check me out, you can also click in the presentation. The things below. And yeah, guide you through you, Christopher. Yeah, hi. My name is Christopher Jamba. I'm located in Bonn, right next to the headquarter of Deutsche Telekom Group. I love to cook, actually. At night, I'm a software and hardware tinkerer. And in my free time, I'm actually a passionate skier. And I'm a DevOps engineer at Deutsche Telekom, the Telekom cluster or container service team. And we are building platform for cloud native tech workloads. Yeah, now we come to our organization. So you all know DT as a company. It's a little bit bigger than most of you think. So we have like T-Mobile US, which you all know, I think they bought Sprint and all of that. We are not part of that. We also not T systems. We are in the German subsidiary of the German company, which is called Telekom Deutschland GmbH. And yeah, so we are below that in a subsidiary company. And then we built a platform, this shift. So yeah, so let's go to our company and what we do. We have around 33K of cell towers and 75% of them are connected via fiber. And we want to build in the next years, like one and a half K new cell towers. And also want to cover in 2025 90% of all Germany with 5G. Then we also in the broadband and fixed. And then we have like 650 kilometers of 1000 kilometers of fiber. And we also have a lot of over over earth lines and masts, maybe for old Turquoise lines. And we want to automate our fiber planning for Germany. And also we want to cover in 2030 every household in Germany with fiber connectivity. Yeah, now back to our platform and what we do inside this organization. So what what is the shift? I don't know if every one of you know. So we are building an internal get up spaced Kubernetes cluster as a service platform almost exclusively built using open source components. And that was exactly the arms I got from one of our developers sitting in front here. Thank you for that. And that is exactly what we do. So everything is GitOps managed. And we have this stack here. So we have the use class API as some of you know. And we deploy our workload clusters also like Daimler showed in the keynote. We currently are not into the open stack provider that is why this is grayed out here. We're currently using just a bare metal part with meta three or meta cube. And we use the copy provider vSphere. So yeah, we reconcile everything through Git. So all clusters are managed in Git. So we define them in Git and reconcile them in our control plane. And also our clusters are, let's say somewhat our tag and run their own reconciliation loop. So if there is connectivity missing between management cluster and workload cluster, they still work, which was important in our tackle world. So we either run that in a call locations or in edge locations, which we currently doing some MVPs around, also in file locations, which we still don't started with. And that is the shift. And now I will hand over to Christopher. Yeah, so cloud native in the tackle world. So I think when you when you kind of start thinking about telcos cloud native is not the first thing that comes to her. I had to be honest, we tried to change that or trying to change that, especially on the network level and kind of trying to dig free telcos of like being heavy tankers or container ships with cloud native technology. So our mission is to reliably build Kubernetes clusters with well defined APIs for our customers. We also want take a great network integration for our containerized network functions that run on the platform or the cloud native network functions, however you want to call them. We say we want to work upstream first. We want to work with the community to build that kind of thing, consume upstream, consume open source, but also contribute back. But we also want to of course, at the telco view, because I mean telcos, there are a few of them. But we are kind of like for some parts, especially in the networking world, abusing Kubernetes a bit, I would say. So coming over the parts like why we did this and what we did. So this is kind of how it looks like in your, let's say, normal VMware Kubernetes spam meta cluster. You might have a firewall somewhere that firewall might connect to different networks. It allows connectivity to some of them. It does egress nothing. It does some other nothing, whatever. But you have one Vlon. Your nodes are attached to it, your Kubernetes nodes. And you're running, for example, Kaliq or Syllium atop of them, which is building a full mesh across it. But you still have that kind of firewall as a central component to manage. Of course, there are operators that do that. But it's a central component and especially in high traffic cases, firewalls are becoming quite expensive. Now, what people did is there is a RFC which is like, okay, please run BGP into enlarged data center networks. If you don't know BGP, BGP is kind of the border gateway protocol that is used to route internet traffic or to steer internet traffic over the all like world. And this is what should also be used in a data center because it is proven to work in the internet. So why shouldn't we use it there? Now, with IP fabrics, of course, you want to have somewhat isolation between your networks. So you end up maybe doing, for example, BGP EVPN, which is like a VPN kind of solution on top of it, implementing VxLan across your switches and building then to your server. You might have, for example, connectivity through something called multi-chassis link aggregation on your leaf level and or EVPN MH multi-homing, which is quite new, which I think other than cumulus doesn't really support it anywhere. But the traffic flow in IP fabric kind of looks like this. You have the border leaves which connect to the outside networks. I left out the firewalls in that pitch by purpose to make it a bit easier. You have the spines, which are like your spines in the network, of course, like the name is saying. And you have the leaves, which your servers connect to. And from a traffic flow, you have VLANs going to the server. And then from the leaves, you have it kind of re-encapsulated into VxLans. And then, yeah, the VLAN is, for example, terminated at the border leaf, and then it's leaving to different networks. But now from a perspective of kind of a layer 2, I mean, there's no firewall anymore in here. But comparing it to the picture before, you now replace the VLAN by VxLan. But you still have your Kubernetes nodes, and they still run a full mesh across them. And they still run encapsulation across them, which is a standard, I think, IP and IP. VxLan is also used heavily there. So now of encapsulation and encapsulation, which I might not want. So what we did is we made the host actually part of our BGP fabric. So it is directly peering with the leaves with BGP, but not like in a traditional sense with we just do IPv4 peering. But we actually do the full BGP EVPN peering, so the full address family EVPN. So we actually participate in the whole, oh, I learned about the networks that are there, the different networks, the different virtual networks, and so on, which makes it like from a traffic perspective like this, you actually encapsulate at the node level directly into a VxLan that you want to talk to. For example, in this case, VxLan orange, which is your cluster VxLan, you can talk to your other Kubernetes nodes, you can talk to, for example, the border leaf, and from the border leaf, it's getting into other networks. From VxLan view, this also brings one additional thing. Everything can be routed now. Nothing needs to be layer two kind of Vlan, but every node IP address, every pod IP address can be a so-called host route or EVPN type 5 route, however you want to call it. And everything is really routed in layer two, no multicast broadcast whatsoever, what you have there. However, this brings one more issue on the border leaf, actually. You have to kind of select, okay, maybe my nodes want to send to this destination, to this network, oh my, for this IP address through the other network. So you have to do some kind of configuration centrally at your border leaf. This brings in complexity, automation complexity, which can partly be software automation, but still it needs to be managed and also observed at some point. So what we proposed as a solution internally and also kind of rolled out here is we already have the nodes part of the IP fabric. Why should they participate in one VxLan, VRF, VNI only? They could be in more than one. And actually, why should we then do, actually, this kind of steering where the traffic should end up at the border leaf when we could, for example, just do it at the node. So now at the node level, you directly have connectivity to your like orange VxLan, which is now your tenant VxLan, which is just for your east-west traffic in the cluster. But also your, for example, blue VxLan here, which is then where we actually directly learn all the routes from the network one on our Kubernetes node and can say, okay, this is not a destination for traffic. It looks a bit like that. For example, if you can have connectivity to both networks, you have the three VxLans, one for network one, one for network two, and another one for the other Kubernetes cluster east-west traffic, pod to pod node to node. All of them is type five, so everything is host route, everything is a direct route, no layer two network engineers are happy. I don't want to go into too much detail of that picture, but I've just shown it because it is kind of also from our internal documentation for it from it. So we heavily use FRR on the nodes to do that. FRR has a feature complete EVPN stack. We use Linux VRFs, which I will come a bit later on Marcel, I think actually. We have underlay VRF, but we have one issue. There are containerized network functions which require SIOV. And SIOV, I can't put into a Linux VRF. I mean, I might be able to, but in the end it will be slow path. And for CNFs that have chosen SIOV, they've chosen it for a reason. They do not want to be slow path. So that's one thing where we actually right now have like a split between the network cards. This one is just going out with VLANs from the host, which is a bit sad because the other concept is a bit nicer. But we are looking into some solutions there. Also, there is right now a bit of requirements from like multi-secondary interfaces for some layer two, VNIS as we call them, or in general, think of them like VLANs, because vendors require them or want them, but we are working with them to actually eliminate them as well. Introducing network operator, which is kind of one of the tools that we built purposely for that use case to actually configure FRR on that node from a config file, from custom resources inside Kubernetes, which I will show on the next slide, templates for FRR, and all that is going into Netlink, VRFs, Bridges, VXLines, and so on that you need, and for our config. So basically it does that. You see a VRF configuration there, for example, you see import, export, yeah, policies I can define, like which stuff gets imported and exported. And it also includes some workarounds for layer two VNIS with Mac VLAN. I can talk to you about what that is in specific, but don't want to highlight it really that much. If you want to have a look at it, QR code for GitHub there, QR code for I think the shift repo in general is there, and heading over to Marcel for the Netplanner. Now, as you heard of, that was how we do the running configuration in our clusters so that our onboarding teams and customers can configure what gets exported to the networks and also what gets imported from the networks so that they can decide based on the use case what routes they need. And then we had the other issue. As we have this newly nicely VXLines setup, Netplan doesn't work because Netplan doesn't know about VXLine, doesn't know about dummy interfaces, doesn't know about any VFs. So we needed something so that we can connect our nodes to our management network. And that's where we built a new tool called Netplanner. And that one is consuming exactly also a Netplan compatible configuration and then templates out system D network D files. And yeah, it is also able to configure SIOV on the cards which support it. So what it is, it's heavily based on layer three. So currently there's no data support on it. It has normal standard Linux configuration available for SIOV as I already mentioned. And it looks like it looks very familiar to Netplan configuration. Yeah, it has currently no Wi-Fi support. I think you can build it in. And it is just very, very imperative. So you can have a single file configuration which holds all your wiring on the node, or you can also have multiple configuration files to separate your configuration a little bit. And it is mostly Netplan compatible. Yeah, if you also want to check that out, it's also on GitHub now. And as mentioned, just scan QR codes or in the presentation you can also click on the link there. Yeah, we had some KVs. So as Christopher already mentioned, VRFs are complex and also a pain in Linux. But we still needed them because we have a lot of VRFs in our network. So he implemented this nice way of having an IBGP peering inside the node to do any route leaking there. Yeah, we put most of those things into a network operator. Which we learned there and recommended. Yeah, and then one of the downsides or from an operator's perspective or a platform perspective, it's an upside. You don't have any support for multicast or broadcast traffic except this Layer 2 use case you saw on the slides before. Yeah, and this one outlier we currently have in our stack is SIOV. And yeah, for that we currently heavily look into DPUs and also started a talk with Cilium. And I think, yeah, that's all I hand over to you for the conclusion, Christopher. So what we've shown is and also worked on is Kubernetes OS can be directly participating in our IP, EVP and fabric. For us, it adds flexibility because we don't need to heavily automate any route leaking configuration on border leaves. In addition to that, it kind of saves costs at that point of view because if you do heavy route leaking, you're programming them multiple times into your hardware chip, which at some scale becomes really expensive. Linux VRFs can be a pain, especially with local endpoints. Need to talk to someone who's more familiar with that one, maybe. But they can be used with caution. We've done that through like a Veth interface and IBGP peering and no route leak on the house, sadly. SIOV is an open question, but there we might actually see vendors moving into different directions, maybe, like use AFXDP with EBPF, which might fit natively into Cilium, for example. Or maybe we have a VPP stack on the host running, for example, with Calico VPP, and then a member between the network function and the host. That might be a solution to it as well, but it should be an industry discussion into what we should really move to. Netplanner or Netplan has no support for VXLAN, VRFs and Veths, so that is what we implemented, or mainly UMassail, implemented in Netplanner. Network operator serves as the glue between Kubernetes and FRR, so customers or we as the platform team can configure FRR, like this route leaking prefix list stuff from inside Kubernetes. And we're also looking a bit into DPUs if SIOV stays around to offload that kind of stuff down to a network card, like all this abstracted away from the cluster again, so the cluster just sees one default route. That's it for our talk today, and I will take a selfie with all you in a couple of seconds, because I can't still imagine that all of you just came to a network talk on Friday afternoon. If you have questions, either I'm not sure if you will get a microphone or just come to the front, hang around with us. And as we had the last talk, and I'm not sure if anyone is saying goodbye to you, thanks for seeing you all in Valencia. Maybe see us at the next KubeCon. Thank you.