 Hi, everyone. So today, we're here to talk about navigating network topologies in CFCR. So before we get started, we just want to give a little bit of a prequel of why we're even here or how we came up with this talk. So basically, we work with Kubernetes and CFCR a lot. And the thing that we wanted to dig into and understand for ourselves was really what are the layers of networking that exists in Kubernetes and also in Bosch so that we could understand it better for ourselves. So for those that are experts in Kubernetes, hopefully this will give you, like, you might get a little bit of information out of this. But for those that are brand new to it or coming from the Cloud Foundry world, then hopefully you'll find this useful, at least for being able to help debug stuff that you all are working with Kubernetes, maybe. So let's get started. So I'm Neil. I'm a software engineer working at Pivotal. I work on the Pivotal Container Service Team in Dublin. Hi, everyone. I'm Ravashi. I am also an engineer at Pivotal, and I'm based out of the San Francisco office working on the Cloud Foundry Container Runtime Team. Cool. So today, we're going to do a quick recap of what exactly Cloud Foundry's container runtime or CFCR is. If we say CFCR throughout the talk, that's what we're talking about. And then once we've gotten that little bit of background, we're going to go into the main purpose of the talk, which is basically what are the layers of networking that are made up in CFCR, both in terms of Bosch and also Kubernetes. And we'll go a bit more of a deep dive into those. And then we'll kind of finish up at what exactly are the things that are being worked on at the moment in both the Cloud Foundry and also the Kubernetes communities in terms of networking. So let's kick us off. We're going to talk a bit about what exactly CFCR is. Cool. So Cloud Foundry Container Runtime, what is it? It's a Bosch-deployed Kubernetes cluster. So what our team works on is packaging up a lot of the Kubernetes dependencies and things that, basically, you need to be able to create a reproducible cluster. We provide it into a Bosch release so any Bosch operator can go onto the platform, deploy it, and get a case cluster. We try to strive to make it configurable through your standard mechanisms for a Bosch release. So via job spec properties, you can configure system components. You can. If you have a lot of Kubernetes knowledge and want to configure your API server in a particular way, we kind of expose a lot of those properties to allow you to do so. I think the biggest thing that we're kind of paying attention to is continuing to keep the Kubernetes experience, whether you're on Bosch or off of it, a multi-cloud experience. Because most people don't want to be locked down to a particular IaaS. Bosch, as we know, is multi-cloud, so is Kubernetes. So it just makes sense for CFCR to also be in that fashion. And the other thing that we kind of, more or less, are adhering to is keeping in line with GKE and being compatible with what features it's actually deploying instead of the clusters that you can get. So GKE is that Google Kubernetes Engine. It has a hosted and also on-prem solution. So what we do is actually take a look at how they're configuring their features whenever a minor release of Kubernetes is available. And we kind of use that as a bar or metric to make sure that we're meeting the needs of the general audience that's looking for a Kubernetes cluster wherever they're deploying it, but especially in Bosch. So that was just a little bit about CFCR. But now let's get into kind of like the meat of this talk. We wanted to just do a kind of overview of the three networking topologies that we're going to be talking about in this talk. And that first one is actually Bosch. So in the past year or so, they came out with a internal DLS solution that actually helps you to resolve different VMs inside of your deployment. So we're going to talk a little bit about how we leverage that inside of CFCR for particular features. We're also going to talk about overlay networks and just to kind of give you an overview of what that is and level set on an overlay network. It's very simply a network on top of another network. What it really strives to achieve with routing traffic between different IPs and things like that is by doing this method of encapsulation on data packets. So we're going to talk a little bit specifically about how this is achieved via Flannel, a little bit more in depth. And just some common examples of when you're thinking of standard overlay networks, just to kind of seep in through examples is something like a VPN. So if you're in your office and you have your physical network setup and someone in your IT department has already set up the network that's going to be allowing you to have access to your hosts in your office and things like that, that would be through a VPN setup. So that's just another network that's on top of another network. And same thing with CDN, so content delivery networks in just the internet. So you have internet being your networking layer. And then the CDN on top of it is your ability to hit CDN so that you can cache websites and access them a lot faster. Another example of a network on top of a network. And lastly, we're going to talk about Kubernetes services. So services are a grouping of pods with an access policy. It's kind of like an internal load balancing inside of your Kubernetes clusters that you can set up for your pods. So we're going to actually take a deep dive into that and look at how Kube Proxy helps us to achieve Kubernetes services in a cluster. All right, let's start with the first layer. That was Bosch. So hopefully you're all familiar with Bosch. It has been like the cornerstone of Cloud Foundry for a number of years and is kind of the principle of why the CFCR project started, right? So we're providing Kubernetes on top of Bosch. But what exactly does that mean in terms of networking? Like, is there additional networking on top? And basically, we're just going to cover a little bit about what that is and what's added on top. So let's spin us up a CFCR cluster. So for those familiar with Kubernetes, you'll recognize this instantly. We have three masters and three workers. And they've grabbed IP addresses from the Bosch network range. So you can see here the masters are from 1 through to 3 and the workers are from 4 through to 6. So let's look a little closer at the masters, for example. So I posed this question, right? What happens when one of these virtual machines goes away? It dies, right? One of my masters just disappears for whatever reason. What do we do? Panic. No. So this is what you would typically think, right? Something has gone away. What do I do? So we all know that Bosch, and we know and love Bosch, and we know why it's there. It brings things back when it recognizes that things are gone, right? So Bosch has recognized that the VM is gone. Nothing special here. And it brings it back up. But if you notice from this one, we're using IP addresses. And it's come back up on this .7, right? So for the student among you may have noticed that what this means is that nothing can talk to that master anymore. Because they're expecting it to be on the .1 IP address, and it's come back up on .7. So this master is now basically come back up. Bosch brought it back, but it's kind of useless because it can't rejoin. So operators are not very happy, right? Wasted resources. This machine is just sitting there. It's kind of basically useless because it can't rejoin. So this is, as Ravashi was saying earlier, something that came in about a year ago to the Bosch project was Bosch DNS. And very simply put, it just allows you to be able to assign DNS names to your virtual machines instead of using IP addresses directly. So Bosch is DNS to the rescue. And if you actually look up this, there's actually a number of white papers with this name. So if you're interested, go check those out. DNS solves all the problems. So here we go to, for example, just create three names, m1.internal, m2, and m3. And now anytime we want to reference those masters, we just use the names instead. And basically, we don't need to worry about IP addresses. And our master will be able to rejoin the cluster as soon as Bosch brings it back up for us. So hopefully, there's nothing too new or special about that. And hopefully, you recognize, and maybe that's all really simple and obvious to you. So we're going to kick off the kind of Kubernetes side of things and talk about the overlay network. Great. So one thing I kind of want to mention about overlay network, not specific to CFCR, but just in Kubernetes clusters in general, is that it's actually not required for you to set up an overlay network in any of your Kubernetes clusters. But it does ease your pod-to-pod communication. That's kind of the main value that it's adding. And anything outside of that is usually some extra fancy stuff that the software-defined networking layer is kind of giving you. So if you weren't going to use an overlay network, what you would probably end up doing is using a flat routing network. This is actually the operator or the person who's installing that Kubernetes clusters would have maybe much more, I would say, at their IaaS layer, we're saying these nodes that contain these containers or these IPs actually have mappings over to other containers and nodes, sorry, other containers inside of nodes and can talk to one another. So it requires the operator to do a little bit more configuration. Whereas if you had something like Flannel and an overlay network, that would happen for you, and you wouldn't have to do that mapping. And depending on the IaaS that you're using, you could actually hit some limitations on the amount of mappings that you can actually configure between those two things. So what we use in CFCR is the simple, easy-to-use setup that's recommended for Kubernetes clusters. It's more or less almost considered sometimes even the default CNI plugin that you could have inside of your cluster. It's called Flannel. It's run by CoreOS. Like I said, it's a CNI. It's not just specifically for the Kubernetes use case. You could actually use it for various other things that you're deploying. But for us in CFCR, it's kind of the easiest way to get going. If you want some of your fancier things that you want to do at your networking layer, such as creating network policies, Flannel is not going to really be able to provide that much in that space, but it's a great way to get started. So we're going to talk about Flannel a little bit more in CFCR. To show you, in a CFCR cluster where you have a master and three workers that are running, these are workers that are running the KubeLit process inside of a Kubernetes cluster, where is Flannel? So the biggest thing that we should call out actually first is that Flannel heavily utilizes at CDs to understand those mappings where containers are and what IPs they're addressed at. They also store that information in at CD. And as a result, we actually have Flannel running on every host inside of the cluster in a CFCR deployment to be able to kind of understand where those mappings are and easily talk to at CD. So focusing in on just that kind of one node, so one worker that's inside of the cluster in Kubernetes, we're going to take a look at how pod-to-pod communication is achieved. So explaining this diagram just a little bit is that I have P1 over here on the left, and it's to denote a pod that's existing inside of this worker. Pods are a group of containers that are in the same network namespace, effectively, so you could have one pod that's actually comprises of multiple containers. But that pod itself is assigned a unique IP. And it's actually Kubernetes delegating over to Docker in CFCR to create a virtual bridge that has a CIDR range that will say this container will be created inside of that CIDR range and a virtual ethernet device that actually maps to F0 so that it is possible for containers that are inside of that node via Docker to be able to communicate with one another. So that's kind of just like standard, not really specific to Kubernetes per se, but Docker in particular, about how containers are able to talk to each other when they're all within that same node and Docker, Docker's virtual bridge is kind of handling the communication between those two containers. But if you're trying to do pod-to-pod communication across nodes in the cluster, so how does FNL help you with that? So I have the same image, but I have now have two nodes, two workers inside of that cluster. So P1 on the left and P2 on the right. Those pods have been allocated their own unique IPs by the Docker processes that are running inside of that node. So you know it falls into the Docker zero bridge range. And so you can imagine that we have data that's coming out of P1 and it wants to talk to P2. So how is FNL achieving this? What happens is that in CFCR and in most cases of Kubernetes, we have a CNI bridge that gets configured by FNL. We call it in CFCR CNI zero, and that's gonna be for FNL basically to set up some rules and basically dictate that data that's flowing inside of this wider rule, this range, is actually gonna go through the FNL bridge instead of the typical Docker bridge. So that's kind of communicating that this traffic is actually going outside of this node, not necessarily within the node where Docker is running. And what FNL actually does is attach a UDP header on top of that data packet, which now dictates its new source and destination IPs. The way that FNL knows how to do that is once again by actually talking to SED, because that's where it stores information about, oh hey, I know this pod is registered at this IP and lives at this node's IP. So it attaches that UDP header, which then allows it to actually communicate over to the other node and fall into its respective CNI bridge, in this case, FNL. And it actually chops off that UDP header and says, okay, now route traffic normally and you know that this route traffic to this IP normally and find 10.200, 2.1. So that's how it makes it over from P1 to P2. So that was kind of like pods and containers and how communication is offered between those two, but now let's look at how Kubernetes does services and kind of does that internal load balancing thing that we were talking about earlier. So in the services network, let's go back to pods again, right? So pods are ephemeral in the Kubernetes world. They can go away, they can disappear at any time and their IP addresses can change at any given instant in time. So just like Bosch, we can't rely on IP addresses at all. So how do we do this then, right? So if the IP addresses change all the time, what do we do? And this is where the services network comes in. So the first thing to know about it is that it doesn't really exist. It's totally virtual, which means that whatever IP addresses that this service network is gonna give to you, it doesn't map to anything physically in the infrastructure or any device. There are IP addresses that are stable and they will give you the IP address of the pod eventually, but they don't actually map to anything physically in the infrastructure. So how does it do this, right? So that's what this part of the talk's all about. So the services network provides a couple of things for us in the Kubernetes world. First of all, it provides us the internal routing, essentially, how I get from my pod A to my pod B, the squiggly line up here on the diagram. So I was talking about like, inter-node communication, but we're just talking about pods that all live on the same node, on the same machine. How do they talk to each other if they can't use IP addresses? So it provides us, first of all, with the internal, but it also provides us what a way to access those pods from somewhere outside the cluster. So it allows us to go, for example, from the public internet if we so expose it to hitting one of these pods that's running in our cluster. And we don't really need to care about IP addresses or anything, it just, it does it all for us. In the Kubernetes world, there are a couple of constructs for this, the main one that most people use on the public ISs is just type load balancer, and what that will do is Kubernetes will just go, provision you a load balancer in your IS of choice, and then you can just use that and you'll be able to access your workloads from the external public network. It also provides you with other ways that you can do it, so you don't do it public-facing, but that is the main way. So let's have an example, right? Let's see if something more concrete of what our service actually is. So let's kick off with a deployment. So we're gonna create this deployment. It has three databases, so three pods running. These are the databases storing our persistent information that we want for our application that we're just gonna create. And let's create basically how we access it, so how we can access these pods. So let's create our service. And what this will do is, Kubernetes will give us something known as a cluster IP, and I was saying earlier that we need some kind of stable way of being able to access these workloads, and this is what Kubernetes provides for us out of the box. So it'll give us an IP address, and when we hit that IP address, it will wrap the traffic from there to one of these pods that's running somewhere on our machine. So let's spin up a front end for this application that we're creating, and it needs to be able to talk to the database for whatever reason, be able to get some data, just like any other application. So what does it do, right? It doesn't, not gonna hit the pods IPs directly, because that doesn't really make sense. We say they're ephemeral. So it's gonna hit this cluster IP instead. So it's gonna send this request to the cluster IP, and this is basically where the load balancing of Kubernetes comes into play, and it will basically wrap the traffic from this IP address to one of these pods. In Kubernetes, in one of the particular modes, this is random distribution of choice, so you will, once you hit that IP, your packet will go to one of these three nodes, randomly, basically, 33% chance of hitting one of these. So how does it do it? This IP doesn't exist. How does it actually do those then? So there's something running on each of the workers in Kubernetes known as KubeRoxy. You might have possibly seen this and be like, hey, what is that thing doing? So hopefully you'll have some idea after me explaining what's going on. So it's responsible basically for this implementation of how I go from this IP address that's virtual, doesn't really exist to one that actually does exist, one of these pod IPs. And how it does that is when you have your service created, it installs something called IP table rules onto the machines, and basically those rules are the things that determine where the packet goes from this IP address that doesn't exist to one that actually does really exist. So we're gonna dig a bit into exactly how this works. But before we do, there are three modes that you can configure your KubeRoxy in, in Kubernetes today. These are kind of chronologically from left to right in terms of when they came out. So in the early days of Kubernetes, user space was the main and only routing mode available for your KubeRoxy. Along came IP tables and just went GA is IPvS. So in CFCR, we are using IP tables at the moment. So let's deep dive a little bit and see like, how this kind of virtual IP going to real IP works. So before we get into it though, we need to take one little detour and just kind of define a couple of things in this world. Because Kubernetes currently lives in this world really, it's not very Windows friendly at the moment. So let's talk about Linux. So there are two main concepts. They're pretty simple that we just need to understand before we go further. User space and kernel space. User space is basically where all your normal processes run. They kind of can only access a restricted amount of resources, they can only do certain things and they have limited amount of functionality. And then we have kernel space, which basically has full access to everything in the hardware and also the system resources. So it has all the power. So those that may have noticed, the first row, first routing mode was called user space and that actually ran in user space. Confusingly, they're the same name. But IP tables runs in kernel space. So how does it do this? So in Linux, there's a module called NetFilter. And NetFilter basically provides this way of being able to go how to like, when a packet comes in, it does like an internal way of routing from one thing to another. So it does this translation of virtual IP to real IP. So as I said, it runs in kernel space, so it has all the power, so it's pretty fast. It's one of the reasons why they moved away from user space. And what it does is basically it just matches packets that come in to the machine. If they match a certain amount of rules, then basically it will redirect the packet to wherever it needs to go to. So if we see here on the right, this diagram basically kind of sums it up. So the first thing when my packet comes in to my networking interface, it goes through a filtering process. And then a decision has to happen. And that decision happens in another module in Linux, of course, which is known as contract. And contract is basically a way of being able to connect or to have connection tracking in Linux. And basically, if something has been seen coming from an IP address going to another one already, it just skips over everything. It takes the true path and just gives it back the IP address that it already knows about to rat it to. Otherwise, it does this mapping of virtual IP to real IP. And then the next time that packet comes in, it will just skip over that and just go through the true statement and just continue on. So let's take a look back to our example of our app with our databases. And what does the contract entry look like for this? So here's the contract entry on the first time the packet comes into us. So you can see that the source IP is coming from our application on the right and its destination is for the cluster IP. And what happens after it has gone through that process on the previous slide is it actually translates and this is what you really get. So your app one is the source and your destination is actually to one of these databases. As I said, random distribution. So I just chose one. So that was an overview kind of the services layer. There's lots more in it. But basically we're just gonna round out this talk by giving an update on what's going on in the networking community of Kubernetes today. Yeah, so if you've been in the Kubernetes community before, chances are that you've come across these things called SIGs, they're special interest groups and networking is no different. It has its own SIG and this is a group that meets pretty regularly to talk about networking initiatives inside of Kubernetes and how they can grow that and mature that. So one of the things that we kind of noticed in the past year that was one of their bigger changes was actually replace their internal DNS, CUBE DNS with core DNS inside of Kubernetes clusters. So that's kind of like a configurable option in CFCR right now whether you wanna use core DNS or stick with CUBE DNS. But in newer versions of Kubernetes and actually I think the next coming version they're gonna actually default it to core DNS. And it just is, I think it's just basically a rewrite of CUBE DNS and offers a lot better performance. And another cool thing that they're working on is external DNS. So as a Kubernetes user, if you wanna set up external DNS for anything or any types of services in your cluster, you can use a new custom resource called external DNS and that's currently in alpha. They're working with different providers to mature that experience. But if you're a cluster user and say you wanna use like AWS for 53, you could use this resource to configure those DNS entries which is pretty useful. They've also been working on in the next upcoming releases multiple pods having multiple IP addresses. So that's a feature that a lot of people have been asking for especially when it comes to IPv6 where it could be useful to have multiple IP addresses registered to a pod. And another thing we wanted to highlight was that support for our IBVS and CUBE Proxy actually went GA in the last release of 1.11. So we had talked about a little bit how that was kind of the most recent stage for CUBE Proxy to use and it's kind of the fancier new thing. As for kind of Bosch DNS and where it's at right now, we currently leverage it to do DNS resolution for Bosch instance groups. But actually in flight for the Bosch team is that they're working on doing DNS resolution for jobs as well. So this gives us a better more fine gain control over putting in more DNS entries and we definitely wanna use that inside of CFCR. As for our release, I kind of mentioned that Flannel is our default CNI that we have inside of the release. We in the future actually want to provide the ability to have more of these CNIs be pluggable inside of that release and give people a lot more flexibility, especially if they wanna do some fancy stuff like network policies and things like that. We've definitely seen a big trend in networking interests inside of the Kubernetes experience. So it would make sense for I think CFCR to go down that path when we are ready to introduce that functionality. So in summary, I hope you took away kind of the three networking topologies and got to learn more about how Bosch works, how the overlay network works in CFCR as well as Kubernetes services, a little bit of a deep dive in each of these. And maybe you came away with the idea that maybe it's a little bit complex at first, but maybe just giving you a high level overview that you'll see that it's a lot of technologies coming together, making a really great experience and it actually does work out pretty well. So thank you so much for coming to our talk. If you wanna talk to us about networking or really anything CFCR related, we have the CFCR channel in the Cloud Foundry open source Slack. You could shoot us an email. We definitely wanna hear about different networking use cases that you're using, any different CNIs that you're particularly itching to use in CFCR. And if for any reason you wanna talk to Neil and I, here's our Slack handle in the Cloud Foundry community space as well. But yeah, thank you for coming to our talk. If you have any questions, we'll be up here. Feel free to come up to us. I think we're just a little bit short on time, so we'll probably do questions like in person. Thank you.