 Hey everyone, I'm Surya, and I'm an engineer working in the OpenShift networking team at Red Hat. And this is Patrick, my colleague, he also works in the same team. And we are both together here today to talk about OVN Kubernetes, which is the default CNI in OpenShift from the 4.12 release. This is the brief agenda that we'll be covering today. We will first start with the basics of what is Kubernetes in OpenShift networking. Can we have a show of hands if you have used Kubernetes OpenShift before? Yeah, that makes my job easier then. So once we finish the basics of that, we'll go over to saying what is OVN and what is OVN Kubernetes. And then why did we move to OVN Kubernetes? And I realize I have not explained what the acronym OVN stands for, right? So we will also look at that in a minute. And what are the differences of OVN Kubernetes from its predecessor, the legacy CNI plugin, STN, software-defined networking. We will also look at what that plugin is. And finally, we'll talk about how this works, right? How does OVN work? What are all the components under its hood? We'll try to show a live demo of how everything works together and how do you get networking on your cluster. So Kubernetes networking. So the fundamentals are, you know, every pod must have a unique IP. So you must allocate an IP to a pod that's unique across the entire cluster. The pod must be able to talk to other pods in the cluster, to other nodes in the cluster, and even to external entities outside the cluster, right? So basically, Kubernetes networking model defines that you need to have networking set up as soon as the pod comes up. And this is like the basic. However, we all know that Kubernetes is just a bunch of APIs, right? So it does not really provide a networking solution by default. And that is where we need portable plugins to come on top that can work with Kubernetes cluster. And such plugins that provide networking to a Kubernetes cluster are known as CNIs, container network interface. And this is an upstream community that defines a set of standards and how a CNI is supposed to be written. And as per those CNI specification standards, OVN Kubernetes has been written. So it is one such CNI plugin that provides networking in a Kubernetes cluster. There are plenty of others out there, like Calico, Sillium, and Treya that you might have heard of, but OVNK is just a counterpart or another yet CNI. OpenShift Networking. So like I mentioned, OVN Kubernetes is the default CNI in OpenShift. So OpenShift is Kubernetes plus a bunch of operators. And even for networking, we use an operator. It's called the cluster network operator. And that is in charge of deploying all the demon sets and the deployments that are required for the network infrastructure to come up. And CNI supports two plugins. Like I mentioned, the legacy one, which is OpenShift SDN, and the new one, the shiny one, OVN Kubernetes. And that is the default from the 4.12 release. And between the CNO and the specific CNI plugins, we have a jam, which is Maltis, right? And Maltis is a meta CNI, we say. What it does is it allows an inception, so it invokes other CNI plugins. So it is a CNI that invokes other CNIs. So let's say, for example, in a pod, you want to set up multiple interfaces on it. So you can invoke the OVN Kubernetes or the SDN CNI to configure the primary interface for a pod. And then you can have a secondary interface using SRIOV, right? Maybe you want to have a secured connection to a storage, which is isolated. So you can use another plugin, and Maltis lets you call multiple plugins to get networking set up for a pod. So these are some of the components in OpenShift that are involved in networking. And we thought this slide is pretty cool because it shows all the CNIs that are supported officially by OpenShift. But the first two are the plugins that Patrick and I, our team manages. The rest of them are third party supported vendor CNI plugins. So coming to OVN Kubernetes, right? So what is OVN Kubernetes? Let's start with defining the acronym, OVN. But before I move to OVN, I realize I should tell you what's SDN, right? Because I told you that we're moving from SDN to OVN in OpenShift, and that is our story. We are here to see why that happened. So let's take a look at SDN first. And SDN stands for Software-Defined Networking. It's a legacy network provider in OpenShift. It is also written as per the CNI standards, and it uses two main networking technologies. One of them is IP tables, and the other one is OVS. OVS is Open Virtual Switch. It's just a virtualization technology which creates a multi-layer distributed switch. So you get a virtual switch on each node in your cluster. That's how you can put it simply. And using OVS, SDN provides networking on a part, right? So it hooks up and it creates the necessary entities, the floors, everything you need to get the part up and running in the networking arena. But we also have services in Kubernetes, right? So we have cluster IPs, node ports, load balancers, in Kubernetes clusters. How are they implemented in SDN? That's where IP tables come into play. So SDN uses IP tables, a lot of nested IP tables for setting up services and OVS to set up the pod networking. And using these two technologies, it's able to manage the entire networking lifecycle of a pod in a cluster. Another interesting point is when you have pods across the nodes, you need them to be able to communicate to each other. And pod-to-pod communication on a node, often known on a cluster, often known as overlay networking, is achieved in SDN by using BXLAN, the virtual extensible local area network protocol. And this supports east-west traffic. And now that we've seen what SDN is, if you now move towards OVN-K, which is the star of our show, it's an open source project which provides robust network solution to your clusters, in Kubernetes clusters. It has OVN, open virtual networking stack, on top of OVS. So that's the extra piece here, which is really what you should take away from here. So SDN is also using OVS, like I said. But what does OVN-K do extra? It brings this new piece called OVN on top, which is the abstraction layer on top of OVS. And that lets you create simple logical network constructs. So it maps things like nodes to routers and switches. So that makes it easy for administrators and infrastructure developers and even users to be able to see those virtual entities that are very similar to your traditional networking stack. Like, this is what you see day in, day out as network administrators. So that layer of OVN abstraction gives you this huge advantage, whether you're a developer, whether you're an operator. So that's the key and one of the main reasons why we moved, in fact. And another subtle difference is that we use Genif as a protocol, because it also offers some other subtle advantages to VXLAN. Like, in its header, you're able to store more information, for example. So if you were to compare these two technologies on a high level, like the highlights of them, is both of them use OVS in their base. Both of them have an internet protocol address manager controller. Well, don't get too worried. It's just an IPAM controller that allocates an IP to each pod in your cluster. So it's centralized pod IP allocation. Both of them use OVS flows at the end of the day to create their network policies or any other things that you want to implement. They use either the VXLAN protocol or the Gini protocol to achieve communication across nodes. But that's just a subtle detail. STN uses IP table rules, like I mentioned, for implementing services. OVNK is using OVN abstraction and constructs. And I keep using the word OVN construct. Over and over, we will see what that is in a moment. And NATting, network address translation, this is required for every Kubernetes cluster so that the pods can talk to outside the cluster, so to internet. And this is achieved, again, using OVN constructs in OVNK versus IP tables. That's the SNAT and SDN. Now that we've seen the walk, let's look at the why. Why did we move to OVN Kubernetes? OVN is the new abstraction layer. I think we already mentioned this. This allows for easier development. So as developers, if you want to add new features on top, we have a whole new engine that can let you create such abstraction constructs on your network topology. And that can help you easily add on new features versus in SDN, you had to directly touch open flows. And this is complicated. And you had to have more understanding of another technology, and that made things a bit convoluted. The other thing is the flexible and expandable architecture that OVNK provides using OVN. So you're just connecting nodes across the cluster using routers and switches. We will show this in detail in a demo. Versus in SDN, you were using IP tables for services and OVS flows, but it was too restrictive. You could not do many of the things that you would want to do with telco use cases. The final thing, open source. So SDN was something specifically created for OpenShift. And we wanted to take our project in open source direction. So OVNK has contributors who are not just from OpenShift or not just from Red Hat. We have a vibrant upstream community. So we wanted to intentionally move in that direction. So these are the three wins that I would say that we get out of OVNK. And finally, coming to the how part of our story. So how does all of this actually work under the hood, right? So with any component in Kubernetes cluster, even the networking component has some parts running on the control plane nodes in your cluster, and other parts that are running on the data plane. So if we look at the control plane parts, the parts that are in Violet or Popo here, the OVNCubeMaster, that's the OVNCubinatus component, right? And the pink ones, they are just details you can even ignore them, but they are the core OVN pieces. So they are the bits that provide that abstraction that I mentioned. So what OVNCubinatus mastered us is mainly two things. One, it gives out IPs to pods, right? So it allocates IPs across all the pods in your cluster. The second thing is it's continuously watching for all the Kubernetes API objects that are created. It translates each of them into OVN logical entities, and it stores these entities in a database. So this is happening across in a centralized manner in the three control planes. And once these entities are created, let's say OVN and OVS are doing some magic, right? So they are creating, they're taking all these entities and doing the necessary plumbing and creating the flows that are required for networking to function on a cluster. And we talked about OVN constructs. So this is where they come in, like the logical entities. So if you take a node in a Kubernetes cluster as an example, and when you create a node, a corresponding thing that OVNCubinatus master does is it translates this node into a specific switch, a router, some policies. So it does that abstraction for you where it converts a node as, okay, I need to plumb a switch for this node. I need to now create a router for this node, right? So, and then imagine multiple nodes in the cluster. So you're gonna have multiple routers and all of these are connected by switches. So you get this whole network topology, logical apps traction at the end of the day. And this is very similar to how you see in traditional on-prem networking. And another thing, like let's say namespace, right? It's a very common entity that we create. And a namespace is just a bunch of pods that are tracked over there or services, right? So if you take pods and a namespace as an example, we might want to track all the IPs of those pods. So we create a logical entity called an address set, which is a collection of IPs, right? And that's used to provide a specific functionality for features. If you take a pod as an example, a pod is translated into a logical switch port. So it's a port on a switch. So you have a switch creator for each node. All the pods on that node will be a port that are hooked to the same switch. And the switches amongst different nodes will talk through a router, right? So it's basic networking here. And the same for services and endpoints. So each of them are translated into entities called load balancers. And Patrick will talk about them in detail. So those are the things happening on the control plane. And once the control plane aspects are dealt with, on the data plane that is running on each worker node in the cluster, we also have a lot of things actually going on. So like I mentioned, the master has now allocated an IP to the pod in the cluster. Once the IP is allocated, the CNI aspect, the actual binary that is doing the CNI add event is located on the node. So the OVNCube node pod is continuously also watching for these IP annotations that the master does. And once master's finished its job, the node takes over. And the node is literally creating the virtual ethernet pair that's required for the pod. It's actually setting up the plumbing inside the OVS that is required for the pod to have its networking up and so on. So the CNI executable commands, like we mentioned, which is done as per the CNI specifications, is all executed by the node component. And in the node also, we have these pink magic containers that are translating these entities which we created in the database in your control plane into flows, traffic flows, let's say, for simplicity. And now that I've covered all the fundamentals and the theory aspect of it, we go to the cool part, which is the demo aspect where Patrick is gonna show everything that I said but on an actual live cluster. So over to you, Patrick. Hi, everyone. My name is Patrick. I'm from Red Hat and I work with Suria on Open Kubernetes and general OpenShift networking. So what we have prepared here is how all of what Suria just shared with us ties into an actual cluster. So for our general upstream development, we are using kind. So in the recorded demo, I'll first clone our upstream repository, which all of you can easily do. And then we'll move to actually starting up a cluster. So as I said, we are using kind. Kind is Kubernetes in Docker. So for every container that Docker creates, you will see a Kubernetes node. So we've prepared in our upstream repo a very neat script that allows you to, with one command to create a kind cluster that actually uses our CNI, Open Kubernetes, as it's networking. As you can see, there is a ton of options that you can choose from to set up different scenarios like IPv4, IPv6, dual stack, and all of different features that are available in Open Kubernetes. As I said, to set up the cluster, you just need to run one command. It will take around five minutes to set it up. I'll fast forward here so we don't waste too much time. As you can see, we build all of our containers, we've set up the Kubernetes cluster, and at the end we apply simple YAML files that take care of deploying Open Kubernetes into a cluster that initially doesn't have any CNI on it. Okay, just tying it back to kind, if any one of you is not aware. As you can see, we have three nodes, Open Control Plane, Open Worker, and Open Worker 2. These are, of course, represented as Docker containers, and we have a little bit of an exception here, and if you exit into any of those Docker containers, you can see that inside there are more containers that are running inside. So there is cryo and you can see that there are additional containers running on our Worker node. Now, moving on to Open Kubernetes. As Surya said, we have a Control Plane and the Data Plane. The two top pods contain our Control Plane components. The first one contains Open Databases that we use to store the logical entities. That's the guy here. And the second one, OvenCubeMaster, that's the main piece that does the translation between Kubernetes and OVEN. So this is the component that will create logical entities in our NVDB that will be, that I'll talk about more in the next slide. Of course, we also need to provision the Data Plane components, and as you can see, the next piece is OvenCubeNode. This is a Damian set that runs on every node in the cluster, and you can see that in the nodes section, and it contains OvenCubeNode pod. That pod runs the actual CNI. So when a pod gets created, Kubelet will call the container runtime, which will then make a call to the CNI, and will take that call and provision pod networking for every pod that's clustered network. Additionally, and that's something that's upstream specific, you can see that there are OVS node pods. These are running OVS upstream. So these are responsible for setting up the actual flows and applying them to actual packets. So after all of these layers of abstractions coming from Kubernetes to Oven, we configure flows that get applied to each packet that flows through our cluster. This is not something that we have downstream, and in downstream we have a host network service, sorry, we have a system D service that runs on the host. So this is not possible with a simple kind cluster since we have just Docker containers. Additionally, please note that we do not have HA here. I tried to keep it simple for the demo purposes, but of course you can enable it so all of the control plate components would be multiplied to provide the HA functionalities. The key in the IO view can see that all of these pods have IPs that are actually from the host network. This is because we are the CNI. There's no CNI to provide us with any cluster network level networking. So that's something that you will often see in Oven Kubernetes that our data plane pods and control plane pods are host network. And yeah, you can see that they share the same IPs out as the nodes. Let's get back to the slides for a second and talk about how we represent a cluster in Oven. Okay, yeah, as Seria said, the cluster is represented with a bunch of switches and routers and that's it. It's very simple to grasp if you know what the switch and the router is. So as you can see the dotted boxes represent every node in a cluster. So here we only have two but you would get everything that's in the dotted box. You would see on every node of your cluster. Up top you can see the underlay network. This is the network that allows us to connect the nodes and to provide external connectivity to the cluster. And this is not something that we configure and it comes as underlay network so it's provided to us and we can use it to join the nodes. You can see here that every node in the cluster gets a host subnet. This is a subset of the whole cluster network. It's specific per node and all of the pods in that specific node will only have IPs from that host subnet. Starting at the bottom, as Seria mentioned, every pod will get its own logical switch port. Every pod will have it created by us so during the pod creation, we'll create a VIF pair and one of the ends will get connected to the pod net name space and the other would get plugged in into an OVS switch that would then be represented as a logical switch in OVN. So this gives us the advantage that if there are two pods on the same node, they will use that logical switch and its local so there is no need for the traffic to traverse node boundaries. Now, if the pods are located on different nodes and they need to communicate with each other, they need to have something that connects them. So we use a centralized OVN cluster router for that. This is a singular router that will be only one per cluster. So this gives us an opportunity to connect logical switches from different nodes and additionally, it gives us a path to northeast, north-south connectivity so if a pod wants to talk to the internet, it will go through the gateway router. So the gateway router is a node-specific router and it is responsible for providing the ingress and egress functionality out of and to the cluster. As you can see at the top here, it actually takes over the default interface that's on every node. So when you provision OVN Kubernetes, that interface that was the default interface connecting the nodes will get taken over by us and we'll manage it and that thanks to this, we can provide external connectivity and direct traffic as we need to. And one note I wanted to make here, if your pod is host network, which means that it doesn't need the cluster network IP, it's not going to be represented in OVN. It's not going to be represented in OVN and most of the traffic flows it does to external entities will not use OVN. So thanks to it, our CNI uses that approach and we can set up everything before we start handling pods. Let's see how all of that looks in an actual cluster. So I've prepared the trivial YAML file that contains a client pod and a service backed by a deployment with three replicas. So I did that to showcase the very minimal approach that you could take to see all of the components that OVN configures to provide the service and pod to pod connectivity. So how do you actually see what's there? So we applied the pods, we see them here and they all are running and right now to look at the logical representation in OVN. What I do is I enter one of our pods so we can directly interact with the database. So for the remaining part of the demo, you can treat it as the left side contains what's in Kubernetes and the right side represents what's in OVN. So what networking components have we configured that allow the connectivity between pods and services? As I said in my previous slide, we have three nodes and all of them get node-specific gateway router that we use to provide external connectivity basically. And additionally, as you can see, there is just one central OVN cluster router that gives us the possibility to connect pods and services that run on different nodes. There are also logical switches. So every node gets its own logical switch and additionally we use external switches to take over the interface from the host. And we use one joint switch that allows us to basically connect gateway routers into the cluster router. Okay, moving forward. On the left here, you can see pods with a wide which means that you can additionally look at the IP addresses that we've allocated and the nodes that the pod is running on. So here, I show the OVN worker switch. And on that switch, the first port that you see is one of our packing pods for the example service I created. Additionally, you can also see that on the same switch, we have more ports that are not related to that deployment. So all of the cluster network pods that are on a particular node would show up here. We of course have a port that allows us to connect to the router and a port, the worker port that allows us to provide cluster network to host network connectivity. Yeah, same goes for the other switch that hosts the default client and additionally one of the packing pods for our service. So you can compare that to what we see in the Kubernetes world and in matches. And additionally, we see that we have the IP addresses and MAC addresses that we've allocated in OVNK. Let's take one of the logical switch ports as an example. And this is the object that we store in the database that represents one of the logical switch ports. And one interesting thing I would like to point out here, we are using OVNK port security, which is a feature that allows us to restrict the traffic that's allowed in the switch. So anything other from this IP coming from this particular pod won't be allowed. The same goes for the MAC address. So if some rogue user enters the pod net namespace and changes the IP address on the VIF pair from that namespace, we won't allow that traffic. It will get dropped, so that's a neat feature. Additionally, we also see this. This is useful if there are any issues with that port, but here you can see that it's up, so it's functioning correctly. Okay, so we've talked about pod to pod connectivity and how those ports are connected to a switch, but what happens if a pod wants to connect to the internet? What we do in the gateway router, we create an SNAT rule. So we're going to translate the source IP of that particular pod into the IP of the node. So to the outside world, the IP that's egressing the node will be the one of the nodes. So that's the IP address we take from the default networking interface, and then we'll an SNAT, so we'll translate it back when there is a reply from the external line. So as I said at the beginning of this part, we've also created a simple service, and you can see the details of it here. So it's a service with that IP address listening on port 80, and these are the backing pods that are the endpoints for that service. And let's see how that behaves in Oven. So in Oven, we have a construct called load balancer, and it does basically load balancer traffic from this IP address on that port into all of our endpoints that we've defined in the service. The name of this entity is matching the service that we've created. So as you can see, you can easily get the networking representation of what's actually in a Kubernetes cluster. You don't need to look at particular OVS flows, you don't need to dig through layers of IP tables, and it gives you an insight view of how the networking is configured in Oven. Of course, all of this, it's logical representation of the network. It's not something that will actually handle your packets. So what you can see here is a dump of flows that we configure in OVS. So every logical switch, every router and every nut will eventually get translated into an open flow rule that will be applied by OVS and its module in the kernel, pair packet. So it's extremely valuable when you are deploying features and when you are doing it fast to have an abstraction layer that allows you to not deal with all of that because it is really difficult to dig through all of the layers and steps that the packet takes when it traverses your cluster. And that concludes the demo. Yeah, I'll just briefly go over the... Okay. We do have some additional features in Oven K, but we unfortunately don't have time to get over them, sorry. But you can check them out on the slides or you can check out our OpenShift or Oven K documentation for that. And that would be it. Thank you all. And do you have any questions? So the question is, is OVN better at throughput compared to OpenShift SDM? That's an extremely difficult question because to get the answer for it, you would have to specify the exact traffic flow that you use to pull that throughput because it's different when there are multiple hops between pods, when there are different traffic paths and especially if you consider packet sizes, protocols and all that, it gets difficult. At the lowest level, when everything's configured, it's OVS. So it will traverse the OVS data path and then based on the connectivity between the nodes, this will be the engine that handles the packet. So this OVN as itself is an abstraction that in the end will still configure OVS flows. So I would consider them to be similar, but to get any detailed results, it's not really easy to answer that. We use Geneve, so with all of that, with either VXLan or Geneve, there is an overhead, right? And it's something that you can alleviate by using something like hardware offload, right? And as far as I know, it's more difficult to get hardware offload for Geneve but yeah, there is an overhead and I wouldn't think that Geneve is much better than VXLan in terms of the overhead, especially that we pump more data through Geneve because we use the optional header options in Geneve that will increase the packet size. Yeah, you just need two jumbo frames and one traffic path and you'll be good. It seems we are out of time, so thank you all.