 that form part of the design, what they are and what they're used for, etc. And then I'm going to tie all that together in terms of what the high-level design for the integration of IPsec in our own communities and open up a bit of work like that. And then we'll have potential future work and I'm going to do that in the audience. So we'll start off by looking at the IPsec use case. So our use case is, as a cluster administrator, I need IPsec to ensure traffic between pods and the pod network is confidential, authenticated, and has not been tampered with. So if we look at this diagram here, which is a diagram of a three-node OpenShift cluster, we have three nodes, node X, node Y, and node Z, and they're all interconnected on an underlying network, so every node has a connection to another node, to every other node on the network. And on top of these nodes, they're hosting a number of pods, so pod A, B, C, D, and E. And what we're trying to do with this work is we're trying to ensure that any traffic between nodes has been encrypted with IPsec. And the reason we're trying to do that is we're trying to prevent a bad actor who is sitting on one of those underlay networks, so in the physical connections between the nodes is unable to view or tamper with the traffic between the nodes. We're not looking at encrypting the traffic between, for example, pod D and pod E, which are on the same node. We're really only looking for traffic between the nodes, the traffic that has left the node on this pod network. The initial requirements for this were outlined in an OpenShift enhancement proposal, and the link is here in the presentation. And I'll just go through the list of requirements that we worked with initially, and it's listed up here. So the first one is this has to be supported in OpenCubernetes. It requires support for IPv4. However, now subsequently IPv6 support has been added. It has to work on rail 8, that's the version of rail the cluster was based on at that time. It needs to have support for CA sign certificates, and I'll talk about what that means in a later slide. It's a cluster installation time parameter. So that means you can enable this when you're deploying a cluster, but you can't enable it after the cluster has been deployed. And then again, that is something we're going to look at at a later stage, being able to enable this at runtime. And initially, we weren't really focusing on performance or the ability to offload in order to improve performance. We were focused on functionally getting this working and just enabling the ability to encrypt traffic with IPsec in these clusters. Okay, so now let's look at some of the components that we use to enable this. The first thing to notice is there are a lot of components. I won't go into the details of all of them. We'll focus on a few key ones over the next couple of slides. But this slide does highlight the kind of breadth of components that are needed to enable this feature. I'll also highlight some of the version numbers required to enable it. So let's start talking about IPsec. IPsec isn't a software component per se in this design. However, it is a key part of it because IPsec is a protocol we use for the encryption and the authentication and all those type of features that we talked about in the use case slide. So what is IPsec? Well, it's a suite of protocols that provide secure encrypted communication between two endpoints. So those endpoints could be two hosts that are talking directly to each other. They could be a host talking to a gateway, an IPsec gateway, or two IPsec gateways talking to each other. Where a lot of people would be quite familiar with IPsec, I guess, is in the use case of a road warrior. Someone has their laptop, they're working remotely from the office, and they set up a secure VPN connection with the office. And a lot of times that's done over IPsec, where your host is talking IPsec to IPsec gateway and then the IPsec gateway, and it does so in an encrypted manner. And then the IPsec gateway provides that unencrypted access to the internal network of the business. IPsec is a little bit different to other similar protocols such as TLS, because it works at layer three at the IP layer. So what that means is you can take, for example, an IP packet, and you can also encrypt the header information within the IPsec packet. IPsec is made up of a number of protocols. I've listed the kind of main ones here, and we'll talk about them in a little more detail. So AHESP and ICACAM. AH stands for authentication header. The RSC is mentioned here. And what that provides is it provides authentication and data integrity. So authentication means it provides a way to ensure that the end point that you are talking to is who they say they are, and you can confirm that. And it also ensures that the data is transmitted in a way that you can ensure the integrity of the data that's received. ESP provides data confidentiality. So this is, I think, what most people would think about when they think about IPsec. ESP provides the secure encrypted traffic. So you create a packet with clear text data, and the ESP packet that gets produced will have an encrypted ciphertext that a bad actor will not be able to take and then determine what the clear text data is. And it also provides authentication and integrity as well. It can provide it. ICACAM. IC, which is Internet Key Exchange and Internet Security Association is a key management protocol. That's a protocol that it's a framework for establishing keys between the two hosts, establishing the authentication and basically negotiating something called a security association, which is a set of parameters that define how these two end points should talk to each other. So it defines these security parameters and then it allows communication of these parameters, negotiation of these parameters between the two end points. What they do as well is it defines it in one direction. So if you have two hosts, host A and host B, you have two security associations. So you have one for communication from host A to host B and then one for communication from host B to host A. I think another quite an interesting thing about these protocols is they're actually IP protocols. So they work at the same level as TCP and UDP. So whereas TCP has a protocol number of six and UDP has a protocol number of 17, ESP has a protocol of 50 and AH has a protocol number of 51. And when you're doing a TCP dump on the network, you can see that it's a little bit different than say a TCP protocol where you would see the port number of that TCP packet, but it's still a TCP IP packet. So it is a little bit different in that respect. IPsec works in two different modes. So there's a transport mode and a tunnel mode. Tunnel mode, I haven't presented here because in the work that we've done, we use IPsec in transport mode. But basically what tunnel mode is, is it'll take your entire IP packet and it will encapsulate it in an IPsec tunnel. And so that the IP header, in this case, in this diagram would be the IDER IP header, was fully encrypted. However, in this work, we're using IPsec in transport mode. So in transport mode, the IDER IP header remains the same. However, all the stuff within that IDER IP header gets encrypted in a ESP packet. So the two software components that provide the IPsec services for Open Kubernetes and OpenShift are Libreswan and XFRM. So Libreswan is an open source implementation of IPsec and ICANN. It's a user space application. And XFRM is a kernel space framework that provides the cryptographic services required for encrypting traffic between two hosts. So Pluto is the main Libreswan daemon. There's various other bits of software within Libreswan, but Pluto is the main daemon. And its responsibility is for negotiating the security associations with remote hosts. So as a user, you would configure Libreswan in such a way that you would define, for example, with host B, I want to encrypt traffic to host B in a certain way. And using that information, Pluto would attempt to negotiate a security association over IC with that remote host based on the configuration that you pass to it. If it successfully negotiates that security association, it will then use that by installing it into the kernel, into the XFRM framework. So the XFRM framework defines two objects. It defines a state object and a policy object. And the policy object is the thing that hosts the security association. So Libreswan will set up some policy objects saying things like, for example, if you're talking to host B, I want you to encrypt the traffic in a certain way. And once that's there, then traffic that is sent will get encrypted that way. So if I look at the next slide, we add another application called Alice. And if Alice is communicating with host B, Alice will send traffic to host B as per normal. However, XFRM will intercept that traffic because it will match a policy in XFRM. It will encrypt it based on the security association that was configured by Libreswan. And then it will send that traffic, that ESP or AH traffic to the remote host in a secure manner. So when you're configuring Libreswan to talk to a remote host, there's a number of ways that you can authenticate yourself and mutually authenticate two hosts. Three ways you can configure it. One is using a pre-shared key. Two is using a self-sign certificate. And three is using a CA sign certificate. So when using a pre-shared key, what that means is that when you're setting up an IPsec session between two endpoints, you need to define a shared secret, so a long, complicated password, for example, and then configure that manually into each endpoint. So when the endpoints are communicating, they can ensure that they're talking to the right person and then they can negotiate their session based on that. That's not very practical from when you're trying to scale this to hundreds of nodes, for example. So you can also use certificates. So for a certificate, basically you can generate a public-private key pair where the public key is something that you can share to any node on the network as a way, and that can be used to then authenticate you as a valid endpoint. Those certificates can either be self-signed or CA signed. Self-signed means that you sign the public key yourself, which is less secure because it just means you're kind of guaranteeing that your key is valid to the other users. A more secure way of doing it is through a CA signed certificate where when you get a third party to sign that certificate and say that this particular key is valid. So in the case of open Kubernetes with OpenShift, we're using a scheme in which we are using CA signed certificates and we'll talk about that a little bit in a later slide. Obviously, the pre-shared key just wouldn't scale in our case and so the CA signed certificate is the way to go. Another thing to note, and I'll talk about this a little bit in a later slide, is when we're negotiating sessions between endpoints in Open Kubernetes, we're ensuring every connection between a node has its own CA signed certificate key pair. So the next software component I want to talk about is Open Kubernetes. Open Kubernetes is a container networking interface, a CNI network provider for Open Kubernetes. So what that means is the software, it's a suite of software that gets deployed to the cluster and it enables networking services for pods that reside in the cluster. Open Kubernetes is part of an ecosystem of software based on the Open V-switch project. So Open V-switch or OVS is a community-developed implementation of a virtual switch. So what that is is a piece of software that resides on nodes and it allows containers or virtual machines on the node to talk to each other or with the physical network. So OVN Kubernetes is made up of the OVN Kubernetes software itself and also software called OVN. OVN stands for Open Virtual Network and it's a community-developed vendor agnostic network virtualization solution. So what happens is that an administrator will configure networking in Kubernetes. OVN Kubernetes will then take that networking configuration and using that it will define a logical view of the network. So how the different pods should be connected and what are this networking services associated with those pods on the network. And then OVN will take that logical view of the network, this virtual view of the network and it will translate it into a configuration that can be used by OVS nodes on the cluster to enable this virtualized view of the network. We don't need to go into too much detail in this presentation. However, it is important to note that OVN Kubernetes is the software that configures the network and can configure the network in an OpenShift cluster. The other key thing to note is that in the way OVN Kubernetes sets up the network any traffic between nodes configured by OVN Kubernetes is encapsulated with the Genev protocol. So in this diagram here we have node X, node Y and node Z. We have a tunnel set up between node X and node Y, tunnel A. And all traffic within that tunnel is traffic between pods on node X and pods on node Y, at least pods in the pod network, and similarly between other nodes on the network. Another important piece of software in this design is the cluster network operator. So the cluster network operator, it follows the operator pattern and using that it's the software that installs and upgrades networking components on an OpenShift cluster. So it's responsible for deploying OVN Kubernetes and all the various components of OVN Kubernetes on the cluster. And that includes it's responsible for deploying the OVN IPsec pod, which we'll see in a later slide, which hosts the LibreSwan software and an OVS component called OVS monitor IPsec. So it's responsible for deploying all that software within the cluster and also it's responsible in this case for hosting a certificate signer. So this is a piece of software that implements the Kubernetes certificate signing request API. And what that means is the cluster network operator can act as a CA for the IPsec sessions. So when a node creates a key pair it can send it to the cluster network operator for signing and then use within the cluster. So let's talk a little bit about how these software components interact together to enable IPsec in this cluster. Because OVN Kubernetes is responsible for configuring the network it has a visibility of all aspects of how the network has been configured in the cluster. And it can then use that information to configure LibreSwan to provide IPsec for various traffic, some traffic on the cluster. So in this diagram we have three nodes, X, Y, Z, forming part of a cluster. And as we mentioned previously, every node in the cluster sets up a Genev tunnel with every other node in the cluster. So in this diagram we have three nodes and therefore we have three tunnels set up between the nodes. Pod A wants to send traffic to pod C, so pod A is a node X. It will send, node X will then send that traffic over Genev tunnel A to node Y where it gets decapsulated and sent to pod C. Whereas if pod D and pod E want to communicate with each other in node Z that traffic will not get encapsulated by Genev. So all this pod traffic is encapsulated over these Genev tunnels. So OVN Kubernetes is responsible for configuring these tunnels and what the Genev traffic will look like. So for example for Genev tunnel A it knows that the source IP address of tunnel A is node X is IP. It knows the destination IP address of Genev tunnel A is node Y's IP address and it knows that the destination port is UDP port 6,081. And it can use all this information then to configure LibreSwan. So it can say hey, if node X, if I see traffic coming from X's IP address to Y's IP address for this destination port I want you to use this particular security association that I've negotiated between node X and node Y to configure the traffic between those two nodes. And vice versa. So also in the case where return traffic is coming from node Y to node X. So that was a high level view. So now how are these different components that we've talked about deployed to achieve that? Well in the cluster we have two classes of nodes we have worker nodes and master nodes. And master nodes are responsible for configuring and managing the cluster where worker nodes are the nodes in which we deploy our pods for example. So in Oven Kubernetes the master node hosts a pod called the OvenCube master which provides, which has all that contains all the components required for Oven Kubernetes and correspondingly on the worker node there's also a pod called the OvenCube node which provides the per node, per worker node software required for configuring Oven Kubernetes. The cluster network operator is deployed in the master node as a pod and the IP sec pod is deployed on every worker node as well. And as I mentioned the IP sec pod contains all the LibreSwan software it contains also an OVS daemon that's used for integrating OVS with IP sec. Then on the worker node you also have OVS which isn't actually provided in a pod it's provided as part of the host and obviously then you have the host kernel which could contain XFRM which we were talking about previously. So how is LibreSwan configured? When you add a node to the cluster, a worker node to the cluster the first thing will happen is if you've enabled IP sec it will see a node will deploy a pod on the worker node with all the various LibreSwan binaries and stuff that we need to enable IP sec so that's the first thing that happens. After that the node will generate its own public private key pair and it does that completely locally so the private key never leaves the host. However, after that the OVS will request that the public key gets signed by the cluster network operator so it's safe to send that certificate, it's actually a certificate to the cluster network operator and the cluster network operator CA will then sign that the certificate has sent it back to the individual worker nodes and at that point the node, just after being added to the cluster, will have its own private key that's never left the node and it'll have a signed certificate that it got signed by the CA on the master node and it's those two pieces of keying material that are used to authenticate that worker node with every other worker node in the cluster. So once it's done that, OpenIPsec will read the OVN configuration so it'll read the network configuration for the cluster and it'll understand, okay, how are all these Ginev tunnels set up, where are all the other nodes and it'll use that and the keys that it now has created and it'll use that to configure Libriswan to encrypt all Ginev traffic to other nodes in the cluster and once Libriswan has all that information Libriswan will then use that configuration to configure XFRM in the worker nodes kernel, so the traffic that is sent via that worker node over the Ginev tunnel will get encrypted. So how is the traffic encrypted? So in this example we have two worker nodes one worker node hosts a pod called Alice and the other worker node hosts a pod called Bob and Alice and Bob want to talk to each other. Alice and Bob are in the same virtual network in this cluster. So Alice will send a packet to the IP address of Bob and it'll get intercepted by OVS and based on the configuration that OVS has got from OVN and OVN Kubernetes it will decide what to do with the packet and it'll determine that Bob is on a remote node therefore it needs to encapsulate the packet in the Ginev tunnel between the two worker nodes and then when it sends that traffic out on the wire it gets intercepted by the XFRM framework in the kernel and the XFRM framework will look at that packet and determine it matches on a policy that was being previously configured by Libraswan and based on that match it'll determine that's got to handle that packet using the security association that's been configured by Libraswan. So once it's done that it'll encrypt the traffic it'll send it back out the wire and once it's on the wire that traffic is then encrypted between those two worker nodes and when the packet is received by the host that the worker node that hosts Bob the reverse happens so the network will receive the packet, the encrypted packet XFRM will determine it matches the security policy and the security association that's been configured by Libraswan it'll then decrypt the packet send it to OVS and an OVS based on the configuration from open Kubernetes will determine that that packet needs to be sent to the Bob's pod and the traffic will then be received by Bob in clear text. So what does the future hold? Well the roadmap is still open and there's still a lot of discussion about what the next steps are for IPsec in open Kubernetes and OpenShift some things have been discussed and some things have been implemented so as I mentioned previously initial support was only for IPv4 but in the subsequent release we added support for IPv6 and dual stack configurations currently IPsec can only be enabled at cluster installation time however and the reason for that is due to limitations about changing the MTU during runtime however in a future release we will enable support for configuring IPsec support at runtime so enabling and disabling that support currently it only uses one cipher type and that cipher is AES GCM256 however it could make sense to allow the user to configure the cipher type used by IPsec and that may be something that will change in the future the other thing is performance it performs how it does at the moment however we may eventually need to add additional performance improvements or potentially offload as well to various NICs so that's something as well that we may look at in the future and other things so feel free to reach out to me with any suggestions or questions about that and that's it so that's my email address there and now I'm just going to open it up for any questions from the audience thank you so much Maria that was a great talk I don't see any questions at the moment but audience if you have any questions feel free to put it in the Q&A section or the chat