 Hello everyone, welcome to our session as part of the Open Infrastructure Summit 2020. Today we're gonna talk about run your Kubernetes cluster on open stacking productions. My name is Ramona Celor Rodriguez and we have today with us Anita Druckler and Frank Budin and we are product managers in the Cloud Platforms business unit of Redskirts. Let's start by a general vision of Kubernetes and the hybrid cloud and how the hybrid cloud powers Kubernetes. And when we think about what developers are doing with Kubernetes, they are developing applications. They care about that, about the experience when developing applications and they would like obviously to have a consistent experience across different platforms, right? They don't really care or not too much at least about the underlying platform, whether that is on Previx or on a public cloud and which one of them it is, as long as they can do all the tasks that they are used to do with Kubernetes. And in line with this, the open hybrid cloud concept again rests on four footprints essentially. One is bare metal, that means, you know, your cluster will be directly installed on bare metal nodes. Virtual, traditional virtualization, any of the popular platforms for traditional virtualization. It's another very successful and popular footprints. And private clouds, which is one we're gonna talk about today and public clouds, obviously, but always remembering that containers are hosting applications and the developers doing this really want consistency across the four footprints. And Kubernetes has been pretty good at doing that. Kubernetes, along with CoreOS, in this case, allowing us to do deployments on different types of footprints and keeping the experience as consistent as possible. So as I said today, we're gonna be talking about Kubernetes on OpenStack. OpenStack is our private cloud infrastructure private cloud infrastructure that is going to allow us to provide this consistency that we're talking about or the experience that we want to offer to our developers with Kubernetes. And start by, why Kubernetes on OpenStack? Well, as I said, Kubernetes is workload-driven. What we care about as developers using Kubernetes is about the workloads. We love Kubernetes. We really like to do very sophisticated or simple things with Kubernetes through its API, CLI, but in the end, what we are doing is working on our workloads, right? But at the same time, Kubernetes and OpenStack are deeply integrated. This integration is the result of years of work that we've been doing to make Kubernetes consume OpenStack resources properly, to be able to understand what these resources are and how to use them in a programmatic way, actually, which is another of the advantages that we have when integrating these two platforms. OpenStack is 100% API-driven, and when working on this integration, in reality, what we are doing is Kubernetes as the consumer, as the client, is using the APIs to request requirements of compute, storage, network, anything that you may need for the applications that you are developing. And then more things then about OpenStack itself, OpenStack is a platform designed to scale out. So in the private cloud, when you are using OpenStack, chances are that one of the main reasons is because of its scalability, right? It has been designed to scale across all the data center and make an abstraction layer of all the infrastructure that you have in your data center, storage, compute, networking, all of it is made, say, standard so that you can consume it in a standardized way. And in this case, OpenStack makes this abstraction layer to be consumed by Kubernetes. And this stack is on top of Linux, which is a solid foundation. Here I mentioned Redcat and Christ Linux. This is the Linux we are the most used to work with at Redcat for sure. And it's a solid foundation above which the rest of the stack rests and we can have the peace of mind that this gives us. Here we can see the three main integration points more in detail. Compute on the left is used to deploy, scale, manage the instances, whether they are virtual machines or physical machines, provided by OpenStack, right? And here you can see Nova and Ironic. Nova is handling, well, these instances, Ironic more specifically with their metal nodes. And this is yet again another example of how to consume OpenShift, how OpenShift is consuming OpenStack resources. And if we move to the middle, we have the storage integration. Storage is used by the containers, the pods themselves when they are creating persistent volumes and how do they consume that? Well, OpenStack exposes access to persistent volumes by a cylinder. The same goes for the registry, for example, where it is common to use Swift object storage provided by OpenStack. And when we move to the network, again, OpenStack exposes not only the networks themselves, every network is a resource that can be consumed by the OpenStack clients of Kubernetes in this case, but also load balancing as a service. And then we have an integration point that you can see here that's called Courier that optimizes the performance of the traffic between pods. We will see a little bit more of that later. Later meaning now. Courier Kubernetes, the CNI for OpenStack. So Courier is the technology that improves the network performance of pods running on OpenStack. And Courier itself uses the Kubernetes container network interface, CNI. And essentially what it does is it uses Neutron with any SDN, any plugin that Neutron might be using, but it will use the standard Neutron APIs and it will connect the pods with the VMs through Neutron. And what does that achieve? Well, example, East-West traffic is greatly improved. And the reason, the main reason is OpenChift Kubernetes, it use VXLand tunneling to communicate the pods between them and pods in different physical nodes. And OpenStack does exactly the same, by default Neutron will use VXLand tunnels in the networks that are available for the virtual machines. What Courier knows is how Neutron does that, how OpenChift or Kubernetes does that, and then it connects the two, in this case, VXLand terminal endpoints to make one tunnel instead of using the tunnel that you have in OpenStack and then fit another tunnel in it, another VXLand tunnel with the level encapsulation that we have experienced to be, if not problematic, not very efficient. Now, I wanted to share with you a reference architecture as part of this introduction that you can read online. Here you have a URL. And in this reference architecture, we have added what we see as the base to build your OpenShift and OpenStack infrastructure, including everything from the networking to the storage. And obviously you will have to adapt to your topology, to your requirements. Reference architecture is that reference, but it's a solid one and I would say a very comprehensive one and the best one we've done so far based on OpenShift 4.4 and Redcat OpenStack platform 13 or 16 both. And well, I wanted to share with you in general the strategy and vision of Kubernetes when running on OpenStack. And one of the areas that we are putting the most work on is the deployment user experience. We need to continue improving the user experience when you deploy OpenShift on OpenStack. Kubernetes on OpenStack. And obviously we need to add as many use cases as our customers need being reasonable. And at the same time, always trying to simplify the installation workflows, making all the decisions that we should be making for you rather than exposing 1,000 different flags for you to look at and decide what to do with them. Another area of focus is the interaction with bare metal. Remember that OpenStack allows you to manage bare metal nodes. And we want OpenShift to be able to take advantage of this functionality. And this opens the door for a number of use cases, right? Customers, users who rely or want to use SRIOV and NVME, GPUs, FPGAs, or simply any performance and CTFabs, they will rather run on bare metal nodes. That's what we are learning from the users using this stack. And another important area of focus that we're going to touch on these today is the Telco Edge use cases. OpenStack itself is the most popular platform in Telco and OpenShift and OpenStack is a strategic for many of such users, right? So at the moment, I try to express in this diagram below what users are wanting is to use VNFs and CNFs together. Right now, there are many VNFs being run on top of OpenStack. And many people are starting to use CNFs running on OpenShift, which in turn can run on OpenStack. And this stack with both of them running at the same time and exposing capabilities that are addressed to these type of use cases. For example, distributed compute nodes, access to SRIOV devices from the containers directly is something that's strategic for these customers and for our vision as well. And to finish, well, you can try this now. Go to this URL, cloud.com, OpenShift, install OpenStack and you can start today. And with that, I'll pass it on to Anita. Thanks, Ramon. And welcome, everyone. I will be covering Kubernetes and OpenStack architecture with career. In this example, OpenStack implements the networking for Kubernetes and the use case we've taken is with OVN as the SDN option. OVN is handling tenant networking with overlays and DVR. External connectivity is supported with floating IPs using that. And OVN provides DHCP services, distributed DHCP services, network policy using OVN security groups with contract and east-west load balancing with OpenStack's Octavia solution. Cubeproxy is disabled in this use case. Looking at Kubernetes with OpenStack networking and the options available. If you're using career with tenant networks or another option is using provider networks. If you're using career and you have OpenStack tenant networking, then you're using VLAN trunking for all east-west traffic. This is to avoid the double encapsulation and to also support multi-tenancy in the worker node. Every worker node will have VLAN trunks with trunk ports identifying which pod with segment IDs and VLAN IDs for each pod. Here's an example on how to set this up with the installer. You would need to create a worker port on the worker VM and then create a trunk port on this port and then set up floating IPs for the VM for external connectivity, add supports for the trunk port to support pods with segmentation ID or VLAN ID to identify each pod. Now let's look at the next option which is provider networks when you're running Kubernetes and OpenStack. This is for use cases where you don't want to use floating IPs or NAT or you have external load balancers and you want to avoid overlays in the infrastructure. Let's look at what we have today with the automated installer provisioned infrastructure which is the IPI installer. This requires a floating IP for external connectivity. You have an alternative today to use a provisioned infrastructure as well where you bring your own infrastructure and then install Kubernetes OpenShift on top of that and that allows you alternatives. But with the IPI installer with OpenShift 4.5, we've added bring your own external connectivity or your external load balancer and with OpenShift 4.6, we have now support for no floating IPs with provider networks. You can avoid NAT and career is not needed in this use case and there is no overlay provided by the infrastructure. Everything is provided as the example shows here using your top of rack as your gateway and we have a provider network of VLAN 101 where your compute and your parts are on the same network. So if you're using East-West traffic, you can use OpenShift's default overlay networking which will create over the provider networks. As shown above, you can use standard overlay as an option and you can have your external load balancer as an option as well. So as an option to Kube Proxy, if you're using Kube Proxy, that would be the default for East-West with overlays and if you had an external load balancer, you can have both in your North-South traffic as well as your East-West traffic using your external load balancer. As we said, the top of rack is the gateway and you need to ensure connectivity for both your OpenStack and your OpenShift or Kubernetes API. There is some manual API configuration needed in this use case. Let's look at East-West load balancer options. With the default SDN, you have Kube Proxy and your alternatives are, you can use with Courier OpenStack's Octavia Amphora and Courier has an option for Octavia OVN. The option for Octavia OVN was added recently in the train release and it has some limitations like there are no health monitoring options which relies on Kubernetes for health checks and it's only recommended for East-West layer four services not layer seven where you need more ingress-like services from Amphora. Let's look at this example where an app one web service needs to communicate with a backend app two database service. In this case, for load balancing, you have with Octavia two options. You have the Amphora option where a Amphora VM needs to be spawned for every tenant and then your traffic needs to be routed to the Amphora VM and then load balance to the destination part. This adds overhead both on the control plane for spawning the VM and latency on the data plane. And an alternative to that is using the OVN load balancer where all of the load balancing happens on the node itself without the extra hop or the latency, control the latency for spawning VMs. Let's look at the performance comparison for a use case where you're running part to service to part and your parts are on different hypervisors. In this use case, you can see with throughput performance, there are three options. You have your default OpenShift SDN with double encapsulation. And you have Courier with Amphora, Octavia Amphora and Courier with Octavia OVN. And you can see for throughput performance, Octavia OVN has the best performance, but when you're looking at DCP request responses, and you can see over here that the Courier Amphora option has a higher latency overhead because of the extra hop. Let's look at network policy implementation with where every Kubernetes network policy gets translated to an OpenStack security groups and eventually to OVN ACLs. And there have been improvements at the OVN ACL level. In Kubernetes, pods are transient. There's constant churn, pods are being added and removed as needed. And network policies and security groups need to be updated all of the time. OVN ACLs need to be updated on nodes as churn happens. The new feature being added to OVN is incremental processing, where you have only those nodes that are affected need to recompute their logical flows. And this has improved CPU and reduced latency by over 50%. Let's look at OVN ACL improvements for multi-tier security. In this use case, you have a front-end web server that goes through a business application logic and then finally accesses the backend database server. We wanna add an ingress security group for the database server to only allow the business application to talk to the database server. The front-end or any other application is not allowed to directly talk to the database server. In this use case, you can use new features added to OVN called remote port groups and address sets. Each port group is an application tier. And for example, a port group is added for the database application as well as a port group is added for the business application logic. When a part is created, it is associated with the appropriate application tier port group. And OVN ACLs are now attached or added to the port group directly and don't have to be added to the logical switch or network or port. And at the same time, we have address sets being created which include IB addresses for all the ports in that port group. These are automatically created for each port group so that you can perform match actions on that ACL more quickly. And every time a port is added or deleted, you just need to remove it from the port group. You don't need to go to each of the logical switches and update. This has improved efficiency in the creation and deletion and management of OVN ACLs and network policy by over 50%. Thank you, that's all for me and I will hand off to Frank. Thanks, Anita. Let's have a look at the Telco use case. 5G is pushing for cloud native network functions meaning that network functions implemented so far as virtual machines in the context of 4G are now expected to run as containers on bare metal. However, not all network functions are going to be containerized. For instance, existing 4G VNF are not planned to be containerized. In that context, as 4G is here to stay for years, what are the options to run 4G and 5G? Meaning VMs orchestrated by OpenStack and containers orchestrated by Kubernetes. We foresee three options represented on this slide, from left to right. First option is to run side-by-side clusters, one cluster per type of workload. Second option in the middle is to run Kubernetes as an OpenStack workload within virtual machines. Third option on the right is to run OpenStack virtual machines as Kubernetes workload, thanks to CubeVeat. Let's zoom in the middle option in the following slides. Running Kubernetes inside virtual machines provide all of the benefits of a private cloud to the Kubernetes cluster and to the containers it orchestrates. First one is to abstract Kubernetes from the underlying hardware, which is virtualization most obvious benefit. Then the network interconnection between containers and virtual machines is taken care of by Neutron. Even the interconnection between multiple Kubernetes clusters hosted on the single OpenStack cluster. This is very handy for CNF development as each and every developer can get its own set of Kubernetes clusters and experiment various network configuration without needing access to dedicated physical servers. Finally, the container data plane performance is identical to bare metal performances. And we demonstrated this live on stage last year in Denver at Open Infrastructure Summit. Let's take the example of a Kubernetes cluster needing to be connected to three networks in a virtual radio access network, for instance. We need to connect the Kubernetes cluster to three different networks and we need to connect multiple pods on those networks. Here is how we proceed. We boot an OpenStack virtual machine with as many Neutron pods as needed to connect all of the pods on the three network. A single management interface using overlay will be used and one Neutron port per pod for the front hole and for the mid-hole networks. Each and every Neutron port is incarnated inside the VM as a PCI device. So we just need to configure the proper CNI for those PCI devices. The primary CNI on the management interface and SRIOV device plugin for all others because, again, each Neutron port is a PCI device inside the VM. This solution permit to associate any Neutron port numbers with any Neutron port type to pods. It can be OVSMN to pods based on OVSTC flower hardware offload or OVSDPDK. It can be SRIOV. It can be any SDN plugin that Neutron supports. The mapping between Neutron networks and Kubernetes custom resources is realized thanks to Novart device world-tagging feature. Each and every port has an associated tag which is a free chain of character. And we can retrieve the tags associated to a given Neutron pods in the OpenStack metadata as you can see on the slide. So we just need to pass the virtual machine metadata to create the corresponding Kubernetes custom resources. We have demonstrated before COVID live on stage last year in Denver how to create one pod connected to one Neutron OVSDPDK port and to one Neutron SRIOV port. And we demonstrated, again, live on stage, bare metal performances. In that case, around four millions of packets per second with a single OVSDPDK core. So as a conclusion, I would say that OpenShift on OpenStack is definitely ready for any kind of use case, including the most stringent, telco and mission critical applications. Thanks for attending this session.