 Okay. So, hello everybody. Hello guys. Welcome you. I would like to welcome you on today's session. Unified networking for VMs and containers in OpenStack on Kubernetes. My name is Artur Korzenewski. I'm working as software engineer at Intel. My name is Wadi Meriderman. I am working as software engineer and deployment engineer at Merenties. Yeah, we have prepared some research about how to connect VMs in one environment, networking environment. So, for our purposes, we defined unified networking as VMs and containers being in the same network that can reach each other using IP or L2 or L3 technologies. So, maybe first question to the audience, who is already familiar with technologies like Courier, Kubernetes, Galico, I guess a few. I'm seeing that there's also an expert of Galico and Courier, so it's good. So, why even do we want to join these two technologies, containers and VMs into one single, let's say, network? It's because microservices and containers are gaining popularity and also all the workloads are run inside data center in VMs, so it would be good to have them connected somehow in the networking layer. We can imagine use cases like front-end stateless services deployed in container and background like database services deployed inside VMs. And also, what's nice about having unified networking is have one single API, which will be like OpenStack Newton API for managing all the workloads in your cloud. So, our assumptions are we are using OpenStack on top of Kubernetes cluster, so every service is containerized and we can leverage features from Kubernetes like failovers, resilience, and so on. And like OpenStack is very popular and has stable APIs. It's good to use this API to manage both VMs and containers' networking layer. So, Kubernetes is very reliable, funded by Google, and has proven container-on-crestation engine techniques, and OpenStack on Kubernetes is bringing us ease of deploy, fine-grained control over the placement of services, of OpenStack control plane, and the ability to do the rolling upgrades and self-healing services. And today, we will concentrate on Bermeter containers, which is... We're not talking about Magnum and nested containers that are running inside VMs for security and other reasons. And, like I said, we have chosen two technologies, like KaliCut and Kooded Plus OVS for L3 and L2 connectivity purposes. Of course, there are other technologies like OVN, OVN, Flanner, Wave, and Romana, which is doing the same, but we didn't have time to cover all of them in this presentation. Speaking about Bermeter containers, there is a big OpenStack installation. There are many points for the employers which we are facing too. First of all, we need to make some multi-tenancy technology for big network, and there is actually not such a big variance of options. There is L2 segmentation with VLAN. Tunnels is growing, and VXLan and Ginev and GRE is all the way to segment data, data tenant traffic, and with KaliCut and all the technologies, like Romana, we have another option to segmentate tenant traffic using L3 segmentation, where either firewalling, like in KaliCut case, either IP namespacing, like Romana case, is using to segmentate traffic inside the data center network. For HAP services, in data center cases, we have also not so much of options, because either we need to install a lot of balanceries using IPvase, a bunch of HAP rocks behind the very repeat, or either we bought some expensive hardware which serve 10 geeks or more of traffic. So for designing big data centers, we frequently need to invent some options to terminate traffic on many boxes and we frequently start using OSPF or BGP endpoints to make multi-pass routing and Kubernetes on bare metals and KaliCut uses BGP as pretty helpful for this. Also, for exposing services to the Internet, we need to somehow terminate all this traffic, not only inside the cluster, like Kubernetes does with their cluster APIs, but terminate traffic to the Internet, so we can somehow make a filtration and fair warning of this traffic. Next button. So speaking about Kubernetes model of networking, Kubernetes doesn't really care about how the networking is working behind the stack. It implies that we can communicate it without any other port, without any nut, and it gets the same IP address inside the net name space and outside the net name space, but Kubernetes container somehow detects the IP address and sends it to configuration server, so another port could communicate with this IP address. And Kubernetes afflows all the communicating between ports to their CNI plugins, so Kubernetes doesn't really care about how the networking is working. So CNI driver handles how the VTH pair is set up, how to connect it to some networking gear like bridges in a career case or like routing setup in Calico case, and also it's a type of underlying infrastructure if it exists. Also, Kubernetes implies multi-tenancy models, so every port has a label with a attribute which namespace it runs belongs to and networking technologies that provides some firewalling between namespaces could leverage this labeling to set up multi-tenancy isolation between tenancies. So Calico is providing not only routing for VMs, but also multi-tenancy isolation. So, for example, you could set up Calico policies to firewall traffic between ports so you can communicate between ports of different namespaces. Another hard part of Kubernetes on bare metal is how to set up external access to Kubernetes. Basically, Kubernetes provides two proxy mechanisms that set up kind of load balancer using IP tables and it's set up virtual IP on every node and port could communicate to service which is a set of port using a well-known IP address which is built into service and it could be accessible so every Kubernetes node. This mechanism also could be used to make access to service from outside for implementing this you can set up external IP to the services but there is also a problem because as the IP tables is using stateful node, you need to somehow synchronize your state of node in case of your service IP will be set on different nodes. So if traffic goes to the next node, for example, it's that multi-pass routing for a data center, IP tables on the second node will not be known how to send the traffic and you possibly get your traffic to another port and your TCP flow will be broken. Another case to solve this is using Kubernetes not port technology where Kubernetes sets not for virtual IP address but for port on some of the node and you set the load balancer behind of your infrastructure to somehow balance traffic between your Kubernetes nodes or dedicated network nodes for this. But this requires another part in your infrastructure another single point of failure. So actually the point is Kubernetes on bare metal still have no any clues how to expose services. They try to cover this case with Ingress L7 proxies which could use as a programmatic router to HTTP services or every service which is right now can be proxied by Ingress but the problem is still the same. Nginx is still the Kubernetes port and it still requires to somehow handle balancing for them. Either you could put Nginx to the hostnet mechanism and somehow configure VRP to share the same IP address. So right now Kubernetes for bare metal is handled well on the internal port network using Kaliq or other technologies but still couldn't be used well to handle external traffic. So to have a good view how OpenStack networking can be mapped into the Kubernetes abstraction into OpenStack networking model. I would like to show what's the data abstraction for Neutron. Neutron has ports which has this connectivity information for single endpoint, single VM, single port and subnets which defines IP address ranges, delivers gateways, DHCP and DNS configurations options and networks which holds subnets and ports inside one bucket so every port on the same network should have equal connectivity. Security groups for managing their access control to the endpoints to the VMs, setting up the firewall like in IP tables and so on and so forth like IPs which provides external public irritable IP addresses or in data center use case some accessible IP address that is inside the corporate network range and for how the services are designed are placed on the computer network node the VMs on compute node connected with tap interface with firewall bridge, Linux bridge when you are using the hybrid firewall inside OVS and integration bridge with all VHD interface connected with the Linux bridge then the outside traffic is handled by tunneling bridge or VLAN bridge it depends on your use case and tenant network's configuration and on the network node you have the tunneling or VLAN bridge with integration bridge connected with Vrouter name spaces where the routing is taking place so in usual scenario the legacy scenario when you have centralized network node the traffic being recruited from one VM to the other VM in other tenant network should go through the network node so it's like when even two VMs are on the same node they will need to connect each other using the network node of course if you are not using the DVR and then it will be routed on the compute node and also in both cases DVR and legacy scenario network node is handling the source node so for fixed IPs reaching the outside internet access Okay, but Mkallika network is designed very differently than OpenStack network which provides a big IP segment to all of your cluster which is then segmented to small subnets like 26 for this case and each of this subnet is set on every of Minion and pod gets an address from this name spaces on every node and to make this platform networking works Mkallika propagates every of these small raw prefixes using BGP protocol so for example first node gets the raw name space of second node use BGP that is set by Mkallika internals also BGP could be used to integrate with data center span leaf or L3 top of the rack with routing capabilities or even closed apologist to propagate all the IP routers that are set on every node so you don't need to set up any nuts and internals you just push your routing information from the node to your network infrastructure and it erodes your IP packets like usual without any special processing also Mkallika provides isolation for every node so for every pod or every other kind of workload you set by setting up endpoint firewall like OpenStack does with their security groups so you can stop fine grained firewalling using somehow the same terminology that we familiar with security groups so you can filter traffic by ports, by source addresses and also by belonging to another workloads so you kind of erode traffic from one name space and forbid traffic from other name spaces it's not updated and I can show this so I talk about external network node to make Mkallika working with external traffic you can set up the Kubernetes proxy on every node erode traffic for every of these nodes but another case you can build for made-up installations is dedicate some of Kubernetes proxy node which is running on the Q proxy component and then either distribute routing to your gateway using BGP or just set up common address address around the C for every of your service using either manual setup of Carp, ERP name or using some of Kubernetes helpers that are already designed to set up ERP using a Kubernetes API this case will provide you with much more fine grained control of how the traffic flows by not using or routing to Kubernetes external IPs and using common address redundancy could run into a case where the traffic could get on the node where the IP table state doesn't share it So the second use case is using Qtir to manage connecting VMs and containers So what is Qtir? Qtir is OpenStack BigTent project that aims to provide nodes for networking into the containers So it's already supporting Dockerlib network and Kubernetes support is under development as we speak and as I prepared this presentation one month ago and it was still evolving so some kind of information can be updated in the master branch that may be already available So what does Qtir do is provide the CNI driver for the QBLET on the worker node and the CNI driver and API water which was called Raven You can see this, yes So it's working like in a manner when you start containers inside the Kubernetes CLI Raven is performing crude operations on OpenStack Neutron server creating ports, subnet, or subnet are already created inside Neutron and when the pod is placed on the node the Qtir CNI driver is asking for the connectivity details For example, when you're starting pod Raven will ask for Neutron server to create the port like it's for the VMs like Nova is doing Then return back to Kubernetes API with annotations details Inside annotations there are details how to connect and what's the port created for this new started pod and then on the worker node CNI driver will ask for the annotations from the Kubernetes API server and then connect the pod with VATH pair with the obvious integration bridge As you can see on the screen there's also VMs connected to the same obvious and in this case you can have the L2 connectivity between VMs and containers So the worker layout is networking services on worker node is that you have containerized Qubelet CNI driver, containerized Nova compute and Neutron L2 agent and obvious is also containerized here and for the connectivity like I said the VATH pair is connected also with Linux bridge and the courier driver is doing that setting up the port inside the integration bridge and connecting it to the pod It's similar what Nova compute does which is setting up the tap interface and plumbing it into the integration bridge and with these two ports Neutron L2 agent is responsible for wiring it up with port details from Neutron server What should be the local VLAN if it's the same network on that already existing or on any other and also connecting it to the tunneling bridge So on the network node you can see that it can be done twice So when you install courier it can create default subnet for workload and service service IP and it can be like I've seen that in current in master branch you can specify that don't create new one you can use already existing ones in Neutron but you need to map it before you can start Raven because the IP API watcher is asking for subnet and workload subnet for workload and service cluster IP creation and as well you have this Neutron L3 agent, DHCP agent, L2 agent metadata and so on obvious service containerized So this is kind of when you start up and your courier and Kubernetes plus Neutron you will get this kind of layout when you will start new pod with for example two replicas it will be spawned on network, sorry on worker nodes like I've talked on previous slide and there will be load balancing you can use load balancing instead of Kube Proxy for creating the HA Proxy instance and expose the service like it's designed in Kubernetes networking scheme so every pod can access each other using this workload subnet and as well using the service cluster IP subnet and you will be having access to both of them and this 10.10.223 and it will roundtrob in the request for the pod or any other load balancing mechanism you will set up inside the HA Proxy for example and when you will start VM you can use the same workload subnet so it can be spawned using the same subnet and you can gain the connectivity between the pods and VMs like on L2 level and as well using the router VM can also reach the service cluster IP subnet and the load balancing mechanism and so when you want to expose your service to the internet you need to connect it with floating IPs that you have defined and have the pool for your usage so I have done it manually but I'm not sure how about current status in master branch for courier because you can use the same API that you are creating the floating IPs for VMs and then connect it into the HA Proxy load balancing port so you can define that in some kind in external subnet this load balancing 4.3 IP address is connected to the load balancer and then you can gain access to these two pods underneath the load balancer so it's kind of double routing the mechanism because you will access the load balancer first it will redirect you to the pod currently to serve your request and so there's more the model mapping as I said it's important so the namespace is connected is mapped to network in OpenStack from Kubernetes namespace to OpenStack network and the cluster subnet is mapped to subnet pool because in Kubernetes use case you can have the slash 24 or any other subnet connected to each compute node so you can map it to this cluster subnet and service cluster IP range like I've talked is HA Proxy so it's virtual IP address inside this load balancing subnet external subnet is mapped to floating IPs and port is of course port so it holds the default information about IP and MAC address and the service itself is mapped to load balancing mechanism inside OpenStack terminology so what's the pros and cons about using Courier it's like it's big tent project it has good community and it's have simple architecture and taking advantage of the CNI pluggable architecture in Kubernetes also Courier can support other technologies, networking technologies it's not only of yes but it can take advantage of Open or any other so it's using Newton API so it's what we wanted to achieve single API for all the workloads management system for the VMs and containers so the VMs and containers of course gets its L2 connectivity and you can get the security groups applied for both VMs and containers so I forgot to mention about Raven Raven is also creating security group inside the Newton API and Newton security groups so you can apply the same security groups for both containers and VMs and what cons that I name it next steps it's that it's not ready yet like it wasn't released maybe it will be in short period of time some time ago it was working only on Python 3 but I've seen that there's already merge changes that it port the Asyncio library into EventNet into Python 2.7 and so like I presented the Kube proxy have to be replaced with some other load balancing mechanism and for better scalability and high throughput workloads and what's the bigger disadvantage of using Courier it can be used only for overlay networks so for the workloads and when you will install Courier you need to provide all the Kubernetes and OpenStack containers working in some kind of networking scheme so it's like I've seen that some team has used Flannel for setting up the OpenStack on top of Kubernetes and then we had added Courier or you can add Courier for the workload mechanism to reuse the networking created already in OpenStack yes so the last point is that I've seen that it should be already addressed on master branch in Courier that you can reuse the existing networking created inside Neutron to create pods on Kubernetes site so this is kind of resources that we used for these presentations and I guess legal notice Q&A I can leave the resources page so do you have any questions? I've got a microphone here so if anyone would like to ask a question sorry you need to turn it on I forgot hello so couldn't you use Courier with Calico? Calico has a Neutron yeah yeah so it's also possible because of the Courier we'll be using it's already using OSVF to connect the interfaces it can be plug-inable to use it with other networking drivers not only OVS it has the plug-in for Calico as well right? actually Calico could be used as a Neutron backend but if you use Courier to connect container workloads using Calico as a backend it may be a chicken and a proble and you may prefer to use Calico natively to the containers in CNI and using networking Calico plug-in for Neutron ML2 to connect UVMs to Calico thank you you can use also Calico for the underlying networking so you can set up Calico as networking for OpenStack services and then use Courier for workloads there so you can have also that kind of mix any other questions? if not then thanks thank you very much thank you for attending