 Hello, everyone My name is Michał Dulkow. I'm working at Red Hat Excuse me I'm working on Red Hat and I'm a core reviewer in the Kuru Kubernetes project This is Daniel Mayado He's the PTL of the project working on the same team as us me So we are here to tell you how we've made Kubernetes application from the Kuru Kubernetes, which is an open stack service So first of all, we walk you through in a pretty detailed way To tell you about what Kuru Kubernetes is so you can understand the Requirements it has as an application And then we will walk you through those requirements and how we solve them using the Kubernetes primitives And this is the link to the slides in case you want to see them on your laptops So Sorry, I will hand over to Daniel who will tell you about what Kuru Kubernetes is Okay, thanks Michał. So first let me introduce you first of all our awesome pet. This is the bloody booth So there's our some funny friends with the name Spanish But we also first started as you might notice could it and means Oriani insects could it And this is because mainly we are translating Container stuff from one part to another. So this is the this guy on your right was Oriani our first pet But after the standardization with foundation we migrated to that So Kuru Kubernetes is of course an open stack project We have several repos and if you attended our onboarding session, we went through the details there In case you're interested just feel free to catch us offline and we'll explain to you So we do the main development on the career dash Kubernetes repo and also we package that and it's a Package there on a Python package index We have quite a healthy community. So we have Developers and core reviewers from companies such as of course red hat then away Intel Samsung and a few others and also independent contributors So we went out of the experimental phase and opens a quiz And we started with our version zero for three and so far we have been following the cycle with intermediary this model But we are thinking about changing that in the future So basically our goals is to We'll go through what the CIS but we want to Provide a container that were into this plugin that basically would use opposite neutral open stack neutron for Kubernetes, but then twerking and also we would like to have a service equivalent to keep proxy using open stack native spread such as Octavia or In the past we are so we're using a neutron Alba source and see which is it is somehow maintained So let me tell you about what CIS as I was telling you before Well, see I is a of course contain an auto interface any separate meant to provide you with a networking plugin for Kubernetes it was originally a print governed by the CNCF, which is a top native computer foundation Above all though the stuff that it's noted there and which I won't be going line by line because I think it's a little bit excessive but mainly I look like you To keep this in mind because this would be important for the second part in it's that the CNI plugins are in fact Executables that should be run with parameter in MF bars and we help would be explain you afterwards What's why this is important and how did we get to that science when we were making containers out of our private? So what's the what are the motivations for career? First of all, we wanted to of course get let's say the power from a percent neutron for your communities cluster But with let's say one cabinet. So here you can see we have the Sun So you'll recall what was there the Sun could get a lot of gravity Then we have a neutron star, which of course we are not speaking about the neutron network in Then there's a black hole and there's this neutron. So let's say We would like to get much more speed off of it Basically, we want to use some of the new transnational ideas because neutrons are such a huge spread to get Them for product networking So what we want we want to provide and the connection between opposite vians and coordinated spots and containers So pot VM VM to pod and VMs to services and doing grass controllers at some points So also Let's see and that's the main motivation for the project so if you don't use could it you can of course have networking in between two netties and Open stack, but you will have several network overlays and you will have double network as selection So let's imagine that we are using final as your default a Network solution for To for communities you see that you could of course communicate Your compute knows your VMs on one side To the be another one running containers and so on But if you see you would have of course the neutron overlay and the final overlay And that's not so really effective. So we basically wanted to Squash it up and have one single network solution that would work for everything That's why we basically are Getting and connecting neutron ports directly to to them the neutron ports to the pots directly So let me walk you quickly I'll try not to spend that much more time. Yes to save time for me help art So I'd like yet to walk you through the components that we are mainly working and crew So we have the one controller, which is called a quick controller This is responsible for the open stack operations. So Basically mainly and it is a high level and we'll be seeing this a little bit more in detail now But we have two kind of components inquiry, which are the drivers and the handlers. So the handlers basically they watch for Kubernetes Events such as hey, I created a pod. Hey, I need the service and then we have the driver and the driver They do perform crude operations and double popping stack that mainly those are done on the controller and We in the control also passes Information about open-stack resources and we do you do annotate that on on the Kubernetes annotations field Then we have the career demon. This is relying on every Kubernetes nodes and it watches for new pods and Basically, do they wear operations? The request is by the curious and I recall the CNI was the container of interface Then we have curious and I this is our specific CNI plugin And of course, it's succeeded with that the container went never came to face and this one But it passes CNI requests to the career demon and let me show you a little bit of an overview. So Okay, this is stating open shift API server, but feel free to read that as Kubernetes because it will be the same for this presentation So if you see there on the top, we have the query controller and this guy is watching for Kubernetes events It will perform crude actions. For instance, let's say we do create the pot and Even better, let's say that we create a deployment We are we will be creating a deployment with just one pot and this will be okay But afterwards, of course, we would like to grow our deployment and we will a scale that To a second one. So that would be a service in Kubernetes So in Kubernetes, we would have Kubernetes load balancer, which would do the work, but using career We would really have to be up there on the left side and I'm gonna be there's any activity over here. So thanks guys So one little balancer would be created and then all the network would be load policy in between Isn't that? So also, I want to notice that we have like two different use cases where you could use career You could have like a side-by-side one Kubernetes cluster and your open stack installation Then on this case, we are showcasing and more interesting use cases that were actually like Kubernetes worker nodes are running on top of a VM So in this case, you see that we have the accuracy and I which it would basically perform all the where operations It will you well, we are using transports, but that was also covered in another session. So I won't enter into that You'll be having there Basically, yeah, I just wanted to highlight all the components there and even if it this is not the Goal of this chat on this session, but I'll be More than well willing to if the answer is any question after me help part will be more than willing to cover this So with this I'll be handing over to me help and he will explain you how to Create this into containers. Thank you. How so you have it? You should have now the basic image of How how this this application works? So I'll show you what requirements are Are there and how we translate that into the Kubernetes resources? So first of all why why do we want to write the Kubernetes manifests? write the docker files and fight the translation of the of the requirements We're basically it's about the installation distribution and manage beauty but last but not least Well, most of the C&I plugins offer that so we simply want to use the same Same way of distributing the software So let's go through the requirements of the cooler controller. So first of all We've just recently introduced the active passive HA feature, but before that we need just We need to make sure that there is only one cool controller instance So in open stack that would be solved by using pacemaker that will kill the server that is running cool controller in case it's out of sync or something and In in Kubernetes, this is solved by deployments. So we simply created a deployment Primitive for the for the cooler controller with the fixed number of replicas as it's listed here So this this primitive is able to keep the number of pods which are kind of application instances in the in the constant constant number of replicas I Have told you about the active pacify chase so that requires Some kind of leader election feature So if you would be running as a normal open stack service that would be that would mean using the Oslo twos library With some kind of back-end like zookeeper or at CD But if we are running on Kubernetes, we already have at CD running. So why I don't we use it so in Kubernetes the the pattern that is used is using the Sidecar sidecar containers. So in this case the pod the cooler controller pod will have two containers So one of the containers which definition is simplified definition is listed here is simply doing the leader election and on the HTTP Others it is allowing the cooler controller to query for the For the current leader. So the cooler controller will know if it's leader or not So if it can do the all the operations or not so In a detailed way this works as simply So there's there's a end point which is kind of a Kubernetes resource That is annotated with the current leader name and all of the instances of this leader Elector sidecar container are simply fighting over that annotation and at CD is keeping them from Keeping the exclusion that it won't have we don't have two updates Having at the same time Another requirement is the access to the Kubernetes API. So obviously cool controller needs to work the for the events on the Kubernetes API So it's it's done by service accounts. So simply We create a cluster role which lists all the API that we should have access to and Then we create a service account bind the cluster role with the with the service account with this cluster role binding kind of resource and then in the deployments Pod spec we add the service account name this will make The credentials to the Kubernetes API to appear in a well-defined place in the in the pod file system And that way we can access the Kubernetes API from the cooler controller So we should we should somehow provide the configuration options. So Normally in opens stack services simply deployment tools are handling that in case of Kubernetes We have a resource called config map And so simply we put the configuration in the config map and in the in the pod Specification we add the volume and we mount it to where we want it to be and that way That way we have the same configuration shared between the pods, which is which is quite nice So cool controller needs to create the open stack resources based on the events in the Kubernetes API So it needs access to the open stack API so if the access is Done through HTTPS then we need some kind of Certificates a certificate or all the other keys to access it So we implemented that in the in Kubernetes through the secrets and config map so We put the credentials in the config map But all the all the keys and certificates are put on secret that is mounted again in a in a correct place on on the Pod file system There's one one issue that we've Faced that okay, so cooler cooler Kubernetes is the CNI plug-in right so it's needed to be So it's supposed to provide networking for the pods. So what happens if we want to To make to make this cool Kubernetes run on the open stack service on sorry on the Kubernetes it is supposed to provide the networking for well the problem is that Something needs to provide the networking for that cool Kubernetes Pod while it's created so that The solution that we've refused is simply using the host networking even though it's not required for cool controller But just to make sure that it bypasses the CNI It's using the host networking so It can be created even though the CNI is not yet configured and Also, it's useful to add this priority class name. So this blocks Kubernetes from evicting the pod that that pod in case of of some In case the resources are low on the on the node So this this marks the pod that it's critical for the for the Kubernetes to function So it should not be deleted in any case Yeah, so we have this advantage here, so as we just we are using host networking So if there is something running on the Kubernetes node That is using the same ports as the cooler Kubernetes. There will be conflicts. So this is kind of problematic So now we are going through the cooler demon service. So this it's a bit different So in this case, we wanted to be running on every Kubernetes node. So there's a different Kubernetes primitive that Is solving this and it's called the demon set So this is very similar to the deployment definition Just we don't specify the replicas number because demon set is Making sure that this service will run on every of the of the nodes This is very similar to the cooler controller case So we need to have access to Kubernetes API that service accounts. We need to have the Configuration pass somehow that's config maps. This is the same as as I explained earlier But also cool demon is doing all the networking operations. So It's creating the pop the virtual ports on the on the host. It's Plugging them into the and namespaces of the Pods being created on that node. So it needs To have right access to that to all that kernel networking system So we did that through various various stuff So first of all it needs host networking because it needs to see the same networks as as the node as the kubelet does Then we need to have access to the proc Folder from the from the host So we simply mount it into the container But obviously you cannot mount proc folder into the proc folder of the container So we mounted somewhere else So you need to make sure that your your the libraries you use for the manipulating the core the kernel Networking are supporting that the proc folder will be in different place Then it needs to be privileged container obviously because it needs to have root access to the networking and as we Are using open v-switch it also needs to have the open v-switch Var run open v-switch folder mounted into the the container so we can manipulate the open v-switch database We also need to pass some some information from the pod specification into into the pods so This is the solution. So we pass we are passing it through the environment and variables So for example, we need to pass the node name So the name of the node the core the that concrete core demon is running on It's running on currently Because it needs to watch for the Kubernetes events that are happening on that node So it needs to know on which node is this it's running on so we pass this We've also passed the name of the pod because it's a bit random And I'll show you later on why why we are doing this the second Or we are adding the second environmental variable So last but definitely not least is the problem of injecting the Courier CNI executable and CNI configuration. So When you run the courier courier demon on all of your courier demon pod all of your Kubernetes nodes it will also Do all the installation and configuration tasks on your That's that's needed to make the CNI being configured to to To use courier Kubernetes. So this is very simple We just mount the hcd CNI netd into the container and copy the configuration there. So it will it will Appear on the on the host. So that one that one easy And so we also need to mount opt CNI beans. So this is the place where the CNI plugins executables are supposed to be So we just copy our executable there and we are fine But the problem is that courier Kubernetes is the Python application. So it's not really a binary It has a ton of dependencies and it needs the Python or Python interpreter So it's a little bit problematic to we cannot really just copy the application there So we need to find a solution for that So just stepping one Doing one step back. Why do we want to? To make courier Kubernetes pod to install itself on the on the node Well, simply other CNI plugins do that. So we use the the same pattern We want to use the same pattern as other are using And as I've said there are few few challenges that we we have So at first we've decided to because we've tried four approaches until we got it right. So it was kind of difficult so At first we've tried to use the pie installer to compile the the Python app with the along with the Python interpreter into a single binary and inject that binary This is a pretty cool logo that my installer has it's like from from ten years ago, I guess So there are there are several issues with that approach. So first of all It comp it was the build process of the container was pretty complicated because we needed to have intermediate container with the pie installer installation we copied code code there then we've Compiled the code into the binary using pie installer and then we needed we were able to create the second container That was actually the container with the courier demon We have some some issues that with the HTTP connection termination that we haven't really Solved or looked into too closely, but our QA was freaking out that There are some some false positive errors error logs in the in the courier demon logs So that was another another issue But one day also the guys from the OSV library started to added some checks that were depending on the module paths and find stellar changes that's module module paths. So we suddenly stopped working We asked the OSV guys to to remove the checks and they've told us to fix our application. Well, so we did So another another idea with explored was using the Python virtual environment. So what Python virtual environment is consisting of the Packages dependencies and the Python binary. So it should be it should be fine, right? So turned out that We weren't able to to make it work at all so first of all the problem is that we can have potential architecture mismatches between the CPU architecture on the on the host we are copying the VM in and the architecture that the Python binary was compiled with and also although there is the Dash-dash record table option when you are creating a virtual environment It does not work simply. I've tried numerous version of the VNF binary I've tried different ways of creating the virtual and It's simply they are not portable that you cannot really move the The virtual environment from one place to another so we need to to find another approach and This time we we have a notion that okay, this this might be the way that you should do it so basically you have the on each of the Kubernetes hosts you have the You have the core demon Container running Inside the container the core where that is is installed So it has the all the dependencies it needs to have the correct Python binary and it It has the course C&I executable, right? So why don't we simply execute the Use that use the docker exec to execute the commands from the Execute that comment in the container and pass the s to the in and I set the out to to the C&I to the Kublet that is executing the C&I So, okay, we thought that this is a great idea. So this is a piece of the code that That gets all the C&I Environment variables packs them into the parameters on the docker exec and runs the The core C&I this seems fine. So we can get the container ID by Quaring the Kubernetes API. It's in the status field We cannot we cannot really pass it like we've did with the node name because it's in the status field and not the Specification so it's not allowed to pass it through the same way as we've did with the with the node name Okay, come on. We can we can query the Kubernetes API Get the container ID and put it into the the script that will inject into the host the Kubernetes host This should be fine, right? So there are some issues with that first of all it's passing the environmental variables into the Into the docker exec is support only from the API docker API v 1.24 It's not a big issue because it's fairly fairly old version of docker There is some latency introduced because we are we are running stuff through docker, but it shouldn't be too bad and The main problem that we've discovered is that well Kubernetes API is simply not reliable when when querying for the for the container ID simply we had some kind of race condition that When when the port was starting Sometimes the Kubernetes API was not yet updated with the correct Container ID, so we ended up injecting a file that Was saying new what's having new as container ID and this ended up with of course Being unable to run the CNI plug-in so the pods were not getting no networks So we thought about a little bit modified way and this is the one that we are currently using so simply instead of querying the Kubernetes API to get the container ID. We are querying the docker API on the host so locally to get it so we We use we identified the pods that is running the courier demon through the Labels that are added by the Kubernetes and this this works fairly well There are of course those those two Issues that we had previously. So we need some some version of the docker API and there's some latency There is some more latency because each time the CNI plug-in is being Executed we are querying the docker API to get the docker ID of the Of the courier demon pod But it still shouldn't be too bad. It's the local API Accessed through the fire socket. So it's fine But also we need to assume that those labels that we use to identify the pod that we are looking for are the stable API This it seems that they are but they are not the Kubernetes. It's not defining. It's a stable API So one day we may wake up with stuff not working with the new version of the Kubernetes But it's it's fine for now So the conclusions well basically that's probably no No surprise for you. It's making the distribution and manage Management of the courier Kubernetes easier. That's fine. We haven't got it right at first So the problem was with we've ended up back porting So this this solution came up in Queens, but we've kept Fixing it all the way through through rocky and back porting each of the of the ways we we tried to to make this work so It's it was it was a fairly big back ports So that's not ideal, but well we needed to do that It took us a few time. Yeah, totally. Yeah, yeah, especially debugging that OSV issue that was that was awful So some some of the Kubernetes features makes make Development of new of new features in crew Kubernetes easier. So for example this ha staff that I've mentioned It is only working when you are running on crew on Kubernetes Because we simply use the Kubernetes way of doing leader election and haven't had implemented our own so It makes developers life a bit a bit easier And there is some some new Kubernetes features that are coming in that might be might be useful and probably some of you Already expect that I will show you the operators And so simplified the operators are the Kubernetes applications written to manage your your Kubernetes application So they are meant mostly for the stateful applications And crew Kubernetes tries not to be stateful because it only saves its state into the through the Kubernetes API and That goes into the HCD But it could be pretty useful for example for for ha mode when When you when you scale the core controller from one instance into Let's say five replicas. You need to enable the ha mode. So that's a configuration change You should add this leader electric container. So that's a change of the deployment definition So with operators we could automate that which is which is pretty great But we are not yet looking at that too closely So no ETA when we only will support Kubernetes operators running cooler Kubernetes through through the operators Okay, so this is the slide that I Show show often. So if you if you are interested in the project This list the documentation the IRC channel where we where we are in So we are usually quite approachable. So feel free to join us. We are running like weekly meetings They're at 2 p.m. You to see in case you have any comments want to share Have some feedback about your usage and so forth or whatever. Yes, please feel free to join them and tell us so It might be useful that most of the developers are from from Europe. So we are available to the to the Europe Europe working hours Okay, and with that Here are the links to the slides so You might you might find it useful. Also, there are our nicknames on the IRC our emails So you want to give us some thanks or tell us that we suck or whatever different to the cell So I guess now it's the time for questions Yeah, please use the microphone So I understand that you guys are using the courier controller, right and at the end of the day If I just boil it down basically like you have Making sure that all the neutron resources that you can come up with that and we actually translate into a CNI plug Yeah, that's correct. So then why would you actually have to have another separate controller because open-store already comes with a controller so you could have already used that to To actually make your courier part of that instead of running as a container all together I'm not sure if I get this right. No, you you you are running the curier controller as a separate pod, right? If I understand that's right. Yeah, so my intent was this than that if on a Who when it is if you want to run the kubernetes Basically, you're trying to use an open-stack component into the kubernetes world and using it as a CNI component, right? That's so I was saying that why would we not keep the? Do the form factor, which is basically an open-stack form factor as is for the neutron to continue to be there and then interface into the kubernetes So I've explained the motivations about all the Kubernetesation works like the main motivation was simply all the other CNI plug is are deployed like that and There are some kubernetes installation tools that are Assuming that That you will that the court the CNI plug-in will be installable on kubernetes and Not not sure that what I mean is they intend to be the first class citizen in the kubernetes world is that right? That's right to be first class of the of the CNI plug-ins tribe, let's thank Also, it might be useful to say that we have this two types of service So one is cool controller, which is doing the open-stack operations and the crew demon So we are we are not doing the operations from the crew demon because we assumed that Access to the open-stack API is kind of Scary resource so so we keep only one service being able to access it And that's the crew controller if you want you can run it on the same notes as As the crew deep that's cool control and you can run along with the crew demons, right? Actually, if you just go and deploy over depth sector playing, that's how it's done. It's all another one. No quirks Yeah, I think about it tells us how it's some kind of you know security measure Okay, if there is no more question questions, I guess we can we can close the presentation. Thank you very much