 Now, you might be wondering why I've come here to talk about OpenStack, but it's going to be about the idea of running Cilium somewhere else than in Kubernetes, because we all know this picture, right? We all love it, but life is not always like that. So, standard Cilium usage is in Kubernetes, right? Our story is that we don't have just Kubernetes clusters. We have a few different clusters. I'll tell you in a minute. So, the problem is running Cilium outside of Kubernetes. The features or the possibilities you have in the OpenStars version is Cilium external watercloth. What that allows you is to import. It can be a VM. It can be a bare metal server. You can import it into your running Kubernetes cluster. You will need to just get the hostname, get the IP addresses. You can add labels to it, and you will just put that into a YAML, into the Kubernetes object, put that into your cluster, and the cluster will know the identity of that external workload. So, there will be Cilium agents running on the VM, for example. It will allocate an identity to the whole VM, and that identity will be shared in the cluster. What you will see is that you just need to use at CD, so you need connection to the at CD. You can't actually use the Kubernetes with the external workload. So, our story begins when our manager came to us one day and said, guys, this is all great. We can use Cilium in Kubernetes, but our users also have OpenStack. So this is an OpenStack logo. And they have their databases in OpenStack. They can't just migrate them, and they can't just run them in Kubernetes because they're MySQL databases, Postgres, and they're used to run them in OpenStack. So, can we somehow make Cilium work in OpenStack world as well? And that was a bit of a challenge. We started slowly. We discussed it with the community, with the Cilium maintainers, so thanks a lot, guys. We were able to run Cilium agents on each OpenStack node. The node is actually the machine which hosts the VMs, so there can be 10 VMs, for example, on a single node, just like in the Kubernetes world. So then, when you run Cilium agents on them, they just need, again, a connection to the at-cd. What you can do then regarding the policies, so there also was a security team that came to us and said, guys, is it possible to secure somehow the workloads, the VMs? And we are okay. We can try to find it. And there was this Cilium policy import command which you can run from inside the Cilium container. So you can import a JSON file which will have what should be allowed, what should be blocked, so just the definition of the policy. But the problem is you need to go one by one and do that on every node. I mean, you can do some scripting around that. You can perhaps have a cron job that does that for you. But what about updates and stuff like that? So we came up with the idea of what if we put Kubernetes next to the OpenStack? So what if we somehow configure these agents to connect to a Kubernetes, but apply everything on the VMs themselves? So in OpenStack, just like in Kubernetes, you have worker nodes, you have master nodes. There's the control plane, right? We are used to run at CDs on them. So it's easy, right? We just spawn the Kubernetes on those nodes. So it can be, it turned out to be a K3S with just the control plane. No worker nodes, so no workloads, but just the master nodes so that we have the Kubernetes control plane. We can apply something in there, right? So we can actually use the Kube CTL and apply the standard policies, what you would apply to your standard Kubernetes cluster. You can apply them to these, for example, K3S, and Selium agents are configured to access and to pull everything from those Kubernetes cluster nodes. So then they just apply the policies, can be the same ones as in Kubernetes case. But it's not it, right? If you have just a single cluster, it's not enough. Our users, they want cluster mesh. They want to share the identities across all of it. Which means we somehow need to expose those at CDs to the other clusters. And we've heard how the cluster mesh works. So Selium agents will pull or will synchronize all the identities, the IP addresses, from the remote at CDs and will update their IP caches and the identity maps. And then the communication can happen, right? So if I connect one server from a different cluster, I can see it's IP, I know what it is, I know the labels. So turned out this is actually possible. But I have only briefly said what it was, high-level overview. So now if we take a closer look how it works in OpenStack, there is, this is a single node, right? So there is a node, there is a VM which is running on the node. What had to happen was we had to create a tool. We call it Selium OpenStack plugin, which is similar to what Selium Docker plugin is, which is in the Selium repo. You can take a look at it. So we've created this. What it does is it needs a source of the information, right? So in case of OpenStack, it's LibVert. It's what's spawning the VM. So OpenStack control plane will tell LibVert, hey, spawn me a VM and it will do that. And what LibVert does, what allows you to do is you can hook into LibVert, you can see the events that are happening there, okay, VM is about to be spawned, VM is about to be deleted and we can work with these events. So we're watching these events and taking the information from them and just transferring them into something that Selium needs to know. So we are using an API to tell Selium agent, hey, this is a new VM. It has these labels, it has this IP address and just create all the BPI programs and attach it, please, to my tab interface, which is what's in the OpenStack world. And if you want to use host firewall, you can use that. It will just attach just like in Kubernetes to your public interface if zero in this case. And then we have BGP, so it's no different compared to the Kubernetes. We use BERT in this case. So which means there can be a few different problems if you're running something that is not standard. So I've said that we're using APIs. What Selium does, what allows you to do is to use the low-level APIs. The one we use only is the endpoint one. So what you can do is if you have something that's running on the node where the Selium agent is running, you can just tell, we are doing Unix socket, hey, create me an endpoint. If labels change, patch the endpoint, delete, whatever. So this way, you can tell it if you have this information. So again, you just need an IP address, you need labels, you need an interface name and you need to make address and the data path configuration. In our case, we don't use any IPAM related to Selium, we just use the OpenStack one. So we use the external IPAM true. We needed to let the ARP packets to go through so that the ARP resolution protocol works. And we also needed to tell Selium to install the IP route so that the traffic can actually reach the VMs. So then we just created with the endpoint create command. So this way, we have Selium running on the node and we have an endpoint, a Selium one. What you can notice is that usually you would see here, usually you would see here K8S because Selium, you are used to run Selium in Kubernetes, right? And by default, Selium will prepend Kubernetes label source to your Selium endpoint. What we had to do was patch Selium and we've created a new label source called OpenStack. So we are prepending an OpenStack prefix here, which is important to us so that our users can distinguish between if they are applying or allowing access from OpenStack or from Kubernetes. So this way they can do it. So again, that was a short intro to OpenStack. But now if you actually want to test things out, what you can, for example, our common use case is we have a client or API in Kubernetes and we have a database in OpenStack, so in a VM. So usually if you have something like default deny policies applied, you would expect that traffic is blocked, which in this case it is. But you know the labels which you want to allow. So you just create a network policy, right? So usually you do not have to specify the prefix here, so the label source. In this case we want to specify it because we are explicitly saying, okay, allow me access to the OpenStack world. So we create Selimet for policy on egress, two endpoints with labels, yeah, cluster, namespace, the app, and we prefix it with the OpenStack. And voila, things start to work. So this way you can allow access between these two worlds. But this was only a small part of our puzzle because our manager came to us again and said, guys, this is amazing, but our users, they use something else, something that has been here for a long time. Could you somehow figure out a way how to put this into the picture? And if you've been paying attention today, you have seen this picture. Does anybody know what it is? Yeah, it's Calico, exactly. So our story is that we are still running Calico in Kubernetes also in OpenStack. And we are trying to figure out a way how to put this into the picture, how to somehow connect Calico, which is like an outside world. If you communicate from this cluster, from a pod, and towards your Selim enabled clusters, either Kubernetes or OpenStack doesn't matter. This will be just like in any other outside world application. So it won't be, Selim knows nothing about IP address from Calico. So what we managed to do was create yet another tool, we call it Calico Selim Bridge. And what it does is it will take workload endpoints, which is like Selim endpoints in Calico world. So it's watching the workload endpoints in the Calico at CD. And it transfers them, converts them into something that Selim understands. So you need at CD per every Calico cluster. And this tool can migrate everything that's in the Calico at CD into your Selim at CD, which you've just spawned, for example. And then it can just watch the events again. It uses the Selim library at CD1 to do that. So it will watch the workload endpoints with the prefix slash Calico. Thanks to the Selim library, it will allocate an identity. So you need to tell it which cluster ID should be used. We've heard it today that you need, for cluster mesh to work properly, you need cluster ID per every cluster. So for this work, you need to tell it, okay, this is the cluster ID that this Calico cluster will use. It will allocate the identity for every workload endpoint and save that into the at CD. And then that's the first part, that's the identity. And then you need the IP address and the connection with the identity. So it will also save the IP address as the IP identity pair into the at CD. So this way, in the end, you have in the at CD, you have the identity by value and by the number in the at CD, in the target one. And you have the IP address with the connection to the identity. So these informations are needed for the cluster mesh, cluster mesh to work properly. So now we've successfully managed to create cluster mesh across all these different environments. The problem was our users, right? We need to somehow teach them how to use all these functionality because they are not used to create network policies or, or silly network policies. Now they have, they have option to, to create that in Kubernetes, which is native. They can use Hubble, which will help them. They can use the silly network policy editor. But if you're combining all these worlds, they also need to understand where are you applying this? So with Calico in the picture, what you can do is just like in OpenStack, we've added yet another label source Calico, which allows you to just specify that anywhere you want. So if in case you have, you are slowly just migrating from the, from the Calico clusters and we are trying to convince them, guys, with this, it, it's easier, right? You can just small, in a small, small portions, just one by one, one component by one, convert them to, to the Cilium Kubernetes. So if they have client in one, in Calico cluster, they have the API in the, in the new Cilium Kubernetes cluster and they have the database in OpenStack. With this, they can just allow and block access from wherever they want. It just works, it just works. But the problem is they need to understand where is the policy actually applied and if it works everywhere, they are used to, they're used to use. So in 1.15.3.1, this works. So you have a service of type notebook, right? So policies are applied there. The ones I just shown, in case of service of type bolt balancer in DSR mode, you actually see the original client source IP and it can be applied. So the policies can be applied also in this case. If the traffic reaches the node, for example, that is hosting the, the backends. But if we have ingress in the picture, the problem is, it's a proxy, right? So usually it terminates connections, HTTPS connections. So if you use ingress, usually what you see on the back end is the source IP of the proxy, not of the original client that initiated the connection, but you see the proxy. So what you, the only thing you can, you can do is to allow access from the proxy. But you cannot, you cannot tell it, allow only this specific client. You will be allowed to access the ingress. You can use, I mean, maybe in a, in a cloud world, I don't have those experiences. You can tell it, okay, allow only, only this subnet perhaps. But then last week came the 1.15.3.2 and I want to give shout outs to, to Jarno, which was the contributor, contributor to it. There is a possibility to specify CNPs targeting the ingress itself. So now with 1.15.3.2 you can tell, allow only this specific client to access my ingress. And we are good to go. So with, with that, the mission is accomplished. And hopefully our users will be, it will be easier for them to migrate to, to Cilium. So if this is interesting to you in any way, if you think you would, you could use that, please reach out to me. We have, we have yet to open source both tools. We have not done that because we are just wondering if it makes sense to you. So with that, thank you so much. Do we have any questions? So your, your Calico plugin and your OpenStack plugins, how do they reconcile state if they get out of sync? How do they reconcile state if what? Sorry? If they get out of sync? So let's say like your, your OpenStack cluster, if, if an event is missed by the plugin. If it's missed. Yeah. Yeah. There's a, yeah, just like you said, there's a reconcile which periodically synchronizes the state if it did all it should. So the plugin will, will get the current state from the libvert and it will just try to send that to, to Silly Magent to somehow synchronize if, if it missed something or not. It's not perfect, but yeah, it works. I was wondering what, I guess, lower level of networking model you're using with OpenStack. Was it using OVS? No, sorry. Yeah. I forgot to mention it. It's, it was, it has been based on Calico the whole time. So there is a BGP and L3 networking. So no open V-switch, unfortunately. So maybe it could be a start. The other thing is how, how would you handle IP collisions between like private networks within OpenStack and also inside the Kubernetes networks? Or did you just restrict the IP schemes? Yes. We have a specific subnets for, for OpenStack and different ones. So they are unique. If we want cluster mesh to work correctly, it has to be unique, right? So we are connecting all of them and yeah, have separate subnets for. Okay. Thank you. Just one question about the networking model. Are you using just native rounding mode? Yes. Exactly. And to, to no tunnels or anything? Nope. Okay.