 So welcome everyone, we are finally restarting the regular sessions for the research user group and this was a topic that we had already for a while on Cilium and EBBF that had been suggested before and we got Raphael from Isovalent that was nice enough to join us today. So we'll have a presentation so usually the format is like half an hour and then we we leave plenty of time for questions but I guess if people have questions during the presentation that should be also okay so yeah thanks again Raphael. Thanks so I guess I'll start by sharing my screen, is that fine? Yeah okay so yeah let's talk about Cilium, I put Cilium in France, Cilium in general, Cilium EBBF and the tools around Cilium. So my name is Raphael Bensom, I'm a Solutions Architect at Isovalent based in Switzerland and so as part of my job at Solutions Architect that you customer support on Cilium Hubble Tetragon and I've been involved with with communities in the CNCF for several years I actually co-organized the CNCF meetup in the Normandy part of Switzerland. So the different subjects we can touch on today are Cilium EBBF in general, then different features of Cilium, networking cluster mesh, security observability, a bit on service mesh and I leave a little bit on Tetragon at the end which is not extremely related to Cilium but we usually ship it together at least in our enterprise solution. So first Cilium and EBBF quick introduction so I'm part of Isovalent, Isovalent is the main contributor to Cilium and Cilium is part of the CNCF, it's an incubating project at the moment and based usually on EBBF, EBBF being a technology that has been in the Linux kernel for years and it's not actually being ported to other kernels, Microsoft is doing somewhere to port it to Windows for example. So what is Cilium? These days with Cilium it's first and foremost the CNI it actually started before Kubernetes was widespread or even announced I think. So it provides the pod to pod, pod to service to pod communication, even node to node communication but it can also implement Kubernetes services or a replacement for Qproxy as well as extra features such as multi cluster or VM gateway which is an extension of the multi cluster feature. On top of this it implements network policies using either the standard Kubernetes network policies resource type or specific network policies, TRDs provided by Cilium for advanced features. It is identity-based and this is something very important. There's a duplication of identities that happen in Cilium both for performance and improves the visibility, the observability layer. It supports encryption, pod to pod encryption using IPsec or Guard and there's also observability mainly linked to the Hubble confidence providing metrics, providing flow visibility and service dependency in the form of a service map. All of this is based on EVPF and extending the Linux kernel using EVPF. So what is EVPF? EVPF is a technology that allows to dynamically extend the features of the kernel. So currently in our case the Linux kernel, the future in my extent to other OSes and so the idea is that you can have many programs that are typically injected into the kernel as bytecode. So BPF bytecode usually written in some form some subset of C like the example here that is compiled into bytecode and then injected into the kernel and then the kernel will verify that this program is acceptable. So there is a strict verifier that verifies for both security and stability of that program and then once the program is accepted into the kernel it will be compiled into machine code so it's as performant as the kernel itself and you can attach it to different events in the kernel which is it makes a lot of sense given that the kernel itself is essentially an even driven program. So for example without EVPF a process would call an execv syscall to start a new process and that syscall would be scheduled with EVPF you can actually capture that syscall and decide to observe it or even to act on the syscall itself even prevent it. So the example here for example is an example of observability in the kernel. There's several fields of application that are quite natural for EVPF. Observability is obviously one of them because you get to observe anything that happens in the kernel in form of syscalls or kprobes, uprobes, lots of different things or network events. Another field of application is security obviously if you can act directly on the syscalls even block them that makes a lot of sense and the last one is networking. I said the last one although in the case of the ceiling project this is where the project started actually with networking. It's a little bit less obvious but with EVPF you can also bypass some native networking stacks in the kernel and actually reimplement them. So different ways in which EVPF programs can act on events in the kernel. You can typically add yourself attach yourself to events such as a file being read or the IO reads directly on the disk and do some observability also action on this when it comes to networking. You can attach to connect events or to the send packet event or you could even implement a TCP return submission by attaching yourself to TCPIP events. You can attach yourself to kprobes, uprobes, system calls, trace points, sockets and so on and there's even ways actually to interact directly with the hardware using xdp for example so express data pass which allows to program hardware network devices to do load balancing for example so you could capture packets arriving on an interface in the kernel and redirect them directly to a network card for load balancing for hardware load balancing which can give amazing performance compared to what the kernel itself can do. So selium is used by a lot of different companies this is just an extract from users.md in the repository in the selium repository there's lots of names of organizations that use selium and what they do with it. So at the moment when we talk about selium there's essentially three products that are part of the selium project and that are in this scope part of the cncf there's selium itself so the cni and networking solution this hubble which is the observability component and tetragone which is the most recently open source component that is responsible for runtime security as well as security observability. So the selium networking component the cni itself is responsible for the networking side of things ipv4 ipv6 integration into cloud providers or bgp if you're doing on-premise typically overlay so these two options either direct routing using bgp or overlay using vxline or geneve srv6 these special features such as egress gateway and multicluster the possibility of doing mats 4664 and the possibility of applying network policies so l3 l4 that's kind of the standard all the way to layer 7 using envoy as a proxy and filtering on dns entries either specific dns entries or with wildcards encryption i mentioned this a bit before and load balancing which can be done actually even outside of kubernetes so either kubernetes load balancing as a replacement for qproxy or even outside of kubernetes as a standalone load balancer using selium as a container and docker for example and on top of this because we see all the traffic going through we can actually add up a lot of observability so observe traffic in the form of flows but we also get a lot of metrics from all these components the selium agent itself the selium operator hubble the envoy proxy tetragone and so on i'll talk a bit about tetragone here tetragone allows to plug to a lot of different kernel events and actually observe what's happening there you can get a lot of observability flows from there which can be exported to the same of your choice same for hubble so essentially you get a flow of json events that you can export and process in your tool of choice or you can use the tools that are provided and typically in the enterprise solution of selium provided by isovalent we have ways of correlating this event and then this is the service mesh layer that is essentially using features that we already have in selium and that people expect in a service match such as encryption such as cluster mesh such as observability and adding to it the features that people would expect as well in service matches chas and interest controller authentication traffic management and there's quite a bit of work that is still done on this layer and will come in selium 113 and and following so next next release and following so let's dive a bit into the networking side of things typically on the Kubernetes there is no default network layer there's only a standard which is cni and so here selium can replace both the cni layer as well as qproxy so the cni will be pod to pod uh enter node enter node communication uh and replacing qproxy will actually implement the services which are usually implemented using ip tables or ipvs and qproxy so we can have selium take care of all of this and typically instead of ip tables on every node selium will inject ebpf programs and maps to uh to implement the same features except they there might be it might be a little bit more featureful because ebpf programs are have more possibilities than just ip table rules so there is sorry i'll get back to this there is an agent a selium agent running on h node and this agent is responsible for injecting ebpf programs and ebpf maps that are used either to configure this programs or to retrieve information for this from this programs typically for observability for example so if we talk about services and the possibility to replace qproxy uh typically um on covenetus when you have a service you have a virtual ip that uh works as an lfield for load balancer to pod in the back endpoints and this is typically implemented using ip tables in qproxy or ipvs and with ip tables where you get is essentially a system of seed where you will go through all the rules before you find a rule that is uh proper for you that applies in your case so typically if you have the service with three uh uh pods and in the back uh you'll have three rule and three rules and the first one will say uh redirect to this pod to this ip uh and 33 percent of the cases and then redirect to the second ip in 50 percent of the cases and then redirect to this ip in 100 percent of the of the remaining cases so you see this is a very um an approach that is based on filtering and when you get to hundreds or thousands of services it can get really slow to get to the rule that you need to apply uh let alone just applying the the rules whenever there's a new pod or whenever there's a new service or whenever there's a modification in the network policy the whole stack the whole um list of ip tables need to be rewritten whereas with an evpf-based approach we can have um hash tables that will link directly the identity instead of the ip address the identity of a pod based on the set of labels that are known from Kubernetes by the evpf program and based on this identity you can link directly to the way it should be routed to another identity or if this um this traffic should be allowed or not so typically both routing and network policies can be implemented in a much more performance and scalable way um one of the examples of features that that are provided in selium that is interesting is uh the possibility of having an ingress gateway to access um a workload typically um database for example outside of the Kubernetes cluster and so we have a crd that's called egress network policy selium egress net net policy actually that allows you to target some pods in the Kubernetes cluster and say when coming from the spots and accessing this sider outside of the Kubernetes cluster uh i want to go through this specific ip or this specific node of the cluster and this allows the the application outside of the cluster to know uh to to recognize the ip that it's coming from so typically if you have a firewall in front of this application you can filter on this ip otherwise you don't know exactly which ip would be coming out right it could be from the node itself or it could be directly from the pod if you're using direct routing but it would be really hard to identify which application is reaching out to this external um uh application outside the cluster there's actually a mode that is provided in the isovaluance human enterprise uh distribution which allows for high availability uh egress gateway so instead of having one ip to exit the cluster we have several and selium can load balance between them and failover as well in case one of the nodes uh actually is not available platform integration is very important there's some uh Kubernetes distributions that are very opinionated when it comes to networking uh typically if you take the case of amazon eks for example it needs to work in a direct routing uh approach using ips that are provided by aws for the vpc um and so uh selium has a way to integrate to connect to uh these different clouds in order to get the proper ips in order to integrate with uh rules as well like security groups or other metadata from the cloud so typically we have a component called the selium operator and while the agent is a demon set running on every node in the cluster the operator is a deployment that has the credentials to connect to the cloud um and it will allow typically to uh play a role in the ipam for uh selium uh getting the the sets of ips that are available to assign to the pods in a direct routine uh environments such as eks for example but it would be similar for gke or other approaches uh this operator also has a role to do uh some um garbage collecting in the cluster in the case for example one node is removed obviously the agent on this node is not there anymore to remove stuff from the uh from the api seven so the operator can do this there's a possibility to cni chaining so using one cni as the base layer and then deploying selium on top some people want to do this it's more and more rare i would say um this difficult case is because they don't want to lose possible uh support from the cloud provider by using something else fortunately selium is getting more and more uh common and supported by cloud providers so it's not necessary to use cni chaining anymore in a lot of cases that's for the networking part um one specific feature that is interesting is the possibility of meshing different clusters at l3 l4 so typically we can have clusters that have different uh commit as this uh distributions uh different uh versions even and you want to mesh them directly uh at the l3 l4 layer uh this is possible using selium and the way this is done is that when you activate cluster mesh on a cluster selium will create a new api seven on on this cluster and this api seven will be used by the agents on the other cluster to get information read only from the first cluster so typically they'll be able to share uh service discovery information the the services in the back ends the end points for the services uh and to perform load balancing between the different clusters you can also share network policies so typically you can have network policies with the labels that uh labels that typically target one or the other cluster so you can say this back end from this cluster is allowed to talk to this front end or the other way actually this front end from this cluster is allowed to talk to this back end on the other cluster and this allows for cross cluster network policies and creation can also be extended between the two clusters as well as obviously routing otherwise it wouldn't work so typically this allows to deploy the same workloads on two clusters and make sure that if the workload is not available on one cluster it will fail over to the other cluster um another option is to have several clusters with shared services so here I have a service that is global between the three clusters but I don't actually have pods on cluster one on cluster two to implement it the pods are only on the shared services cluster and typically one use of this is using a stateful cluster where I have like a database running and I can easily scale this cluster so I'm keeping it as a as a spot in my architecture whereas I might have stateless clusters or clusters that are dedicated to stateless applications that can scale I will on different availability zones for example or even different providers and then I'll have a virtual service here that provides access to my database and this service will be made global in a cluster mesh approach so that it actually points to the stateful cluster it's also possible to integrate with a service mesh so typically having an ingress in front of each cluster and then making sure that the backend service is always accessible using either the local pods or the remote pods and in the latest version of Selenium you can actually specify if you want a local or a remote affinity so if you have a local affinity you will say preferably one access to service I want to access the local pods on my cluster if there's no such pods then go to the other cluster to fulfill the service access so that's the case with remote it looks like this it's actually a normal service definition except it has a mutations for global service and for service affinity there's actually other options available for this just to give an example let's dive into security like I said security is very important in Selenium and it's clearly based on identity and I've shown you how you know Selenium can use a concept of identities that is native to it and that associates labels and Kubernetes workloads directly to their identity this identity in the case of VxLand can actually be transmitted and encapsulated into the network packets going from one node to the other so that the identity is propagated between nodes typically for observability and network policy applications in the case of direct routing it works a little bit differently but the idea is that every time something a network flow arrives on a node there is a knowledge of this identity which allows for advanced observability and network policy enforcement based on this identity based on this we can have three layers three levels of network policy application either L3 so just connection between two pods L4 connection plus the port or protocol TCP UDP and L7 which allows to actually parse the the application networking layer and extract the the protocol the application protocol and filter on this so typically you can allow HTTP get on slash public that's an example and in this case it will actually go through an envoy proxy provided with Selenium on every agent this is an example of L7 filtering using Cassandra and you can see here that we're specifically allowing one action select on one table there's possibly as well in network policies to filter on DNS so typically when traffic is exiting the cluster at the moment we have a 2FQDN rule and you can actually use either exact DNS names or even wildcards and the way it works is that Selenium will cache the answers from tube DNS or core DNS and allow connections to IPs that are known based on the resolution that was cached there's an HA option for DNS proxy so that when the agent gets upgraded or if the agent crashes which shouldn't happen the DNS proxy continues to work and this HA option is available in the isovalent Selenium enterprise distribution so these are all the the possibilities of matching for network policies all the standard things like pod labels namespace service account service names cluster names as well when using our cluster mesh DNS names I just talked about this ciders either external ciders in this case cloud providers and this is one role of the Selenium operator is to actually resolve instance labels or subnets or security group names into ciders and in logical entities logical entities can be the host on which the pod is running the container is running typically it can be another host in the clusters that would be the remote node entity it can be the api server it can be the world it can be anything in the cluster so we have logical entities allowing to write advanced rules without caring about IP address age observability because again we see everything going through we can have great observability thanks to evpf and we have several ways of accessing it typically through a cli so a hobble cli or hobble ui and both of them use a component that we call hobble relay which gathers all the observability flows from all the nodes in the cluster and allow you to filter them and view them either on cli or uh with the ui and on top of this every one of the components Selenium agents Selenium operator uh hobble um the envoy proxy the dns proxy they all provide metrics which you can send to your favorite um to your favorite um observability platform right because they're prometheus metrics essentially so this is an example of the hobble cli so here we have some pause running and with the hobble cli you can see traffic going through your cluster so here we see the dns lookup that a pod is doing to core dns we see the reply in udp and you see the ids that are associated the htps requests and here we see we have dns visibility actually the the ips are replaced with the dns names that are known to be associated in the cache um and we can even see here when our traffic is blocked so that's a network policy being applied and here the traffic is dropped at the syn request tcp request so this is the the cli this is the ui for hobble this is the open source version of the ui the isovaliant uh selen enterprise has a slightly different uh version of it with a few more features and essentially this uses the same source of information as the cli except it actually builds a graph of dependencies between the pods between the pods the services that shows how they actually communicate so here you only see gray lines they could be red lines as well if traffic has been dropped so you can see where it's been dropped and you can actually click on the boxes to get more information and here in this case in this namespace you can see that uh htp visibility using the uh envoy proxy has been activated because you see actually which htp requests were performed on which services and here you have the flows and you can click on the flows to get more information on every one of them i'll finish the sodium site with service mesh uh this is the extension this is where we're going at the moment because we know that people are interested in this part and selen is quite low level but a lot of people are interested in features that are a bit higher level and the idea is that selen already has a lot of the features that people expect from service mesh so one thing we've added in selen 112 is the possibility of programming the envoy proxy that is already provided with the selen agent using a crd so selen 112 provides a selen envoy config crd that allows to program your the envoy proxy on every node uh using logical identity so you can see from the spots to the spots i want to apply this uh envoy configuration and one layer of control plane that we've added in sum 112 is an ingress controller that bases itself on it so essentially now you can use the selen ingress class uh if you've activated ingress controller obviously and this selen ingress class will will essentially implement the ingress by creating an envoy configuration dynamically for the ingress for a specific part in the future what we're adding what we're currently working on is support for the gateway api and support for specie for foreign monumental authentication directly implemented in selen this is planned for 113 for the next major release of selen you can already use this deal with selen as a sorry as a as a novelly on top of selen but we're also planning in the future to integrate each tool so it uses the envoy crd natively uh as a as as the implementation for for the each tool abstraction instead of injecting an envoy proxy into every pod like it does currently all of these actually also allow to get metrics and to get observability in the form of flows in isozolyn selen enterprise we actually provide a fluendi um um service which allows to easily export all the the flows that we have either from hubble or from tetrogon to your favorite cm platform we also have a component called hubble uh hotel that can turn hubble flows into open telemetry traces so you can import this as well and correlate it with the traces from your applications so the way we see this is just just the way that uh some decades ago um tcp went down from being an external library uh to being a standard library to being uh uh emptied in the kernel so that everyone can use tcp without even thinking about it we think that a lot of the features and service measures today uh that started at as libraries that you had to use in your application for instrumentation are now at the external implementation in the form of sidecar step and we think it could go even lower uh into the kernel obviously not patching the kernel the way it was done for tcp back in the day by using ebpf to inject this feature so that it becomes totally transparent for the users and it's just their observability encryption mutual authentication and so on we also gain a lot obviously in performance because instead of having one um one proxy one envoy proxy per pod which means a lot of uh envoy proxies running on your nodes if you only have one per node we gain a lot in cpus and cpu and ram typically so the the view that the idea is whatever we can do natively in ebpf we try to do natively in ebpf including observability security traffic management because we gain a lot in performance and whatever we can't do natively in ebpf we can still use an external proxy for it and ebpf can still allow to directly route into this envoy proxy provided per node so we're still getting performance compared to a sidecar the performances this is a graph that compares performances based on a proxy or the visibility directly in a kernel using ebpf note that some of the the selium projects actually use directly ebpf based this to be visibility this is the case of tetragon for example and it's totally possible in the future we might actually use these libraries that already exist to parse htp or other layer seven protocols directly in the kernel to gain performance again so last point on tetragon and system observability and and enforcement so tetragon is kind of a complement it's been in the iso-villain cinema enterprise offer for many years as a close source product and it was open sourced in the beginning of this year as a complement to selium and hobo and it can attach to a lot of different events in the kernel and provide observability that the really nice thing about this is that because of ebpf we can do correlation and aggregation of events directly in the kernel using maps so that we export from the kernel into user space only what you want to see already organized already aggregated and this is a huge gain in performance and then based on this in user space you'll have the tetragon agent agents sorry that will read from the different structures in which the ebpf programs have written extract the information and make them available as metrics or as flows of information flows of events logs or traces based on this you can observe a lot of things file access network namespace escapes privilege escalation access to data on disk network protocols and so on and so forth one of the users that we have this is an example from iso-villain cinema enterprise where we have a process tree view that has existed for many years where we have a correlation between what's actually running into inside a container and the network flows that result from it so maybe you see in hobble hey there was a connection to this weird thing here something that not reverse shell.com which is definitely not a reverse shell right and you want to see where it came from you know in hobble that it came from this pod and so and this view here you can actually look into this pod see exactly what was executed in this pod this uses tetragon and you can see that five minutes after the server.js started application the container started there was a shell that was started and an NC and then a curl that connected to elastic search and you can say okay there's something fishy here but at least you have a trace of what was executed and what gave what led to this network traffic that you could observe in hobble so this is some information again on iso-villain.com we have some some labs with some of the features both open source and enterprise features in these labs and the community obviously there's the Slack and EVPF community Slack 7 and there's EVPF.io when it comes to EVPF itself. Awesome that was awesome and a lot of information so I guess we can we have plenty of time for this question. Any questions? I don't have any specific questions but I've seen that before and it's really really exciting and we are planning to I think you probably know really actually in GR planning to use it easily and we've had to implement some very basic features ourselves previously and this will allow us to get rid of all of that sort of accumulated tech I suppose but yeah it looks really really powerful so it's good stuff. So actually Jamie out of curiosity which features are you mostly looking at? Well there's a few things we wanted for I mean we haven't implemented this or something just haven't got a solution for it but really basic stuff like not basic but being able to see you know real product real source IPs and that kind of stuff for our traffic I mean as well clusters the Tetra concepts really exciting because we can use that we've got a lot of requirements around that the things we have implemented ourselves the more around the sort of service mesh you kind of stuff so the concept like egress gateways and that kind of thing we use and boy ourselves and figure out to do this kind of thing and then I've also seen them seen demo of the whole concept almost like a sort of global network policy thing working across policy which is really exciting but we've had to come up with I guess but we've pieced this together ourselves is the existing communities, primitives and CNCS stuff and having something like this which wraps it all up for us as an attractive proposition. We've used Calico for a long time as I see an iron generally it's okay but it's exciting to try something new and BPF just opens up a whole load of new having these for us. I don't know if you guys recall if you've done nothing with this one. Yeah so actually we discussed with Rafael very recently about this as well but the one of the main things that we've been looking at is the possibility of doing cluster mesh and this is for because we still push for people to have this kind of disposable clusters and to have applications deployed across multiple clusters instead of having clusters that need to be upgraded in place and things like this and for stateful workloads this is this can be problematic so we've been looking at cluster mesh to kind of expand the boundaries across cluster which will allow us to potentially do use this model even for things that have stateful workloads running like databases or I don't know we have some we have some even even for like batch or ml where you have long running jobs so this is something we're looking at. So you're considering specifically the shared service model was like a stateful cluster and then several state-last clusters that would access the services on the stateful cluster in a transparent way using global services. So yeah so that's that's one option and the other one is even to do to run the stateful workloads that have multiple replicas across multiple clusters even if we have not to know connectivity which we have for gem premises and we could consider even having replicas in multiple clusters and and eventually just just make them also kind of disposable as we add more it really depends on the workload of course how easy this is to do but but by having mesh for the pods and even like one thing I don't know if you mentioned or not but even this restriction that exists today that I know will disappear about having and non-overlapping subnets across multiple clusters this is not really a big deal for us because we orchestrate everything centrally but yeah it will become simpler. So I had one more to mention Lady Timothy you have a question? Yeah I have a quick question around how does it compare to admiralty in that in that sort of feature space? I can't answer I don't know I wouldn't be able to answer this. It's sort of like so admiralty is sort of like the ability to federate clusters so you know run one workload from one cluster into another where it sounded like that there was a lot of those types of features to provide some sort of federation. The use case I'm looking at is sort of like on-prem cloud bursting type things where you know you have a non-prem cluster and then you provide a cloud bursting capability and sort of stitch those together in in some sort of manner. Right so I don't know because I haven't looked at admiralty is it? Well no I'm just thinking in terms of not necessarily admiralty that's the use case that it's currently being used for just does can you do those types of things you have within Celium? I might be able to answer that actually. I know I think admiralty was more around sort of federated deployment of applications rather than accessing them so that would be a single pane of glass to deploy stuff and have it replicated in lots of different clusters environments where Celium is more around but I suppose the CNI and the network access to applications in multiple clusters and stitching together post deployment I don't think Celium does any kind of deployment of applications for you. But you but from what we're trying to accomplish you could build could you do that bursting type of federated thing across multiple clusters? So actually that was the other point I wanted to say that we are looking at and I already mentioned it to Raphael which is exactly that it's the ability to do sort of cluster mesh across regions or data centers which means that you won't have necessarily no to no connectivity because you might not have a VPN or something that will allow this. The way admiralty I am also not an expert but my understanding is that the way it works it kind of has like a kind of gateway pulse that then communicate to the remote ones. Yeah but it's really targeting not only the services but also the kind of batch workloads where you could submit pods to a cluster that then the actual workloads run in a remote cluster and they are kind of tracked through pods in local clusters. With Celium potentially we could do this transparently at the cluster level like we do with cluster mesh but we would need this kind of cross-boundary communication without necessarily have a VPN or something like that. I think that's kind of Timothy the use case here also. Yeah in some sense I was looking for motivation to dive into this a little bit further because you know these are not simple platforms to wrap your head around so that was enough motivation to go to go do some deeper research and potentially do some PLCs. Thanks. Yeah but maybe we take this kind of I had one more question to Rafael but maybe we take this as a kind of action item on the group and also Celium which is to track how visible this model is with Celium which is to kind of burst more than just across clusters but across network boundaries. So we have customers that do this but they actually have VPN between the different regions. But maybe it's worth because I have the feeling that several labs will be interested in this kind of functionality so maybe if we summarize this it would be nice for everyone. The the other question I had was like Jamie also mentioned that he's there using Calico. Is there anything that is worth knowing if people would move from Calico to Celium? Is there something that that would be lost or to be considered? I guess this would be a question for Rafael just if you know like of other people that have done this transition from Calico to Celium if there's anything specific to consider. I haven't heard of anything specifically. For us we will have some dependency on because we have an influence of some network policy as Calico specific network policies. Yeah the migration yeah migration of network policies but there's no as far as I know there's no specific feature in Calico that you wouldn't have in Celium. And obviously this is kind of like a fundamental thing so you wouldn't migrate a cluster in place with built-in clusters with the UCNR taken. Ideally yeah this is what would be recommended to start a new cluster with Celium and redeploy. And if you have if you have a good github's approach it shouldn't be too complicated theoretically. Unfortunately it's not always the case but yeah. If you just have one sort of X cluster that you separate that into the other problem yeah. There will always be one. Yeah I won't name anyone but you know people using hyperconvergent databases on the cluster and and then this cluster becomes a little gem that you can't touch or reinstall or replace anymore. There's a enterprise versus second source aspect here as well I assume. What about on the enterprise side of things do we have there any sort of extra complexity around licensing and that kind of stuff and stuff you have to set up or does it just work? No no at the moment it's just different images different ham charts but there's no there's no license per se. I've seen in the past with products where actually it's sometimes the most annoying thing about using the enterprise edition even though even the part from paying for it is oh and then you have to provide license keys and set up some complex infrastructure especially if you're on-prem and don't have internet access then it all gets a bit difficult. So at the moment it's not like this it's just in fact you can actually upgrade from Cillium open source to Cillium enterprise. I see you afraid to come forward if you have questions. Otherwise I think yeah I guess so Alex yeah there's a little late announcement. Yes you did. But yeah so I guess I was just trying to summarize what the main motivations are from this community to move to Cillium and I guess the ones we got here were the ability to do cluster mesh potentially doing hybrid bursting things and then Jamie you mentioned because you have quite a lot of clusters as well is this something that you're also looking at? Yeah hundreds of clusters so I guess these are things we can track together and you mentioned also the egress gateway for that you're setting up with a custom envoy and simplifying that. Yeah and what else was there? How usage? Things like access to more global network policy and then all the security sort of things that touch upon stuff. Yeah so just to note somebody on my team has started looking at that directly. Tetra gone Cillium as of last week from a security standpoint. Maybe I have a related question which is regarding so for our load balancers we have a different solution because you mentioned tracking the source IP and we always have this issue if you have like a GLS SEL pass through across the load balancer to the back end that you lose the initial IP and we've started enabling this proxy protocol to be able to propagate the source IP of the client to the back ends. You still don't lose it through a key proxy? Well this you can fix at the Kubernetes level though. I don't remember exactly but there's I've tried a file to do that actually using that external traffic policy. Okay that's what we are doing and actually that works and what we were losing was the source IP in the LB because of GLS but with proxy protocol and using something like nginx for the ingress this is all working and for the LBs we are actually using HA proxy with a kind of active passive mode. Is this something that Cillium would also be potentially a good option for for the external LB? With the external LB I couldn't say at the moment. One thing I've seen was keeping the source IP but that's that's kind of a different situation than what you have is the possibility of actually doing routing directly keeping the IP address when accessing the service and there's two ways of doing this so there's either a DSR or oh it's the other way I don't remember actually so there's ways to so DSR will make it so that the back end will actually reply directly to the to the source IP without routing without doing an SNAT so going back through the node in which it entered but that doesn't really solve your situation right because you want to go through an L7 proxy right yeah and well at the moment Cillium doesn't work as an L7 proxy actually it works as an L field for proxy so I don't think we would support this so we have HTTP or HTTP as visibility but Cillium in itself using MVPF as far as I know doesn't doesn't do L7 protocol but but in this case it would actually be an L4 because this is like pure TLS and this proxy protocol is actually doing just a blob at the start of the binary packet and this this blob is understood by some implementations of oh yeah you have no TLS termination there yeah so it's just a pass through and yeah I think it's yeah it's called proxy protocol and it's not wildly supported but it's supported by nginx, htproxy all this stuff okay well I'd have to check on this I don't know all right but I think this would be pure L4 there's no like we do just it's like TCP no no HTTP all right yeah that would make sense I'd have to check if proxy protocol would explain this because this could be something that we could look at as well to have a solution here for for LB as well all right I don't have anything else so last last chance for a couple more questions to Raphael otherwise thanks a lot Raphael again and especially for the immediate reaction to the call so that was pretty awesome yeah thanks very much yep and we'll keep track of how people start using Selim also in the research deployments thank you never know so we'll have the next meeting in two weeks I think I'm always confused about this first and third I think it's two weeks this so yeah we'll we'll circulate the topic in a bit and then maybe next time we also prepare the topics for at least the rest of the year yeah we need to refresh the backlogs with a little bit dry or something yeah I can spend some time next next next session thank you bye bye thank you all right have a good day all right thanks