 Yeah, so I'm from IKEA. I work as a senior systems engineer at IKEA retail, also known as the Inca Group. So what is the Inca Group exactly doing? It's operating close to 400 stores in 32 countries. It's also responsible to be employing 170,000 co-workers. There's some numbers on the 2021 terms of the visits to the IKEA website, including downloads. We have 4.3 billion visits to the IKEA website. And we also have around 700 million visits to our stores. So it's quite a bit of an enterprise. If we're looking at what I'm part of the digital organization within Inca, it's handling 3,000 employees that is doing all of the technical development that supports most of our business, including... I've lost it. That's good. It's been a while since I've been on stage, so that's maybe why. But anyway, we also like a lot of other people looking for... people to join our forces to actually deliver on our journey. So let's go into why we're here today. Let's actually talk about the project that I'm working on. It's part of our modernization efforts in terms of modernizing our enterprise. That is a lot of other companies or enterprises that are in the same situation as us. We're not just modernizing the IT landscape. We're also modernizing the business as a whole. We're moving towards more circularity and reusability of our products. But in terms of the technology, we are moving from a lot of operational-focused operations to more agile and DevOps approach. As part of that, we're also moving a lot of our workloads from our data centers into the public cloud. But it's not all workloads that are movable in our situation. Since we're operating in that many markets, we're also operating under regulated environments, so that there's a ton of that that needs to be handled in a good way. So we're setting up this private cloud initiative in our data centers in order to get the same kind of operational experience or developer experience that you will have in the public cloud. So what is needed from the private cloud initiative is that it should be easy to use, just that you would experience with any other public cloud. It needs to have a cloud-like consumption model in our data centers. And also, we need to build trust to our consumers of the private cloud and consumers in this context. It's not IKEA customers going to the stores. It's our developer community within INCAP. So we are also one of the initiatives that is really needed for this is that it's multi-tentancy, meaning that multi-tentancies within different developer communities in INCAP. And also, we are putting a lot of effort in the Zero Trust initiative so that we can run application with confidence in a multi-tentancy environment. And from this private cloud, we looked at different options, like there's a lot of enterprises has been down the open stack platform, but we're kind of building this environment to support our needs of tomorrow, not as such as a cloud provider as such. So we decided on moving or leveraging the Kubernetes infrastructure initiatives. So it gives us this possibility of actually extending the APIs a lot with different types of CRDs so that we can get, for instance, we can run QBird as an operator to handle VMs in this environment. There's also a very highly demanded interoperability between our current environments that has been built many years ago and this environment. So we have a need to actually be able to coexist these environments for a long time before the old systems has been modernized in order to fit this new environment. So from our design thinking point of view, it's an open source first platform, meaning we kind of went away from dealing with traditional vendor approach where they deliver something that we can use and operate. It's been needed at least for some time that we actually have lots of flexibility in terms of picking and choosing from the different components in the CNCF initiatives so that we can gain speed and flexibility that way. And we have been looking at how a lot of the public cloud providers has been doing their infrastructure with regions and availability zones. In our context, the availability zone is a Kubernetes cluster. And then we can have one or more in a region. We have load balancers as a shared resource and the same with the storage. And since we are building this environment for being ready for tomorrow, we're also looking at running this as an IPv6 single stack and also we're looking at multi-home parts in order to support some of the more traditional workloads. And to tie these zones together, we're using the Asilium cluster mesh because it provides us with some nice integrations to running this multi-cluster environment that we have been trying to do the same type of thing before with CoreWiz. It actually supports having cross cluster lookups. And you can do writing your own network policy operator to make that work. It seems to be a lot easier when it's all built into the Asilium tool. And the last piece for this environment is that we're doing BGP routing within our data centers. What it gives us is a lot of flexibility. And you can actually move your workloads around much easier in our opinion. And also this gives us the possibility to scale out the environment both in terms of the number of nodes and clusters, but also if we're running low in the data path, we can scale up or add an additional layer of leave switches in the top of this that will bring us more bandwidth within the total environment. Why is this important? It's because we're moving from a scale up solution into a scale out. We also need to have the possibility to scale out networks. And with this setup, we can actually fairly easily get our reusing egress gateways as well to connect to the legacy environment. We can use these egress gateway IPs fairly easily to get this all tied together. And then you get global port routability, pod IPs, and what's not to like about that. And then we're looking more into this networking and the Cilium piece, as well, why people are here today, I guess. So I guess this, Thomas had a slightly more updated version of this. But in essence, we think that or see that EBPF is gaining a lot of traction. And since we're building this environment for tomorrow, I think it's a right time to actually address some of the challenges that we see today with actually some of the technology that is really picking up today. So it becomes actually one of the key things for us as a team is that it makes it easier for us to deliver a trusted environment when we're going in this path. And the next, at least it's no secret that our team is not really developers or kernel developers. So we have teamed up with ISOLevent in order to co-create some of the features that we were missing in the Cilium product. Firstly, we started out this project of the private cloud, started out two years ago, more or less. In that setup, we were leveraging our on-prem load balancers with Live and its operator in order to control it from Kubernetes cluster. It's more or less a traditional load balancer setup where you access the cluster from through an F5 load balancer outside your workload. This is a fairly straightforward environment, but it also brings the need to actually run your load balancers outside, either on a physical device or through a VM. And in this environment that we're building, it's actually also meant to run inside our stores. So it should be able to run at any scale, so from a small environment with one cluster to a larger data center deployment. So we're looking at the XTP, the express data path that allows us to actually bring the load balancers back inside the Kubernetes cluster. And what we were missing in this earlier adoption phase was actually that we were running, as mentioned, with Multinic ECMP through the BGP integration. And when we started looking at Syllium, it didn't support Multinic XTP. So luckily, we worked on that together with Iso Luven to get that into our stack. We're not quite there yet. We hit other problems with XTP. We found that apparently it's not that simple to get it working, even though we're looking at some of the supported NIC or network device drivers. We ran into this kernel bug that we're currently working with the vendor to get resolved. It locks up our nodes. But for now, it's doable, at least to run our current testing and continued work on moving forward. The multi-homing piece is a different story. It's where you actually can get multiple IP or networks attached to your pods inside your Kubernetes environment. This is interesting for us to support some of the multi-tentancy problems or problems that what you want to say about it. You can easily end up in situations where a tenant will actually consume all of your bandwidth with doing things like data migration or AI workloads that consumes a lot of bandwidth to consume data. There's a tool, Multus, that supports this. It's actually, in our opinion, it brings also a lot of operational complexity. It's not just one networking component that you need to run. You need to run multiple different ones to solve your problem. And then from a zero trust slash multi-tentancy point of view, it doesn't really bring network policies. Again, we teamed up so that we can actually run multi-homing inside Syllium. It's currently a proof of concept that we are doing. And it's where we have the possibility to run two different sets of networks, the default Kubernetes Pod Network and a different network with the same kind of feature sets. So you can have service discovery and network policies on your secondary interface. And in our BGP backend, we can then route this through different nicks and different paths in our networks in order to not have network conjunction. We also have the possibility to have this network manager that was also discussed earlier. And there's been some work put into that that makes it even better so that we can actually even further tighten or secure that the different tendons in our environment doesn't really break things. So that's really nice. And in terms of the IPv6 single stack, we think that the time is right to actually move into that direction since it's a new platform, but also since IPv6 has been around for that long. And other projects that we're using, Cube Word, for instance, is also bringing IPv6 single stack support. So currently, we're not really routing IPv6 within our data centers. So we have some outstanding things that we need to address. For instance, we need to isolate this environment so that we can actually get NAT64 and 4.6 working. So we're working on that feature together with Isolavent to actually be able to have the translation happening at the load balancer node so that the environment could run IPv6 and everything that is communicating to it will be addressing it as IPv4. So this brings us closer to be able to roll out our IPv6 moving forward. And to connect all the things, it's quite simple to actually connect your network from your Kubernetes environment into your BGP environment. We're using BERT as the routing daemon on our nodes. And it has a nice integration to local host. That's like for those, you cannot really do multiple BGP sessions out of a single node, so you need to kind of correlate them on your node. So that's the integration piece. For the observability piece, I was actually thinking of doing a demo, so I hope that will work. So as mentioned, we were implementing this dual-necked XWP XDP support in order to actually look at how things were going. And how do we then test this when your old-time sysadmin is trying to use your TCP dump, and it shows you nothing? It's quite interesting. I think our setup is that we have these nodes and maybe it would have been easier with a different mic. It'll help. That was actually why I was a little worried that it's working. Our VPN was just two minutes before this, so the node that I'm on now is actually one of the clients. That's not it. That's not the client, sorry. That's the it would have. Live demos are hard, people. Give them a hand. Thank you. We've been out of practice at this. It's going to take a while. I don't know if all of you remember the excitement and butterflies of standing on stage. They've gotten worse after two years in no practice, least in my opinion. So the top screen is my client. And when I'm trying to look at, in my button left screen, look at TCP dump, that's at least my go-to tool when doing network debugging. Apparently, when you query, it's not doing anything. In particular, you cannot see it's actually getting this request coming through. So it's going to my node port service. And then it's hooked into HTTP. So that's not really going to give me anything in particular on that. So luckily, Syllium has this tool that's the Hubble Observe that will actually give you some of that back. And you can use your Hubble Observe to actually look at traffic coming. The app will record. That's my mistake. It's you can record the traffic flow coming in. And you can see that for some reason something else happened. Yeah, I think it's a bug in the system somehow when you start this up, the recorder, it actually needs to get to this stage in order to not show things and be recorded. So you can see that you're picking up the packages in this scenario. And then you can get back to your actual. So I have a cheat sheet or not. I cheated a little bit like in the pre-baked your cookies in order to not spend all your time on downloading things. So if you're looking at it and you at least go to, whoa, but anyway, you can see the traffic from our client going to the node that hosts the node pool. And then you can see that traffic is actually leaving towards the pod IPs. So if we exclude this one in order to see that will go. So the source MAC address is one of the nicks on this machine. We can exclude that just to verify that it is actually using also a different source. So you can see that's a different MAC address. So in that case, we can verify that we are actually leveraging both nicks leaving this there. So that was kind of the demo. The one thing that we are still kind of investigating a little bit is actually how do we get these data into some more convenient tooling than doing a Hubble record and then looking at them in Wireshark. It would be really convenient to have them. We're using the Grafana stack to do metrics together with Prometheus, but it would be really nice to have a way to actively to get this into this stack so that we can actually support our developers in their journey on being self-serving on also networking components. So yeah. Do I have a mic? Oh, I do have a mic now. Here you go. Hi. Hello. I wondered, you mentioned the BGP thought advertising to your network. I was wondering, I also saw the small YAML paste there. So I was wondering, how is that integrated with the rest of the infrastructure? Because there has to be a service discovery and different things. You mentioned the bird demon. How is that integrated? So bird is connected to our BGP infrastructure. So it's advertising the IPs to the switch BGP demons. And in that case, it's routing the traffic. We are using the service type load balancers to actually run services inside our network. And that's where we are needing the net 64 and net 6546 in order to actually do the integration perfectly. OK, other questions? No? Oh, yeah. Thank you. I love that everybody is helping me. This is awesome. Hello. How do you handle multi-tenancy in this, like you mentioned? So we're doing multi-tenancy based on namespaces. And then we're installing the network policies in order to ensure that everything is blocked by default. So that's kind of our current way of thinking. In order to ease it up a little bit, we are actually thinking on leveraging the namespace for the hierarchy plugin. OK, thank you. Oh, more questions. I love this. Active audience, you in the back. Think of questions. Thank you. Can you please talk more about the CordianS cross cluster plugin? Sure. So CordianS, at least when we did it, it has a cross cluster plugin that you need a service account in the other cluster in order for it to do this service discovery. And then you can actually specify what we did was to have different types of cluster domains. So we had different types of cluster domains. And we can leverage that for our service discovery. And then, of course, you need to have that cross cluster functionality in all your clusters. The second part of that is also having your part network routed. But I guess that's a different story. Thank you. All right, so now I'm looking at the back half of the room. Any of you have questions or want to point me to someone that you know has a question but hasn't raised their hand? No? OK. Oh, hands here. I missed the middle bit. So how do the developers interface with Cilium? Can they use Cilium to observe traffic and to debug their applications? Yes. So we're using Hubble, and we're allowing all team members to view what's in Hubble. We also have been using the Hubble Hotel integration to actually have network traces inside our Grafana tempo instance so that they can look at that. We have briefly been playing around with actually doing metrics based on traces and see if we can actually further enrich data and also give them much more insights into specific networking components. OK, I think we've got this last question. Thank you so much, Karsten, for a great presentation. I was wondering about your multi-homed network interfaces and how do you handle asymmetric routing for that use case? You said you were using ECMP together with the network stack, or are you only using one interface at the time? Yeah, we're using ECMP in order to gain that it can pick any path, basically. So it doesn't really matter since it's routed network. So either way, it can pick any interface, both in and out. Our announcements is for the multi-homed. We are announcing different networks on different NICs. So that's how we are handling that. So POD is never using both networks at the same time? So the initial use case for the multi-homed thing is to be able to, if anyone has been trying to run an elastic stack, there's HTTP transport and then there's the transport layer. We want to separate those two so that the search queries goes in on the separate NICs, where that in our environment is the client facing interfaces. And then all the transport is handled on the back end networks. And you can handle that with being able to have multi-homed pods. Thank you so much. Thank you so much, Karsten. Can everyone give him a big round of applause? Thank you. Thank you. Thank you. Thank you.