 So, our kind of almost final session of the day is going to be a really rather interesting panel. Now, how many of you here have watched Raw Code and in particular, clustered before? You've got some people who are familiar with clustered. So it's a really great, like, YouTube show where people break Kubernetes clusters and people try to figure out how those clusters were broken and fix them. So we've been doing over the last couple of weeks, we've had some teams fixing some broken Kubernetes clusters using EBPF tooling. And so our panel today is going to be hosted by David Flanagan, also known as Raw Code, who is the host of clustered. And we have a panel with representatives of our three teams who fixed those clusters. We're going to learn about how they use these EBPF tools to solve problems that are representative of real-world problems. So we have David, we have Duffy Cooley, we have Margot Monteirola, and we have Loris Dejane. I'm going to get your name right, Dejane, something like that. Sorry, Loris. OK. There's going to be a couple of minutes here to just, I think, get the screen sorted out and mics. And oh, we've got lots of microphones. That's good. So I guess once the microphones arrive, maybe we can start with Loris and have the speakers just introduce themselves. How about that? Yeah. My name is Loris Dejane. I'm the CTO and founder of CISDIG, CISDIG is a company that does security invisibility for modern cloud-based containerized applications. I'm also one of the early contributors to Warshark, the network analyzer, and one of the also creators of Falco, the open source tool. So a career in open source and kernel hacking. I'm Duffy Cooley. I'm a field CTO at ISOvalent, and I've also been working in SIG cluster lifecycle on QBDM and kind of helping people fix and break clusters for quite some time. So this is a really fun thing for me to be a part of. I'm Margot Monteirola. I'm part of the KingFolk team at Microsoft, leading the team that does Inspector Gadget, which is an eBPS tool for Kubernetes for figuring out when the cluster isn't working as expected, which is really fitting for when you're trying to fix a broken cluster. Hello. Hello. Hello. My name is David Flanagan. I go by Rocko. I'm excited to have this panel. So we have slides. Okay, we're good. So let me fill you in a little bit for the people that are not familiar with what Clustered is. I often call it the worst idea I ever had. Having people break Kubernetes clusters and then live streaming fixing them is extremely stressful, but also a very valuable learning experience. And we have two different versions of Clustered. One is Solo Edition, where it's just one community member and another community member. And we also experiment with Teams Editions. And I'm always looking for new victims. So feel free to reach out to me. I also want to thank Equinix Metal. They provide the hardware for the three clusters that we are going to watch get fixed today. So we've got Teams. We've already done the introduction. So we'll skip by this. And I want to just say before we show the videos that this is not easy. So I want to thank everyone involved, the breakers and the fixers. Like being given a cluster and you have no idea what is wrong with it is intimidating and stressful, but they all absolutely smashed it. So well done. And all of the tools that we're going to see today to debug and fix these clusters are open source. So this is stuff that you can play with and experiment on your own time as well. All right. So we've got our first video. And this is our Isovalent cluster. Oh, we don't have audio. Can we get audio? Daph is just going to do the voiceover while the video is playing, if the audio doesn't work. Yeah, it's so fun. It's so fun. It looks like you're there frozen on the screen. That's definitely my troubleshooting face. These clusters were broken by Thomas Graff and a few other folks from Isovalent. And so I was definitely expecting some very interesting breaks. Yeah, for sure. I don't know how nice your colleagues were with you. Mine were very nasty. Oh, really? Well, one of the problems on our cluster that we'll talk about here in a second was obviously a DNS problem because it always comes down to DNS. It's always DNS. And then another one, we thought that we had fixed it, but it was a decoy problem. Like you get past that problem and you're like, no, no, it still isn't working. So we're starting off by just taking a look at what's happening inside the cluster. We're just trying to see what's happening. And we can see that initially we see that there is a pod that isn't up and running. And we're trying to figure out why that pod isn't running. And we find a very important init container that is basically keeping it from running by trying to make a connection to Google.com as a pre-check for the container. So until that init container happens, it won't start up. That is your mic? Hello? Yeah? Yeah, that's what he's doing. I can keep narrating, I guess. Yeah, the narration was actually pretty interesting. I don't think I recall what was working with our cluster to narrate it. I guess we can discreet what's happening? Yeah, I guess I'll keep going with the narration, yeah. So we were, you know, the goal of our efforts is to make it so that container that we're referencing here is able to come up and that we're able to see, like, a success image on the website is being hosted by this container. And so first we try to figure out, like, what was happening with Google.com. And we eventually came to the conclusion that what was happening was that someone had actually put Google.com into the Etsy Hosts file pointing at, like, some other, you know, URL. And so it was never going to actually resolve because it was against some, I think it was against local hosts or something. And so that was actually probably one of the more creative DNS problems that I've seen in a while because basically it was just intercepting it since Core DNS was using the Hosts file to determine what the result of Google.com was. And at that point we started moving on to the second set of problems. And the second set of problems, actually, I think we're still talking about the DNS one here. So we're trying to figure out, like, what the problem was. And we had a policy that would allow for us to communicate to Google.com, but for some reason it wasn't resolving. And so we were trying to figure out why it wasn't resolving. And the reason was because of the Hosts file, which was very clever. Thank you. Thank you, Breaking Team, for that one. Finger crossed. The scope should be allowed. Your idea is that the init container is going to use the Hosts DNS. OK, let me restart this then. Team eyes of villain, best of luck. Take it away. All right. So let's look around this thing and see what we got. Morning. Got some previous things. Something's not running. Small splash. OK, so init container is not ready. And init container has a very important name of probe do not delete. So that special version of cluster to just for team eyes of villain must be able to reach Google before it can start the cluster application. Given that we are a network things, you know, plug in, maybe there is a networking problem here we need to look at. That looks good. This is nice. Yeah, everything's running. Yeah, the total stuff. And we have a UI. We have a UI running. So we have Hubble running so we can use Hubble. But if we rewind a bit, so we had this init container and it was not in red state. And I believe it was doing paying Google.com. So I'm wondering. It should use the host resolver. So even if the core DNS was not functioning, like it was not able to. So I see here we've got the rule that allows anything to reach QDNS. It also has a rule to allow to start at Google.com. So that should that FQDN lookup should be allowed. Your idea is that the init container is going to use the hosts DNS services. Correct? And there you go. That would not be. So how's our pod looking now? Well, it's running now. So is this why we're supposed to try and upgrade it? Just changing the tag to be two in hopes that that one reasonable thing to do. We have an error. Fail to connect to database. We definitely saw some policy drops between Postgres and things. So let's take a look at that policy. OK, Gary. It is policy denied. What? Cluster talking to a clue. What's that got to do with clue? So is Gail and Rollins' service selector here? Shouldn't redirect to those endpoints. It should be redirecting to a different endpoint. That doesn't seem right. Do we have a service at this particular thing is targeting? What services is it trying to connect to? And let's look at the service for is it that's the database Postgres? Could you? OK, I think OK. Where do we go? Let's look at this. This looks yeah. Big fan. Big fuzzy hearts all over the place. I actually don't think you would find that one so quickly. What we're seeing here, yeah. So what we're seeing is that when we do the Hubble Observe, we're noticing that the clustered application, when trying to communicate with the database, is for some reason going to the Fluent Depot. And we're like, that's weird. Now, there is a function in Cilium that allows you to do local redirect policies. So if you were going to do something like you had a node local DNS cache, you know how you can use IP tables to intercept the DNS traffic and send it to the node local piece? This is a mechanism that can also be used for that, but it's implemented in the EBPF where a policy might look like this. And that policy is basically saying that when traffic within this namespace happens when something tries to reach the service Postgres in the default namespace on port 5, 4, 3, 2, and it's TCP traffic. We're going to redirect that traffic to a different back end using a local endpoint selector that matches labels at Fluent D. So basically in the EBPF, we can be like, don't send it to the Postgres database and send it to this other thing that is local inside of the namespace. Should we try the web page? See if the web page will refresh. Just one more. Oh, I'm broken. At least it's going to have Postgres. Does the pod have a matching label? Yeah, that's what I'm looking for. Let's take a look at the policy real quick. CNP. Reading this, I'm looking for app Postgres and app clusters. So if I did kubectl. Or you could just do get pod show labels and see what. Use an active. Call the label. See what they actually did call it. It was Postgres QL. Look at how fancy. He's guessed it before it even happened. Well done. Can we do the browser check again? The goal of Clustered is to see the dance. And they did great there. So these episodes are nearly an hour long. So been able to condense it into six minutes. I hope you were able to follow along with the breaks. But it was a whole lot of fun, but well done to my availing. So if anyone has any questions about this, feel free. We're going to try and get you a mic. And I've got a couple prepared here first. But if you do want to ask a question, just put your hand up. So Duffy, Isovalent has gone all in on EBPF from replacing kubproxy, removing dependency on IP tables, and more recently, tetragone. Do you want to explain the buy-in for EBPF and what you're excited about by using EBPF on a daily basis? Yeah. I mean, what's interesting is that priority of product actually describes the value in EBPF across a variety of different verticals, if you will. So with tetragone, when we started talking about tetragone, one of the amazing things that we're talking about is the ability to gather context about what's happening at the application layer and present it to you in a way that you can use it to understand what's actually happening, like which given process is making what connect call, what system calls are they making, and those sorts of things. And then if you look at the way that we're using Cilium, we're using Cilium to leverage EBPF to solve a lot of the networking use cases. Like how do we handle load balancing? When we replace cube proxy, we're talking about using Cilium to be effectively like that internal load balancer that every Kubernetes cluster has. So it's a variety of different really important use cases for EBPF. Awesome. What I really loved about our session was the developer experience, which we haven't got in that six minutes, but the videos will be available. But there was the Hubble UI. There was a Hubble CLI. There were Grafana dashboards that were shipped. And even the use of the Cilium editor for network policies. So you seem to favor the CLI in our session. And I was wondering if you could tell us a little bit more about it and how people can also use the Hubble CLI. A little bit more about what, sorry. Oh, can you tell us about the Hubble CLI, why you enjoy using it, and what people can do with it? Yeah, I mean, for me, the part of it that is kind of amazing is that Hubble CLI gives you a view into what's actually happening at the network layer as far as connectivity and that sort of stuff. And so it's basically just a way of using a text-based visualization tool that lets you see event-based network connections between things. And in this age of microservices, where there's a network between everything, we need that kind of visibility to even know where to begin to troubleshoot. Because usually it's not, I mean, how often has it been an actual application problem versus that there's actually something on the network? Maybe you've misdefined a network policy. Maybe you typed the wrong URL. You actually wanted to see some kind of an ex domain that it is actually DNS. But this visibility, I think, is pretty killer. It's one of the big differences. Awesome, thank you. Does anyone want to ask a question about Hubble, Selium, or Taddafi? All right, let's roll the next clip. Pretty good, take it away, start typing, and best of luck. So yeah, here we have the cluster. So let's see the nodes that we have before going on. OK, so yeah, let's check the pods that are running there. So yeah, here's the pods of the control plane of the QP system in this place are all running and good. So yeah, that's good for us. And yeah, we have this pod that is in crash loopback of state. So probably this is the one that we have to fix now. We can get the logs. So get the pod, the name of the pod. OK, so let's describe this pod and see what we have there. I think, yeah, we have a set-consecurity profile there. OK. Yeah, if we have a permission denied problem, maybe this is related to second. Yeah, we have the annotation here. And it's a custom one, so that's a specific history. Yeah. OK, so yeah, I think we can use inspector gadget to understand what is going on with this set-con. Probably what is happening here is that the pod is trying to use as it's called. That is not allowed by day. So in order to use inspector gadget, we have to install this in our cluster. So yeah, this is gadgets. Set-con. So this is telling us that there is a pod running on this node in the default name space. And this is in fact the pod that was crashing before. This is the name of the process that was trying to perform as it's called. This should be a little bit confused by the output here. This system call is not part of the clustered spec. But it's not the reason the pod is crashing. Yeah, yeah, at least. This is two breaks colliding together to cause a very unfortunate side effect. But what else would give you permission denied if it wasn't assist call? Well, capabilities. All right, so we can trace capabilities. This should be good. OK, here we are. Oh, so we have security contest capability. All, yeah, this is not good. We are running the pod without any capability. Of course, this is going to fail. What we need to understand right now is what are the capabilities that are required by this pod? So yeah, I think we can use the capabilities gadget to get a list of the capabilities that this pod is trying to use. So we have capacity and gadget-dressed capabilities. We are only interested in this one. Oh, here we are. OK, OK, got it. And yeah, this is the name of the container, the UID, the PID inside the container. This is the command that is trying to do that. And this should be the capability that this is trying to use. Netvine service. Does it look good to you? Looks good to me. Ta-da. This is working. It works. I mean, saying it is working is a pretty bold statement. But you can confirm that, right? We wanted to curl from 30,000. Local host, 30,000. We were trying to verify that the application is working. So with this score. It doesn't seem to work. Connection refused. So there's something else that needs to be fixed. So this is 30,000 here. And this should be 8080. So what's the pod? Maybe we can list the sockets that are opening the pod to check what is the real pod name that the pod is using. I like how you don't even look at the manifest to see if it's got a container port. And you're like, no, please, just go and list all the ports so that it's used to be. I don't trust the manifest. Let me try something. I think we can just trace the whole name space. And that should be fine. I'll use the gadgets here. Just delete the pod. OK. Seems demonic. It's working. So yeah. The pod 666, right? So what we can do right now is to edit the service and change the pod. I think that this is why. Yeah, that's good. Nice try. OK, something doesn't work. Another thing doesn't work. Your last break. I think somebody said before that second wasn't the problem yet. Yeah, I think that's it. And let's use that on the whole name space. I think this is OK. I think what's interesting about this is that Postgres traffic means your request has been received and the application is trying to respond. OK, so yeah. It doesn't seem to be a problem with the networking because the traffic is arriving there. The full connection is established. I hadn't seen that TCP connect before. That's awesome. OK, I'm sending a request now. It's shut down. Can you try to send another request, please, Iago? So I turned off the JSON output. So this is easier to read. And this is more terminal. Yeah, it is shut down. It was funny that you said that the JSON is easier to read. OK, so maybe we need to edit the second profile. And because the shutdown is never called, the response is never sent to the client. Let's do it in a poetic order. I appreciate that. And then the queue will apply the new second profile. Yeah, yeah. Ta-da. This is working. All right, so that was quite a tricky one. We dropped all the capabilities. We added a custom second profile that was missing once this call. And I gave them a poll with no idea what port it was running on. All right, so debugging Kubernetes is a huge challenge for all teams. But what I thought was amazing there is you really didn't look at the yaml to see if there was a container port. You just lest eat all of the ports that the container was using. And I thought that was very, very cool. So can you tell us maybe two of your favorite gadgets that you use and why you like them? Sure, yeah. So I think one of the cool things about Inspector Gadget is that it has a wide array of things. And it has very simple gadgets, like tracing the exec syscalls. And so you can see if something is executing a command and it's not finding the command or whatever, you can see what's happening or tracing the open files. These are very simple but very useful. But it also has complex gadgets, like the network policy advisor that can capture output and generate network policies based on the output captured. And so yeah, I guess I like the flexibility of the different gadgets available. Awesome. So we use the custom set comp profile there. And we're actually in more people adopt set comp and apply them to the container images. But I think there's still a lot of frustration around generating those profiles. So could you maybe give us some tips and advice for doing that? Right, yeah. So we also have a set comp profile advisor that is just for that. So you can run the advisor like in capture mode first to see all the syscalls that your pod is making and then just tell it to generate the profile. And it will generate a profile with just the syscalls that your pod is using. And you will have a matching profile and you don't need to handcraft it because it just generates a profile for you. Awesome. Thank you very much. We'll just do questions at the end. I'm going to move on to the next one, which is the sysdig video. And I just want to say, I worked with breakers from all of these companies. And of course the sysdig breakers really wanted to make you sweat. Like they really wanted to cause you some trouble. So exactly. I can tell you that your colleagues have left you some Falco workloads in the cluster. So we have 5-star QQI. Yep. If you want to access the Falco site QQI, I'm happy to open it in a terminal browser on the site just so the audience can follow along. I mean, right now it doesn't look like it's broken. So yeah. Oh. OK. I have to connect to the database. Passful authentication. There's also an image here in the page. Can we open it with the browser? Where is the chat? Yeah. Looks like we're upset. Should we go take a look what the site QQI is telling us? If we get any hint from maybe any Falco output, first of all, maybe it's captured or something. We have an exec into a pod, a DV program was spawned, and Bash history was deleted. A naughty person has been in one of these pods, I think. Scared your current jobs? Keep scrolling. Three times. Deleter in a mystery, attach or exec in pod. Definitely somebody has gone into one of these pod. Which one is it? Yeah. And there is an alter user in the database. It's Postgres, right? There's also the password in the alter command. Not today. So somebody went into the Postgres pod and manually changed the password. Is that what they did? It's very cool that Falco gives you visibility into these arbitrary random commands within a pod. That's pretty sweet. Here, the Postgres pod is running in your next list. Oh, wrong image. The Falco UI also said something. Container image Postgres 1, 3, alpine. So we have a meeting somewhere. Those are nasty. I haven't maybe got rid of that. I mean, your Falco UI here shows you that the web hook to DNS is modifying all container images to be engineized latest. I did it. I did it. I can delete it right now. Just look it. Yeah, I'm just taking my pics are pretty opaque. You don't know what they do. You just know the permissions they have. We do have v1. So currently, our application is working. Oh, look at that. So now you have to upgrade it to v2. Just watch this thing with event.type equal exactly. And we can see the executions in the cluster. Yeah. Yeah, exactly. There's some kind of hidden process probably somewhere here. There's a process code, definitely not. Yeah, that was the question. Where do you see it? Or now that we need it. What is that? What is that? What is that? It seems to be running date and sleep. So what if you just kill the process? We have the PID from sysdig, right? Try just to do a sysdig filter with a definitely prog.name contains. Definitely. Definitely. Maybe let's try from sysdig. But again, I don't know when the damn thing starts. But let's try from sysdig. And before the filter do dash p, prog.name. Can you scroll up in the meantime? Because there is that clone. The clone should contain the parent too, yeah. pt systemd. It's systemd. And the arguments is barcatch, definitely not. So this is run by systemd. Just kill dash nine systemd. That's a good way to do it. I killed the PID one. OK, exactly. Guys, have you seen the Falco UI service there? Do you see that? I would like to check the service Falco UI. System CDO cat Falco UI service. Good call. There you are. Barcatch, definitely not. Yeah, yeah, yeah, yeah. Cool. Dun, dun, dun. Not today. Oh, it's nice. Loops forever. Checks to see if the pod is old and resets the password on postgres. Good catch, Vicente. You have to dance. So what was really funny about that is they had the bad code instead of a Falco UI service that obviously wasn't Falco UI. But they actually went to the extra effort of that bash script, which was running every five seconds initially. And they said, it's too noisy. They'll find it too quickly. And they actually made it check to see how old the pod was so that it only ran when they deleted the pod. Yeah, exactly. So we were trying to capture it, but it wasn't happening often enough to actually be able to see it. Yeah, it was very nasty. They put a lot of effort into that. So that was a lot of fun. And we see mutating webhooks a lot on cluster. They are very opaque. They're difficult to track down. So it was actually really cool to see Falco with Sidekick and the UI actually give you visibility into what was happening and that, which was kind of cool. So something that I often see is that security is still kind of considered like a day two problem. But now that we have tools like Falco coming in and becoming adopted and there's ability to share rules, do you see a shift changing there where people are starting to maybe work on security earlier? Yeah, for sure what we're seeing with Falco. So Falco is to attempt to accomplish two goals. One is the use of modern techniques like EBPF to capture data that is as granular and as real time as possible from the operating system and from other sources as well nowadays. And the other one is creating sort of a simple policy language that people can use really in a community way. A strong belief that we have in the Falco community is that you can have a chance to win the war with the bad guys only if all of the good guys can get together on a single platform and a way to, for example, create security policies that are built on top of a simple language that is working on top of EBPF is one way for the good guys to work together and for sure we're seeing people are incentivized essentially to produce policies and rules that then become a value for the whole community. Do you feel that we are on the path to there being secured by default rules or policy that can be deployed to our Kubernetes clusters? From one point of view, I would say we're going toward the direction. From the other point of view, it's also constantly a challenge because security is always, it's always environment dependent. So building something that works out of the box for everybody is the dream. And definitely with every iteration of security tools we're getting better and better. But I don't think we're there yet with something that just works out of the box for everybody and is secured by default. OK, I'm going to throw a set to all of you now and maybe we can have a bit of a conversation. But I think what's really cool here is that we've seen EBPF applied to networking, security, debugging, it's almost a superpower that anybody can learn and harness and use. So do you have any tips for people here that want to adopt EBPF? And let's try and tackle that as consumers of EBPF tools, but also as being EBPF developers. So any tips and advice for people? I don't know. I can go first. In my opinion, EBPF is a super cool technology, incredibly powerful. And I recommend to anybody to learn more about it. It's such a powerful Swiss Army knife. But at the same time, you don't need to be like a compiler expert, a kernel expert to take advantage of this. What we saw in this panel, what we saw today is that there's a bunch of tools, Cilium, Falco, that expose this kind of functionality in a way that doesn't require you to be a rocket scientist, but still allows you to take advantage of this. So start maybe from the tools, learn the tools, and then use them as a way to go deeper and actually learn how it works. I completely agree with that direction. I think that even earlier today, we saw a talk about leveraging Pixi and some of the capabilities of just leveraging BPF trace to understand what's actually happening on your systems. I'm an old systems administrator, and I've worked in networking for years and years at this point. And I would say that for me, the easy entry point into BPF and starting to learn and explore what was possible was playing with those tools, like being able to use BPF tools that kind of already existed in the community to look at the set of problems that I'm used to looking at in a different way, getting more context presented maybe at a higher abstraction, being able to understand what system calls were happening at the same time as the networking problem that I am having was happening, like being able to kind of widen my perspective with these tools. That's definitely, I think, a good place to start. Yeah, I agree. I'm not sure I have much to add. But earlier today, there was a plug for at least talk from last year. So for those that were not there, that's like a good introductory talk. And I think the BPF trace is also a great tool to getting acquainted with, what are the different trace points that are available? What kind of things can I do? And you don't need to learn all of the BPF to just start experimenting and do a few cool things that maybe are not particularly useful. But it's like the first taste. It's like a hello world that you can then move on to do something more interesting. All right, thank you. Does anyone have any questions for our panel? No? All right, well, I hope you all enjoyed that taste of Clustered. Thank you so much to the panel and the breakers and the fixers. Thank you very much. Thank you.