 Well, hello, everyone. Thank you for coming to yet another episode of Cube by Example Insider. It definitely kind of rolls off the tongue better when you say KBE Insider, but I'm your host. My name is Langdon White. I am a current professor at the University, or Boston University, specifically attached to a new unit called the Center for Computing and Data Sciences. And in a fancy new building that is often compared to a Jenga Tower, which is kind of entertaining. But I was at Red Hat for a long time. I'm pretty interested in Kubernetes and the ecosystem they're in. And the idea behind this show is that we try to interview people who are actually kind of doing the work inside the project. Because what we find with open source is that if you look at the press releases or you look at the talk to the project managers and product managers, they don't always know what is going to get delivered because you can't control what somebody does on the weekends that they might find more interesting. So that's a little bit of me and the show. And my co-host for today is Savita. And she can introduce herself. Sure. Thank you, Langdon. Hi, everyone. My name is Savita Rabanathan. And it's my honor to be here. I have been a guest of the show a couple of times. And then I had the opportunity right now to go to the network dating page. And in my day job, I am a software engineer at Red Hat and I'm focusing on application modernization. That is like in simple layman terms, migrating apps to Kubernetes and adopting the cloud native technologies. And I'm also a conveyor, maintainer and contributor to Kubernetes. So that's all about me. And I'm going to now pass it on to Andrew. Awesome. Hello, everyone. My name is Andrew Stoikis. I'm a senior software engineer in the Red Hat's office of the CTO. My passion is kind of everything open source. I work in Kubernetes as a maintainer of the SIG network policy subgroup to SIG network. And we work on new network policy related APIs. Additionally, I've really been interested in the intersection of eBPF and Kubernetes. And that's accumulated in one of my other main projects called BPFD, which we'll talk a little bit about today, I'm sure. And yeah, super excited to get going today. Cool. And how about Robby? We'll go in alphabetical order by first name. Hey, everyone. I'm Robby Cochran. I'm a manager lead of the Advanced Cluster Security team that works on runtime collection for securing OpenShift and Kubernetes using eBPF. So eBPF allows us to basically tap into the kernel, figure out what's going on, and then we correlate that data back up with Kubernetes primitives. I joined Red Hat after working at a startup called StackRox. And that project was then open sourced. So StackRox is now an open source project. And yeah, excited to talk about eBPF today. Cool. All right, Zane, how about you? Hey, yeah, I'm Zane. And I was the co-founder and CEO of Pixie Labs, which got acquired by New Relic Rarist at GM and GVP. I'm currently an Adjunct Back to the member of Stanford and also one of the main contributors and maintainers of the Pixie Open Source project, which is part of the CNCF. Cool. And so one of the first things I like to ask is what brought you to the open source world in the first place? I particularly enjoy scratcher-owned-inch stories. And whoever wants to kind of go first, what do you think? I can go. I had somewhat of a traditional path. I was actually studying biomedical engineering, super passionate about medical device design and undergrad, and then decided to change it up for computer engineering and grad school at Boston University, shout out to Langdon. And one course in particular, Introduction to Cloud Computing, kind of enter open source, enter cloud computing and started working on KeyLine, which is another big CNCF project around security and attestation. Started working on that, and here we are a few years later. Nice, nice. What about you, Zane? What brought you into kind of the open source world? Yeah, I had a pretty traditional path because I studied electrical engineering actually, but moved into software and I've always been tinkering with Linux and stuff on the side. So it was really a huge kind of open source community and rather than many, many different communities that exist, I think. And with Pixie, when we decided to open source that, that really like brought me into like the CNCF and Kubernetes communities. And that was, you know, it's great to just see the whole community grow because I've been involved with it for a few years and it's been a fun part. It's been fun to be a part of that. What would you say is the, like, you know, you said you kind of were involved with Pixie, right? Kind of both before and after you open sourced it. What would you feel, what do you feel like the biggest change was between the two different worlds for like a better term? Yeah, so before we were acquired by a new relic, we were not an open source project. We were kind of always a very community forward project. We had a bunch of stuff on the open source, like, you know, here's like our scripts and workflows, but they're all kind of tied to Pixie fundamentally. And I think one of the biggest shifts to us was after we, you know, decided to open source Pixie our entire team had to change how we've worked on the project. And, you know, I think it's like, it's very hard to articulate exactly what it is because it's like, you've got to fundamentally change everything you're doing. You know, all their design reviews and stuff, we started having to do them more in public. And, you know, at the beginning it was pretty scary for a lot of us, right? It's like, oh, we're going to put in like half unfinished work in the public and get other people to look at it. But it's something that took time for people to just get comfortable with, just doing everything out of the open. It's like, oh, we file an issue out in the open. Does that mean, you know, if we don't finish it, now it's like embarrassment for us inside and embarrassment for us outside. It's just like a very different mindset to embrace. And I think it, you know, honestly, it took us a little, took everyone on our team a little bit of time to embrace that mindset or just kind of just got the open. Yeah, I think that's a, it is a definitely very interesting phenomenon, right? Like before, before I got involved at Red Hat, you know, I was mostly, you know, kind of a boutique software consultant, right? And, you know, yeah, everyone really, really wanted to own their own IP, especially when they were paying, you know, many dollars an hour for us to build it. And, you know, but now it's kind of like, why would I make a repo private? You know, like, like I have to, you know, sometimes it was actually funny. There's a project we're working on where I kind of automatically made it public and then realized, oh wait, this is privileged. Like I, this shouldn't be out there and it's not because the code is privileged, but because the, basically the project is privileged. And like I had to go, you know, like actually make it private again. And it was a very weird experience, but yeah, definitely changes how you code. It's also a little bit of a different mindset. Like once you've kind of, you know, crossed over, like you said, it's very difficult to go back and make things private because your entire like mindset and development philosophies and everything change. Right, yeah, it's super weird. What about you, Robbie? What brought you into the open source world? Well, how, yeah, how far back should I go? I think- Oh, the beginning is always entertaining. The beginning, yeah. I mean, probably I was, it was a funny question because I was thinking about it. I was like, well, probably the first time I encountered open source was with Lego Mindstorms, like in high school or something. Oh, cool, yeah. I was like, you know, these things can be extended and there's these communities doing stuff. And then I think really what it was was like asking a question on a forum and getting a response from a random stranger and then having that connection. And that was, that was really cool. And then in grad school, I did a lot of work on us in security and using compiler tools. So I did a lot of extensions with the LLVM compiler framework. And that to me just seemed like the type of project that could only emerge through a community where everyone's adding their different architectures and, you know, optimizations and trying out different compiler kind of tweaks and things like that. So I was, I thought that was really, really cool. And then similar, really similar story to Zane working at StackRocks where we were closed source, but I don't think anyone working on a software project can't be utilizing open source tools and like pulling in libraries and things. And so we were ready to give back, but like you're saying it's kind of a whole mindset shift. So when we got acquired by Red Hat and were able to open source everything, it was an interesting challenge thinking through not only like IP stuff, not IP, but like customer protections and things like that. That was the boring side of it, but the mindset shift, doing things out in the open. And then also seeing how difficult it is to even build a community around a project. That requires its own effort and time and, you know, prioritization changes like open source issues versus customer issues and things like that. So just the working model has been really interesting to see, yeah. Yeah, that's interesting. All right, so kind of moving on to what we're really here to talk about. So EBPF, let's kind of say, okay. So first up, can you describe EBPF kind of in terms of maybe not so much what it technically is as much as like what do you feel like it brings to the table? Like why is it interesting? Especially in Kubernetes, if y'all have thoughts on that. Tartan, we have three guests, but otherwise we have to call them people, which is always awkward. All right, who's got an idea? Well, let's think about it. So EBPF is a pretty powerful, you know, very sharp knife. I like to think of it, right? Like before contributors had to, you know, write a kernel module and, you know, get that approved and update and maintain a kernel module. And now you can do a lot of the same functionality with an EBPF program, which is basically just a program that gets injected into the kernel. We make sure it's not gonna totally break and crash our kernel with a verifier. And then it allows you to do really low level, really fast things, kind of nested deep within the kernel. And I think as it pertains to Kubernetes, like a lot of it's still really unexplored, right? Like it's obviously this amazing knife. It can be used to build a lot of really cool things such as Pixie. Observability is one of the major use cases. It allows you to do super performant observability. But putting it in Kubernetes, that's a whole nother story, right? You're no longer thinking about a single node, which is what EBPF is designed for. It was designed in the context of a single node. Now you're starting to think about clusters. And we can talk more about it. I'll let someone else talk, but I think that is like, that problem is the most interesting problem. Like EBPF in itself is really complex and interesting, but then also deploying it to Kubernetes in a smart way is really complex and interesting. Right, how do you balance the single node focus versus like a cluster focus kind of idea? Yeah, yeah, it makes a lot of sense. A tail is old as time, yeah, right? I mean, it's like Kubernetes came about, so. Yeah. I think one point to add though, one of the cool things about running EBPF on Kubernetes is unlike kernel modules, which would be very, very difficult to install on your entire cluster, it is relatively easy, although it comes to its own challenges to deploy EBPF programs across your entire cluster. So in some ways you can think about, oh, I'm installing this new kernel module across my entire cluster to do a bunch of functionality without having some of the pitfalls of having a kernel module or the trouble of doing it. And yeah, I mean the... Different pitfalls, but you know. A lot more dangers, a lot more foot guns, I think. A lot less safeties. Yeah. Yeah. One way that, oh, go ahead. I was gonna ask Robbie, you looked like you had a comment to me. Yeah, I was gonna just kind of mention, I think the power of EBPF and Kubernetes also comes from the fact that like, okay, why do we care about getting data from the kernel? And one reason is that it's kind of the common layer shared by all these different types of applications that can be run in a Kubernetes cluster, right? You have different programming languages and different frameworks, but if you're looking for either security data or observability data, of course you can evaluate things in user space, but kernel space gives us some kind of framework, framework static places to attach and get data that's consistent across different application stacks. So that also becomes really useful when you wanna have complete coverage for security or observability or implementing networking stacks and all the other cool things you can do with EBPF. But that's something that I think about is it's like the kind of lowest common denominator for all these different things running in your cluster and that's a good place to tap in and get data at the kernel level. But then you have all these pitfalls as Zane was mentioning about getting kernel modules running. If they crash, at least with EBPF, maybe you get a soft lockup in the worst case, but you're not crashing super hard. But yeah, you have a lot of safety guarantees that really make it feel more like you're running JavaScript in a browser versus machine code on the processor. An internal module, yeah. And you're not having to re-roll out your base images every time you wanna change something, which is a nightmare, right? Exactly, yeah. Yeah, it's just to kind of go back to your opening point, right? Because at the end of the day, everything's a kernel halt, right? And so you have that consistency for what's happening across your environment, whatever that means. It's interesting from an observability perspective. I also think some of the other use cases are really interesting as well. Kind of auditing or error manipulation or kind of those kinds of things are also really interesting where you can kind of have, it's for like a heartbeat, we actually talked about it a lot in the tech world of aspect programming, right? Is you can kind of have an aspect component that is like shared across your environment without having everybody and their brother having to write their own individual versions. So does, do any of you have any experience with actual BPF, like pre-EBPF? I'm just out of curiosity. Like, did you come from there or did you say, was it more like, hey, there's this EBPF thing. This might be useful to solve some problems. No. No, okay. You never touched the Berkeley packet filter, yeah. Just the Berkeley packet filter. Yeah, I was just curious. I don't know that I ever actually have like done anything with it, but like, I know it exists kind of thing, but I don't think I've ever actually worked on a project. I believe if you've used TCP dump before, I believe you're using. Oh, that's true, yeah. Wireshark might use it underneath too, right? Yeah, yeah. Yeah, so, yeah. So we can claim the fame, right? Whether we actually know what Wireshark was doing or not. But yeah, exactly. Cool, so, okay. So now in your kind of individual domains, you're using EBPF for, it seems similar purposes, but not necessarily the same. So what is it that you think you're kind of most waiting for in the, in that space, I hesitate to say like product or whatever, but like in that concept or space or whatever that you really think is either gonna enable this stuff that you do to be so much better or is gonna allow you to do something new or something like that. What are you, how do you feel about that? I think one of the things that we really nice to have is much more fine grained security controls around BPF. Cause right now you can use like Suckcom BPF or something. And basically it's like here, you get access to everything. And that kind of, when you're talking to customers or something and they're like, oh, I want to lock down certain functionality. It's like, well, please trust us to make sure that you'll never execute bad things in your machines. It's just generally not a good security posture. Well, they gave us cap BPF, right? Like, isn't that good enough? I don't know. I'm gonna give you a hard time on that one. Okay, so can you elaborate on that? What did like, what did cap BPF do for you or against you? I mean, in my opinion, right? Like original BPF was like a super root, you know, rootful application. And in my opinion, it still is, but the kernel kind of tried to break some of these privileges, the ability to load programs to create maps, et cetera, et cetera, BPF maps and BPF programs under this bubble of cap BPF. And there's been a couple attempts over the years of like, even adding unprivileged BPF to the kernel, trying to add more security controls and nothing's really stuck. And so just like Zane, like us and Octo are really interested in the security aspect. Like we have customers and users who want to be able to have fine-grained security controls over their BPF, meaning allow certain customers to create certain types of BPF programs, allow certain customers to attach to certain hooks and others not, right? And rather than relying on the kernel to get stuff done, we've started to kind of tackle this in user space with BPFD. And that's kind of where we're going. Because although BPF is awesome, the BPF subsystem, you know, moves at a kernel pace. And so we're trying to move a little bit faster, especially in regards to Kubernetes. So bringing, you know, Kubernetes native concepts, such as RBAC to BPF. Can you actually elaborate on that a little bit more? What, shoot, you know, what is it, like, or how are you able to do that in a sense, right? You know, this is a common problem with kind of the kernel module development as well, is how do you do, you know, like how do you move at a faster pace, right? Than the kernel without the privilege of being kind of inside the kernel? Well, you can never be like, it's more of a deployment problem than a per-distro problem, right? So like with BPFD, our whole mindset is that we're going to be an integral piece of a Kubernetes cluster. So then if you're a cluster admin, you know that if someone wants to come and load BPF, you tell them, do it via BPFD. Obviously, if you're privileged, you can do it without BPFD. But if you kind of, you know, deploy, right, and win over cluster admins, like at the end of the day, you do everything through a central daemon, and that daemon can control security and privileges. And like today, you can create an XDP program YAML object, right, and apply standard Kubernetes R back to it. So that's kind of the path we're going down for now. And as the kernel kind of developed new security built-in features, we can take more out of BPFD and just kind of use kernel tools that are provided as we go down the road. But we see that as a whole project. That was actually kind of my next question is like the idea is that you, you know, you kind of are able to model everything you want to build in BPFD, but then, you know, kind of long-term, BPFD should just go away because it's all been kind of integrated into actual, for lack of a better term, EBPF or the kernel. The security portion, the security portion. I think that's one part of it. The other part is more of the management and loading of BPF programs, right? Like there's so many different angles. There's observability, which we've touched on. There's actually just getting a dang BPF program deployed across your Kubernetes cluster. It's not trivial, right? Like it comes with some privileged Damian sets. Every application has to manage pin points of programs, map pin points, link pins, et cetera, et cetera. BPFD tries to simplify all that and just let an application worry about interacting with its BPF program and not have to think about the management. I gotcha. I gotcha. Savita, did you have a question back? Yeah, sure. So, from what the... We have established that EBPF is like a such nice, right? It has... It can do multiple things and it's amazing and it's powerful. But it's always really powerful and useful when you know how to use it, right? So, what is the learning curve of this technology is like? And where do you get started? Like I'm a new... I'm interested in it and I go to every KubeCon and I'm like, I have 10 new things to learn and EBPF is the top of my thing. Like an extra wasm, which I've never gotten into it soon. But I wanted to EBPF because I really like security and that is what else I think that ties it together. And I used to be a platform engineer. Like so many things come together and what is the good place to get started? Like for newcomers and what is the learning curve is like? Yeah, let's ask Robbie first. Yeah, I think the learning curve is pretty... can be steep and that's because you are... You essentially can think of EBPF like a debugger where you're attaching to things within the kernel and then you're triggering events and then running the program based on that event. And so what data you're attaching to depending on what data it is that can change things. So I think there are some... a lot of projects that abstract out some of these nuances and that make it feel like there's a steep learning curve where it doesn't actually have to be hard to use EBPF. KubeCuttle Trace, which is using like some of the BPF Trace and BCC tools that allow you to run a EBPF program in a Kubernetes cluster and really just playing with it is that type of like tool I think is the best way to get started where you run a pod, run Nginx, see it's making connections and then observe that in a tracing program and that should run on any Kubernetes cluster. You don't have to worry about things getting set up and that sort of thing. So probably the best place to get started is to use the tools because there's a lot of utility in the tools just for normal Kubernetes development and then you could start to say like, hey, I wish this had another column of data or in extending from that perspective. Actually just kind of starting a Kubernetes project from scratch I think requires almost understanding where the problems are in other areas of the ecosystem and it's not as easy to do that because it is like it's not like a new application that has a playground that has things you want to build things in. It's like you really have to have kind of a targeted use case. But yeah, the kubectl trace is a good place to start. Cool. Well, Yuzane, do you have more to add? Are you 100% agreement? Yeah, I also think sometimes getting started even outside of Kubernetes can be a good option. It's just easier to do things on a single node and play around with things and I really like the BPF trace project because that abstracts away a lot of the stuff but still gives you enough of the details that you're learning and can then move on to more detail things over time. Yeah, BPF trace plus one of that. I think trying to go straight to Kubernetes plus EVPF is a big curve. Like trying to understand the basics of BPF with BPF trace and there's a ton of examples out there. If you go to evpf.io there's a really bunch of great resources there. But that's a good way to get started. Also Liz Rice wrote a book on EVPF that I still reference today. Like it's really great. And if you want to get deeper fork and pull down the Linux kernel and start reading because that's where a lot of the ground up to date stuff is going on. Obviously. I just threw the link in the chat. I wanted to also kind of circle back around a little bit. So I think we mostly heard from Andrew about kind of what you're foreseeing in the future. Zane, what about you? What do you think is the thing that is going to be most interesting for kind of your perspective on EVPF kind of coming forward or also, sorry, kind of like, you know, or OK, so if you feel like observability has at least moved a lot, right? What is what's the next thing that you think would be really interesting to tackle with EVPF? Yeah, I mean, I think there's already a bunch of work on this, but there are a lot of security use cases for EVPF. You know, I think as Andrew was mentioning earlier, EVPF kind of works on this kernel API layer so you can essentially see everything on the machine and that has a lot of benefits versus like trying to install, you know, application specific agents or code that does monitoring because this way you can guarantee there are some challenges with doing security with EVPF, mostly that EVPF, especially if you start pushing events off and the user space can be lossy. So, you know, you want to make sure you get full coverage with things like security and there are a bunch of security projects coming up, but I think it's definitely going to be an area where, you know, you're trying to monitor for loss of, you know, identifiable information, like are you accidentally sending over PII data to or are you, you know, sending over unencrypted data, like all those things could be automatically audited to things using EVPF. Interesting, yeah. Yeah, one of the systems I worked on, it was basically, you know, for lack of better term it was like a pension system but, you know, I had these interesting features where certain users could perform one kind of operation, but if they could perform another, they couldn't perform another. So, for example, if they could issue a check, they couldn't change an address, right? So, they couldn't change an address themselves and then, but I kind of wonder if kind of in the same vein, especially when you're talking about PII, like, you know, is, you know, are there places where we could interject auditing such that you can kind of see where those operations are coming from, you know, and kind of say, hey, you tried to, you know, you changed the address and then you issued a check. So, you know, you can see that you've got a lot of different options for your house, you know, and so, like, you know, even looking at, like, series of interactions being kind of what you're auditing and getting back to compliance, you know, you see this a lot in kind of the stock trading world as well. But yeah, that's you know, I would very much like to see where that goes as well. So, kind of going back to Robbie, did you have kind of what's the next interesting thing in the horizon for you? I think the EBPF, like, tool world and the community has matured a lot and so I think a lot of the things I'm looking forward are probably, like, incremental, small little improvements and polish. So that's one just, I mean, and that's really important because that brings people in, that makes it easier to onboard and then it also makes it easier to maintain because we're building these projects and then customers are starting to use them and people are using them in large production environments and then it's like, how do we, you know, BPFD is solving some of these problems of what happens when you have a lot of EBPF programs all attached to the same hook. Well, if they're all outputting the same data, maybe we could have a single, you know, a single tool that emits process data for security tooling or some network of durability that can be shared. So I think there's probably some consolidation that could happen with all these utilities. The other thing that maybe is less, it's more kind of recent news with EBPF is the core technologies which stands for compile once run everywhere, which is really it's interesting because it's a challenge that occurs if you imagine your EBPF program is attaching at the system call level, system calls are super stable. They don't change from kernel version to kernel version. But if you think of the structs inside the kernel, the task truck or things like that, those fields get moved around or there's optimizations or something added. So the structs are changing within the kernel within each build. And if you want to attach to those points, you usually need to have the kernel headers to know the right offset for your program. And what core technologies do is allow you to figure out those offsets at runtime. And it's a nice set of technologies that all come together in the same space where the compiler that emits the EBPF program, which is a byte coded program just doesn't resolve all the offsets. It says, okay, this is a I'm going to ask you to look up this field struct later when you're actually running the program. The reason I kind of mentioned these things is because this has been a huge benefit for us because we've wanted to we started with a kernel module and then migrated to EBPF and are now migrating the core BPF and in order to make all that the FALCO project works, we pre build the EBPF programs. So for any kernel that we support, we have to have a kind of a back end job that pre builds these the EBPF program so they just work in the environment and core allows us to get rid of all that infrastructure and just have a program that works kind of a universal program. So maybe, you know, for someone that's just getting into EBPF that doesn't sound that interesting, but it solves a huge problem for us to support people that haven't upgraded yet because, you know, we've been talking about, you know, EBPF, the next step of EBPF, some of the cool things are the tools that let you bring the technologies back to customers that haven't upgraded. Right, making it less clunky basically, you know. Yeah, I definitely agree a huge advantage, but often does not get recognized as such. But, you know, when you there's a big difference when it's straightforward enough that it's consumable by, you know, whatever, kind of just a general organization and it doesn't have to be, you know, the top tier, you know, fangs or the, you know, banks or whatever, you know, I think it really does, when you democratize it, right, it really does make a big difference to not only its adoption, but then also you now have so many more people who can conceivably contribute and they might just be contributing with feature requests, right, but, you know, it really does make a difference when you kind of get a lot more perspectives on the software. That's, yeah, that is I, like I said, I think I agree with you in that it usually gets kind of short shrift, but it is often very, very important. Is it core, is it core or coree? That's like, I always see it's always spelled co-re compile once run everywhere. I always say coree, but we can go core. See, I just think Java and think compile wants to bug everywhere, but you know, because I think that was the core promise of Java. But kind of related, so EBPF is actually a very you know, hot, buzzwordy kind of thing right now. The other big one is WASM, you know, or Web Assembly. Do you think there's a relationship between the two? You know, obviously they're both very hot, but like is there a model in which knowing both or using both actually brings a positive to each other? Or are they just completely unrelated and both super popular? If you have any thoughts, especially if you don't know anything about Web Assembly, that's cool too. And I will give you some commentary. I don't know that much, but I feel like it's solving slightly different problems, right? The coree technology of BPF is almost kind of like some of the WASM-esque principles, right? You can compile it on your machine and run it everywhere, right? Well, that's kind of what I was getting at is it's kind of like both of them and this I know is kind of a reserve term in the Kubernetes land, but both of them kind of model sidecar kind of mechanisms, right? Or aspect-oriented development or you know, there's a lot of different names of it over the years and they bring a lot of the same kind of feature set, but potentially in different layers in the stack or different, you know, from different orientations for lack of a better term. And I was just kind of curious if you think that there's any, you know, like I said is there a correlation between the two or could there be a future correlation because they do a lot of very similar things in very different places, but it seems really hard to have to learn both you know, and so some sort of, you know conflation at some point would be interesting. I mean, I think the one interesting area that I could, you know you can see them work together is if you have like a while some programs or something getting policy enforcement at the user space level for events that are being generated from EBPF, that way someone could write like a custom program that could still run inside of a sandbox and effectively make sure you don't damage the rest of the machine or whatever you're doing while still doing the policy enforcement that's even more complicated than just like YAML files or something like that. Yeah, yeah. Interesting. Yeah, I mean we have a similar concept to this, like in Pixi, we don't do any policy enforcement but we allow a lot of instrumentation using Pixel Scratch, which is basically Python and you know, in that case people write Python, but we actually like, you know, effectively run that in a sandbox, you don't actually have a Python interpreter and it's a very similar concept, right, but it's kind of predates the popularity of Walsum running on things. If we have that, you know, we might have considered putting like a Walsum instrumentation instead to do a lot of this work. Yeah. Yeah, that's a little bit what I was getting at. Yeah, sorry, Rog. Yeah, I was, I thought you were going to Zane say that we, it would be interesting to run the policy within the kernel, if you could run some Walsum within the kernel for like a performance optimization. Right, I mean that could be another another option, because you know, one of the things that BTF is, you know, a lot of people see like C programs being run and it kind of gives you this illusion that it's, hey, any C program works, but BTF program is much, much, much more restricted, right? I mean, it basically can't be like a turn complete program because it has to guarantee termination within some number of instructions based on your kernel version. And, you know, that causes all sorts of complications and even, you know, even writing loops and stuff and BTF can be very challenging for the, for anything but the most trivial loops. Pragma Unwrap, what is it? Pragma Unroll? Unroll, Pragma Unroll, yeah. The loop doesn't terminate, right? The problem is that in eBPF what happens is that the code gets compiled into machine code and executed and the kernel has to guarantee that that machine code is going to finish executing otherwise you will probably hang the kernel. And you can't really guarantee like any, you know, solving like the halting problem you can't really guarantee that this program is going to stop executing unless you can guarantee that the program is not during complete, effectively. So it's, it's very, very, very restrictive and what you can do even though it looks like you're just typing in a bunch of C code. Right. So we talked about kernel and a lot of like we don't have to code or write kernel module strictly and take advantage of eBPF and do awesome things. And when I think about kernel, I think about Linux, right? This eBPF work on Windows. Do we have support for Windows? Because like Kubernetes has started supporting Windows and whatnot so and there are a lot of students and developers still use Windows machine and if they want to like try it out, I know like there is always like you can put a VM and you can do things in there but is there a plan for a need of support for Windows? Oh. Yeah. So I, you know, I think there's, I think a language posted there are some BPF for Windows available or Android posted. Sorry, there's a BPF for Windows, but I think one of the challenges with any of those things is that when you write BPF programs are so tied to Windows, like I can say for Pixie when we're developing the BPF programs, we were just like here's the kernel source code, what do we have been to, right? And it's very, very difficult when, you know, there's so-called BPF support but not necessarily any of the same APIs available. Although I saw more thoughts on it. I thought in the old days, I thought BPF did get related to Windows like a billion years ago for that kind of same problem, especially in the days when we were looking for much higher levels of particularly network interop between Windows and Linux land or really Unix land in those days. I actually think you're right, there is a packet filter thing for Windows for BPF but, you know, I think one of the things is that BPF has kind of like outgrown its original. Oh yeah, yeah, yeah, totally. At least the underpinnings are there. I guess that was kidding. Yeah, I mean, I think like fundamentally, you know, actually I'm trying to remember the name is, but Windows has something pretty similar to EVPF that they have natively. But again, like I think the main issues are it's like how do you make your programs portable if APIs aren't the same on the other side? Right, right. Andrew, are you familiar with this GitHub project or is it more of the you found it with Google? I have known about it. I am not familiar with it. I am no Windows whiz at all. But I know like in Kubernetes like Windows support, I have some friends working on Windows support for a lot of different things. So it's definitely something that I'm sure people are talking about, right? Yeah. Well, one of the reasons that I think this it's funny like kernel level stuff crossing over into Windows is often easier because way back in the day in like the early 2000s, NT was not very prevalent in the data center. And in order to make it more prevalent, it needed to interoperate with Unix and Linux much better. And while a lot of that kind of died off there's actually like kernel headers that somehow actually get into the Windows kernel but look like Linux kernel headers. And that's how things like WSL work and how actually on the Windows phone how like Android apps would run on a Windows phone. So at the lowest levels of kind of Windows, there's actually more, you know, sometimes easier integration points because of this longstanding legacy of NT working with Unix. So that's why it wouldn't entirely surprise me that we could see that going forward and that being something that's interesting to Microsoft, right, is making it so you could have, you know, whatever like, you know, your observability actually work across Windows node and you know, a Linux node or whatever. Yeah, I think as Zane was saying the challenges of I haven't done any EVPF on Windows but I'm sure that the EVPF program would need to be modified to run on Windows. Just like I can't write an EVPF program for an x86 kernel and have it run on a kernel. I've never tried that and I'm pretty sure that's not going to work. And but the user space can still use the same interface to interact with the program. And so I think that's the advantage with using EVPF on Windows or that sort of things like we could, you could have a more common interface with the kernel or with the Windows kernel. Yeah. Purely in the, like because you have to solve the problem already with x86 versus ARM, for example. As you know, what's the difference, right? Yeah, in the beginning of this year we actually ported Pixi to work on ARM in addition to x86 and part of the challenge that Robbie was pointing out is that you can't just like directly translate it because a lot of the struct offsets and stuff that they use, because part of the problem with EVPF, right, is that as long as you're monitoring like the system call API everything is great but as soon as you start poking into like kernel structs you start effectively breaking your contract with the kernel that the API layer remains unchanged and now you're starting to poke into some of the internals of the kernel and one, those are definitely not going to match with Windows unless there's some facade layer that gives you the exact struct offsets and as Robbie was saying they're not even the same between x86 and ARM usually because there's like additional fields that are added that move the offsets around so that's one of the challenges that we have to deal with as we ported Pixi to ARM in addition to x86 So Zane, did you have to like when you went under that transition like are you maintaining two different programs or some of your core functionality or are you just maintaining different header files like with the correct offsets different header files and then we added like a facade for some of the things that we needed like test structs themselves that are slightly different and then yeah and one of the things we did is that we got all the stuff to run on Qmoo so we could run all the tests across different processors so that was pretty nice. Another open source project help out there. Cool, that's awesome. Some architectures even use different system call setups. I believe it was for IBM Z they use some their kernels use some old system calls that haven't been migrated to x86 so it's like it goes back to 32 bit x86 like the socket call system call which multiplexes all of the networking system calls and so when you're migrating to different architectures or platforms a lot of interesting stuff can come out of it would work because you're making assumptions about where the data is expected to be on a particular platform. Yeah, I think the biggest difference for us that was annoying over there is that the call order for functions and the way arguments are passed is different so when you're trying to like intercept function calls you're like oh you know this is over x86 because they're going to pass arguments in different ways Yeah, because I assume you're using a lot of kprobes and trace points where like trace points are a little more stable kprobes are kind of a nightmare Yeah, and making all this even more interesting is people are starting to use mixed-apology Kubernetes clusters So running some having some ARM nodes for your monitoring and then you know x86 for your other workloads and different sorts of things like that so that makes it even more fun to support these mixed-apology clusters. Yeah, I mean what's really cool over there is like the ability to deploy multi-arch images because that's what we deal with Pixie so like for all of our components it can run either on ARM or x86 Yeah, we basically use multi-arch images to switch between them Those just have a layer that's like different binaries one for x86 one for ARM Yeah, when you basically pull the docker manifest it basically points to other images Ah, okay, cool It's kind of like another layer of indirection Yeah, so I was going to kind of ask that is it like to the best of my knowledge is there you know this doesn't really exist but do you foresee a like an automated deployment tool chain something or another for these eBPF functions into clusters that are multi-arch or you know like in Robbie's case in particular multi-cluster you know kind of spread out all over the place because there's no particularly good way to distribute that Well, we've started BPFD packages, BPF programs and OCI container images. So following with what WASM does, WASM has written their own OCI distribution spec to distribute their WASM binaries We've kind of piggybacked off that written our own and BPFD to distribute BPF programs so like one day multi-arch yeah, why not Yeah, I mean there is multi-arch deployment with OCI That's what I was basically saying is that there's basically a manifest file that tells you oh is this an ARM pull the ARM binaries X86 pull the X86 binaries Now if you take a look at like the pixie code for example there's actually no difference in the BPF programs between ARM and X86 because there's some if-def and the header that gets compiled and the last level of compilation which converts the BPF code down to you know the instruction set is done on the kernel itself so it'll by default work So you are compiling on the kernel You're not pre-compiling your BPF programs today, you're compiling on the fly Yeah, that's right Pixie still doesn't use core core because we generate a bunch of we actually generate a bunch of BPF code to intercept different types of applications When you say generate do you mean in real time or do you mean in deploy time Well pixie does it in real time so once it's deployed if it sees a new application coming on it needs to generate a program with the right offsets and everything that we need I gotcha We typically watch for new programs coming on if it's a go program we do different things if it's a program that uses the open SSL library we add some additional trace points for that so there's a whole bunch of dynamic stuff that happens in the backend That's interesting So do you have users who are unhappy about then you're shipping basically a compiler Do you have any users unhappy on the security side of things with you compiling on the fly? I don't think people have been unhappy on the security side because that doesn't really change anything from the kernel verifier perspective because the kernel still verifies it and make sure that you're not going to mess with the kernel I think the one annoyance is that our images basically ship at all we have with it And we at BPFD are trying to push people away from that obviously you have a bunch of constraints but we really focus on query and it'll be interesting to see how core kind of intersects with your use case How do we make it more portable? Our goal is to eventually move to something like core because that'll make a lot of things easier for us It's just that I think when we started looking at it last time it was just not didn't have the capabilities that we needed back then but it's obviously an amazing project And it'd be a lot of work right? It'd be a lot of work A ton of work That's kind of the barrier to entry we're running into with BPFD is like all these applications their stuff works today and it'll work with BPFD but we have to convince users like you that like the benefits of using something in the middle is the cons of having to implement implement the work kind of cool Sorry, I was just looking for a link to Cori that here But I'm doing the same too on the side like if we can just share it out and I only found a blog post but yeah, people can It's built on the kernel BTF basically is how the internals that it works so I like to like when I think about Cori think about BTF some low level docs there Yeah, it's kind of more like a part of BPF rather than an independent project per se Yeah, so let's see I was I was going to try to finish that up but then I got distracted So kind of final thoughts like we're almost out of time you know what are you you know why are you going to stick with BPF presuming that you are with EVPF over the next like two years like what remains so interesting about this particular kind of technology that would be interesting to others or not but you can get my idea Let's start with Robbie Robbie So I'm excited about like some of the stuff we've been talking about all these you know new features and I didn't even get into the challenges of like what's the overhead of running an EVPF program in your environment and so it brings all this value but then you know how do we find bottlenecks how do we improve performance and I think there's a lot of like kind of green field for performance tooling for BPF like you know this is running in a different domain how do I measure so I yeah I'm excited about BPF becoming more commonplace and more people understanding it and you know all the new features that are coming on as well with enforcement we didn't even get into LSMs and yeah some of the other interesting things security things that can be done where you're actually able to embed some small little policy logic into your kernel and it's super performance and you know you also get good feedback and insights so yeah I think there's just just a lot of stuff going on in the community and excited about all of it got it cool how about you Andy Andrew sorry no it's totally fine for me it's just a really really exciting technology I started as a Kubernetes developer I did everything and go like everything and go that's how I learned program I love Kubernetes I love the ecosystem but you get to a certain level where you can't do everything and go and you can't do everything at those upper layers of the stack and BPF is so cool because I don't want to be a kernel developer but I can learn a lot about the kernel I can now interact with the kernel I can basically write programs that are running in the kernel and get some of the same functionality without having to sit there and iterate on kernel patches which is super exciting alongside that I think the community is still super young with the BPF and it's a really exciting technology but there's so much yet to be discovered so it will be really really exciting to see how Kubernetes as a community adopts it and how we can take a bunch of go developers who are really good at go and say okay let's start thinking about BPF and we do a lot of Rust but we didn't talk about that either at all but Rust we can write EVPF and Rust which is super fun check out Aya you can write kernel modules in Rust now yeah I love Rust it's awesome how about you Zane what do you think before we were talking about how everything in EVPF moves at kernel speed I think one of the cool things about EVPF is it allows a layer of it to basically move a lot faster than kernel speed this is kind of just building off of what what Andrew was saying is like now you can write a program that basically does a lot of the functionality a kernel module can do obviously can do everything kernel module can do but you can do this and iterate a lot faster because in some ways the risk of breaking the kernel are much much lower because well the verifier should guarantee that I'm not saying that there aren't things that you can do that will completely completely break things but it is harder than with a kernel module where you could just cause a kernel panic pretty easily so it's pretty exciting from just like innovating out of the kernel functionality by working on this EVPF layer and I think that is pretty good place to be cool yeah totally that totally makes sense I think it's really interesting I've been a long standing fan of I keep bringing up aspect programming or having functionality that is kind of available to the entire environment is I think a really interesting development model of an architectures which is kind of the cloud native development world in general and so I think they don't get enough if you go to school and take a computer science class or take a set of them they don't really talk about all these things that are kind of on the side of writing your python script or writing your website that are really quite interesting and really important to making sure your operation really can perform at the levels it needs to perform at especially when they get complex right so let me just throw the book that somebody mentioned earlier but I don't remember who brought it up in the first place but Savita dug up the details on it thank you Savita and make sure I typed your name correctly but I'm just going to throw that in the chat so people can check it out if they want to but with that let's say that's so much for coming we really appreciate your time unless there were any other final thoughts that you wanted to add cool all right Savita thank you for a co-hosting yet again and hopefully we'll see you all sometime soon in the future maybe if you're at KubeCon in Chicago hopefully I'll be driving around interviewing people again and we're not sure what kind of car it's going to be but there's a couple on the table that should be a lot of fun so we're excited to be playing that out so maybe we'll see you all in Chicago maybe Paris or next time on the show bye all thanks everybody