 Hello, hello, hello. Good to see you, Ben. Hi Ted. Good to see you too. Yeah. So we're going to kind of just go over sort of a CNC update based on what we saw when we were at cube con. And of course, because we're observability geeks, we're going to do it from the perspective of observability and open telemetry in particular. So I thought kind of a fun way to do it would be to, to sort of do a little quadrant plot here for our audience. The vertical access is, you know, how novel was this announcement or update versus it being maybe kind of boring or nothing new, but actually still really important. And then the left, right access is sort of how relevant is this announcement to the open telemetry roadmap. This isn't like how important or not important it is just, you know, relative to open telemetry. So how does that sound to you? I like it. I would say, you know, I've, every time I've gone to cube con, it's been fun. And also it's been apparent that there's like a strong newness bias. So I like the vertical access as a way to remind ourselves, especially the button port and at the bottom, like there's, there's not like always a lot of interest in things that actually are, you know, pretty critical, but aren't newsworthy at some level. So I like this. I think it's good just to help keep us, keep us honest. Yeah. And actually our first point on the plot is, is so off-road and so novel. It's not even on the screen, but while we were at cube con, I got to go to a red wings game since we're in Detroit, which is actually, I think the first time I've seen professional hockey in person. And it was a lot of fun, even though the red wings got absolutely thrashed by New Jersey. Hockey is kind of fun. It's like condoned violence, you know, but very graceful at the same time. And I don't know, it's just bad for TV. So no one watches it, but that's pretty fun. Yeah, it's all good, but slightly more relevant. Our first item is a hotel unplugged. So this was like our community open telemetry day that we had as kind of a zero day at cube con. And we had a lot of attendees. It was really good. And we did kind of like a lot of road mapping to see what was, was interesting to the community and get a lot of feedback from people. We did a bunch of breakout sessions. It was really fun. So this is maybe more of an announcement, but yeah, Ben, do you have thoughts? Well, I was going to ask you, like, you know, what do you, what, what, and what was your take on the, like what were the main themes from people who are end users of hotel who are there, right? Cause I think hotel, it's interesting project. It's like a ton of velocity and a lot of committers and a lot of people who are involved in the project, people who are actually making a lot, who are working at hotel more or less 24 seven, mostly work for vendors. And one of the things that's nice about cube con is that it's a chance to kind of get physically in a room with people who are on the end user side. And as curious for your take on the, the themes from an end user standpoint. Yeah. It was validating in the sense that what, what people were asking for is, is pretty similar to our existing roadmap. You know, we do have like an end user working group with an open telemetry now for interviewing end users and kind of making sure, but the main things is there's, there's more and more people showing up who are using open telemetry in production. People were showing up kind of doing novel things with it. So people are using it enough that they're building systems on top of it. We definitely heard from number of the end end users who are building kind of like a high level, like business analytics on top of open telemetry, which I thought was really cool. And the stuff people want. There's big, a lot of interest in rum or client, client side telemetry. So browser and mobile telemetry, which is the thing we're working on. And people were very excited about op amp, which we'll talk about in a minute. People also very excited about a query language. I don't think we actually have a query language on the roadmap here because it is not something that was announced at keep calm because it's not something that exists yet. But I thought it was. This is maybe I'm actually interested to hear your thoughts about this been, you know, we've been kicking around this idea of like we've standardized telemetry, but the other side that's not standardized is, is how do you query the data? There's no standard query language and a lot of people are asking for it. But for, for different reasons with different ideas about what what it might be. So what do you think? Yeah, I think, yeah, the query language is like the third rail. I do have feelings about this. I'll call them feelings almost first and thought second. I get it. I get why people want that for sure as a thing. And I mean, you know this because we talked about it, but like, I actually think it's a really good idea to have some sort of, you know, hotel like project that's more devoted to queries or analytics, I guess on the data. The concern I have about it is twofold. One is that hotel. If you had to say whether I mean nothing can be perfect. If it was either too wide or too narrow, I'd say hotel today feels almost too wide as it's right. It's like the product is thriving in a way, but like we're still not there yet on, on kind of meeting or exceeding the level, the ease of adoption of a proprietary agent, you know, which I think is basically the, the bar, right? To be set just for getting the telemetry out. And yet, you know, there's, I think it totally makes sense and does it should be in scope for hotel, but things like rum and profiling and stuff like that are kind of like diverting our focus from that initial objective. I think of just making telemetry built in high quality telemetry built in. So I, from a scope standpoint, it kind of freaks me out, but the other piece, which I think is more fundamental, like even if we had infinite resources and stuff, I still don't think we should have an analytical query language built into hotel. I think I made some comment about TT that what's now called the hotel telemetry transformation language, which is in the collector, it's more of just like a, you know, it's, it's just like stream processing munging data from one format to another kind of thing. Like that, I think makes sense. But, but if we start getting into the realm of actually analyzing the telemetry, we're going to get very close to the use cases for like actual observability. And I've always thought of the telemetry as the base layer of extraction and then there's like durable storage. And then there are all these use cases above it. And if we bring those use cases in, we're not far from having hotel become like an observability, like full stack observability project, which I do not think is a good idea as one project. It's going to, it's going to get very monolithic if it isn't already. And the only thing that gives me pause about this is that the elephant in the room for a lot of end users around observability in general is that most of the telemetry data is never used for any purpose ever. And yet, you know, you're paying a cost to instrument it. And then of course, to send it over the network and store it. And, and to fix that, to really close that loop does require some kind of feedback mechanism between the usage of the telemetry and the instrumentation side of it. The query language, you know, if combined with configs code and a bunch of other things could be a really important piece of building that feedback mechanism. So I recognize that, but I still don't, even if you believe that, I don't think it requires like, I don't think it requires that that project be part of hotel. Anyway, the TC, I think had a ruling about this recently, right? There was like an issue where this was discussed just straight up, but I think the decision by the TC and I think ratified at the TC levels like that, we don't want to bring this in scope for now, but in hotel, but I do think it's an awesome idea. And I'm actually a champion of it as an idea. But I just don't, I don't think it's going to be good for hotel longterm to expand into the analytical side. Yeah, I totally agree. We want it for various projects around open telemetry, but they're kind of like weird, maybe weird is not quite the word, but they're like edge Casey projects. And as soon as you say you have a query language, people are going to assume you have a database and a storage layer and dashboards and all of these things that we are super duper not interested in. Yeah, because you can't standardize that stuff. Yeah, but you did have a good point around a feedback mechanism, which gets to probably the most popular topic that showed up at hotel unplugged, which is op amp. So for the people don't know op amp is a protocol we are working on for building a control plane for the open telemetry collectors and clients. So this is a way some centralized controller could push out rules and configuration changes to all of these observability pipeline components without needing to restart them. And so, Ben, yeah, what gets you excited about this? Where do you see the big wins here? Yeah, I mean, I do think it's really important. It does have a lot to do with the stuff I was just saying a minute ago, right? This idea of there being like some kind of static fixed set of like the correct telemetry for any particular piece of software is total misnomer, especially when you're dealing with software that can be reused and embedded at different layers of the stack for different purposes. Like you might want very different verbosity levels for different deployments of the same piece of code, for instance. And op amp is on the critical path to fixing that in a principled way. So I think it's one of the most important projects in hotel in that it allows us to get line of sight to solving the feedback loop problem I was talking about earlier without becoming proprietary or bringing in too many vendors specific primitives. I mean, LightStep has been pretty involved with this, but we're working with a bunch of other vendors, which I think is important, right? Just to make sure that it doesn't end up being just a thin veneer over some vendors particular requirements. So I do think this is really important to address that ROI issue for telemetry, among other things. I mean, we make a little more concrete for our audience. Like, do you mind touching on sampling here? Because I think there's a lot of things you can control, but controlling sampling, quote unquote, is probably the most important. I'm not sure if you agree with that. No, I think that's, yeah, I mean, I think that's the first priority. So yeah, in essence, I think it's most important right now. Yeah, I mean, it's pretty simple really. It's like, if you try and capture every detail of every transaction flowing through a system all the time and you are running a kind of planet scale workloads, it's not that you can't do it. I think it's actually, it's a misnomer to say that it's going to affect performance necessarily if your application, but it's just really expensive. And so you end up looking at what you're paying to just transit all that data to your durable store of whatever that is, whether it's a vendor or not. And it's probably not worth it. And so the natural thing to do at that point is to say, well, let's not keep all of it. So you sample on a transaction level. So you keep some of the transactions. You discard the rest, the dapper paper that, you know, I helped work on at Google, the sampling rates very low. It was like we kept one out of 1024 traces. And then we did another 10x cut before we did the global. So it was like really one in 10,000 transactions. Was actually globally, durably stored centrally, which is like fine actually for web search or something like that, where you had some God knows how many queries per second, but it's not fine at all for a low traffic service. And so you end up in this predicament where in order to make that, if you're going to do kind of global sampling, you, you know, you're forced to these really low sampling rates, and then your low traffic services become invisible. So something like op amp allows for sampling to be more dynamic. And you can have different sampling rates at different points in the stack and have that controlled in a way that's again, you know, vendor neutral and portable. That's the kind of short version as a, as long as I'm talking, I will say I personally wish that there was, that we were a little further along the sampling thing. So we could move to the next step, which is controlling verbosity as well. Like in a perfect world, different transactions aren't just sampled yes or no, but you have this whole spectrum of verbosity from like, we propagate context to, we keep track of like the basic, basic method calls to, we keep attributes to, we store like a detailed log to, you know, maybe one in a million transactions, you record like every function call or something like that. Like it would be so powerful to have once an hour, just an absolutely fine grained, capture everything transaction trace. And there's no real reason we can't do that other than that. We just haven't implemented it yet, but, but verbosity and sampling need to go hand in hand. And, you know, we're, we're still getting to the point where we're just getting basic dynamic sampling implemented, which is hard. But as soon as we do op amp could also be helpful for controlling sampling and verbosity as a pair, which I think is actually pretty exciting. So anyway. Yeah. No, I totally agree with that. And I think, yeah, the one bit of color I would add there is like people often ask me is like, well, what is the correct sampling level? And, you know, the answer is always like, well, it 100% depends on what you're doing with the data, right? Like different techniques require different, different volumes of data, different sampling levels, different sampling techniques. So I think I've heard you say once in the past, it's like humans need to be out of the loop when it comes to managing that stuff. And I totally agree. But moving on. The other thing that we were really proud about and has been really helpful to people, but it's not the most novel idea in the world is the open telemetry demo project. So this is maybe just like a quick shout out. This was a popular project among people, but we've got a lot of people that are interested in it. You know, your, your classic kind of big distributed service web store that people can play around with. It includes some feature flags to enable different forms of like problems in the store, like memory leaks and things like that. And it's all instrumented with open telemetry. So people can stand this up, turn on some problems, turn on some of the things that they're interested in, you know, whatever tool they're using and kind of get a sense for like how to use open telemetry to solve problems. And if you're just interested in seeing what the telemetry output looks like in your system, this is a good project to, to stand up. It is really cool. The other thing that I liked about it as a project is that it, it attempts to. You know, to have very high coverage of the languages. And it's like, you know, it's like, you know, it's like, you know, you know, you know, you know, you can go or something, right? It's like, oh, using PHP. Well, first of all, I'm sorry, but also, Hey, good news. Like we have some example code for you, right? So, because that's something that people can tend with. It's like, yeah, it's, we want to adopt hotel. And it's no problem in our absolute green field stuff, but we also have this other thing that's running over in the side that we need to trace. And we don't know how to integrate with that. But we're looking at a lot of integration targets. Yeah, yeah. Yeah, actually a fun side effect of this project is if you're looking to instrument your own services, you can go into the demo project and crack them open. And we tried to document it, make it very clear, like how we are adding open telemetry to these services. So it's also a good reference. For people. Yeah, it's sort of like this in some way. I mean, it's not really advertised as such, but it's almost like an integration test for the breadth of hotel from a surface area coverage, like platform standpoint, which I think is, is, is really, yeah, it's super helpful, I think for pulling, like working example code in a heterogeneous system. Yeah. Yeah. And it, the web store sells telescopes, which makes me really. Yeah, that's what I actually, I didn't actually go on. What is this thing doing? I like played around with it, but I wasn't, I didn't actually receive a telescope in the mail. It doesn't, it doesn't. Yeah, I think we'll have to do some, some digging into why that function is not working. Okay. All right. Anyways, speaking of verbosity levels. Something that has also been kind of kicked off that people were interested about at cube con is profiling. There was a lot of interest in, in profiling. There was also interest in EBPF. We'll talk about an EBPF project in a minute, but I was noticing people were kind of like conflating these two things a little bit really in their head. Yeah. Because it's just like low level kernel, something, something, you know, automated something, something. So I'm curious, like what, what do you think about like profiling and, and how it can be useful in the context of like distributed observability. Yeah, you know, I don't mean to sound like a total, you know, party poop or something, but the profiling thing kind of freaks me out a little bit. And I'll tell you why. So, and also, I noticed we haven't had any questions in the chat. If people are sick of hearing us talk to each other, you can fix that by asking the question of the chat or in the Q&A. We'll take it as they come. So here's the thing. Like when people say profiling, I think what they mean, I mean, knowing what hotel profiling actually does, it's, it's like more about, you know, really widespread sort of CPU profiling, which is fine. It's definitely a thing. The other type of profiling that people want to do is more like, you know, critical path analysis, which you honestly don't need hotel profiling, or that's just like capture traces automatically and lights critical path and summarize it in some way. And there's nothing wrong with profiling. It's definitely an important thing. And I, I mean, there was a thing called GWP when I was at Google in the 2000s, it's called Google wide profiling, which saved Google gobs of money because it allowed you to figure out globally across the state, like, you know, which functions are using the most CPU. Now the thing is, and this is the thing that kind of bugs me about it is, and why I feel like a grump. It's not that it's not valuable, but it's a way of saving money. It's a way of saving throughput. It really like doing profiling. This doesn't really help that much for latency at all, especially for tail latency, which is usually what matters. And sometimes I feel like people are like, Oh, I want to make my stuff faster. I'm going to profile it and like sort of that. I mean, you are making some, you're making the things that use the most CPU faster by doing this, but that actually is totally different question than optimizing a critical path. In fact, you may be making it worse in an odd way because sometimes the sorts of optimizations you do to save CPU cycles actually increase tail latency. And so I'm whenever people say profile, I'm like, are you sure this is what you want? Like if you want to make your stuff faster for end users, this actually literally might make it worse. However, if you're trying to like save on your Amazon bill and reduce CPU suits, this is a fantastic project to pursue. Sometimes I wonder a little bit cynically whether hotel is necessarily the right way to be doing this or if it would be better to just have, you know, something like P prof running, you know, continuously in the background. But anyway, I mean, I realize it's a little bit unfair the way I'm characterizing it and that it's not, it's not quite that way, but I have a little bit of a trigger around the word profiling because I think people can conflate CPU profiling, latency profiling, and they actually in some ways are opposed to each other. Yeah. I mean, I can totally see op amp and distributed tracing instrumentation being a useful trigger for stuff like P prof. Yeah. My trigger around this stuff is like the thing people keep being is like, yeah, so if there's a problem going on, I could like dynamically enable P prof or this like other stuff. And I'm like, that sounds really great because, you know, restarting things when there's a problem is bad. You lose data. But on the other hand, like being like, hmm, this machine's acting really weird. Let's like totally increase the workload on it and see what happens. So I agree there's some work here. But, you know, the thing that I've noticed when people are asking about this, you're talking about CPU profiling and all this. And I swear what a lot of people seem to want, and maybe this is why they get confused with EVP F, which also doesn't do this is just something that's more like, like stack traces or just like a lower level, a higher fidelity kind of information about what their system is doing. So they have like this operation is failing. And now I want more detailed information, but it's not like exactly profiling or, you know, EVP F style, you know, hooks. Yeah. But, but that I just noticed talking to people at KubeCon, that seemed to be in general, maybe the most common thing people wanted. I want more detailed information about this operation sometimes. Yeah, that's a really good point. Yeah, people are, they want as a backtrace or maybe even the distributed backtrace. Yeah. Profiling is a technique that requires that, but it can be used for other things. Yeah. Like, you know, exception tracking and stuff like that, right? So, yeah. But we do have a profiling working group now. And there were a lot of people from that working group who are at KubeCon. So if anyone in the audience is interested in this stuff, you can join that open telemetry working group and let them know what, what exactly it is you want. Yeah, I think it's a, yeah, I mean, my last word on profiling, I will just say is I don't want to be, I don't want my, I don't want to seem too grumpy about it. I definitely think it's important. It's just this locus of several different use cases and then several different implementation strategies. Yes. And people are coming at this from wanting either the implementation strategy of the backtraces or certain use cases. And then they're all kind of jammed together into this, into this, you know, into this coalition and, and nothing wrong with that. I think it's sort of inevitable, but, but I just, yeah, I, I sometimes say, I worry that like it's getting pulled in a lot of different directions, but anyway. Yes. Yeah. Likewise, it's kind of a confusing blob of stuff. All of it's useful, but it needs, we need to kind of, we need to use more words maybe and kind of tease this stuff apart so that we can have a clearer conversation about it. So ho hum, but on the roadmap and super important, there's more and more Kubernetes level instrumentation and telemetry coming online. I know David Asheville is working on a lot of this stuff along with a lot of other people. And one thing they announced at KubeCon was Kubelet tracing. So for people who don't know the Kubelet is the little controller agent that you run on all of your machines that are running application containers for you. And so it's the thing controlling the kind of container life cycle. So your Kubernetes schedulers basically talking to all of these Kubelets and managing the sort of global state of all of your services. And they've added tracing to this component, which includes being able to have tracing around the life cycle of your containers. So it seems useful. Yeah, for sure. And it's also just, it's exciting to me because this has been, you know, a long time coming. I mean, I remember before open telemetry, when we were just doing open tracing, like super early days, like 2016, maybe 2017, no later than that. I can't remember exactly having a conversation about essentially doing this. I mean, like actually instrumenting Kubernetes stack internally. And there was a lot of interest in doing it, but it was just going to be really hard. And then there was just, you know, you know, you know, there was a lot of interest in doing it, but it was just going to be really hard. And then there was just also concerns about, about Kubernetes taking dependencies on anything. It's like it had to be a certain level of maturity and everything. And it's to me, this is just like a, just another pretty big, bright check mark on hotel from a stability and legitimacy standpoint. And it's taken, as I said, like it's been more than five years since we first started talking about doing this. And it's a really important step for the project from a validation standpoint. And then it's also, I think a really good thing for just overall Kubernetes introspection. So yeah, it's awesome. Yeah. Fun fact, this is one of the use cases that got me interested in distributed tracing and open tracing and all of this. I was working on cloud foundry, but working on, you know, distributed system schedulers, different scheduling algorithms and, you know, trying to make sure they actually adhere to what they claim they are doing. And it was really annoying to try to do that without some form of distributed tracing. And since this needed to get, this platform was in use by lots and lots of different organizations who all wanted to shovel the data somewhere else. That it's kind of funny that we're finally now getting this for Kubernetes, but this was one of the things that, you know, it really got me reading the dapper paper and all this other stuff because it's trying to implement it for this use case. Oh, interesting. Yeah. Fun fact. Okay. Super, super boring and super, super on point. Logs, open telemetry logs. And long last becoming stable. Also events because we decided to put logs and events together. But as a slow clap. Yeah. Yeah. I feel like literally don't know what we can say about this. The logs, their structure. That's nice. I actually think it's a, it is a, maybe ho hum, certainly important, certainly on point, but no small achievement. Right. Like, I mean, part of the reason that we started with tracing. Was that it, you know, it was like a gap. And needed to get, we needed to get the purpose, the original impetus like for open telemetry was to fuse open telemetry logs. So tracing and part of that was to deprecate them both as soon as possible. So tracing needed to get down quickly. So that was true. But the other reason that tracing was appealing was that because it was so green field, we didn't have this messy. Multiverse of like, you know, Existing quasi standards to deal with. And in logging, you know, that's, I mean, it's just an enormous, enormous issue. And I, I do want to applaud our frenemies that's been, you know, I think it was the work kind of driving this forward. I think it would have been difficult without some of, you know, their participation. And also, I think it was awesome. A lot of the, you know, elastic was nice enough to, to donate ECS and stuff like that. But, but it's, it's a, well, you know, it's not difficult at all to imagine standardizing the idea of a log line. Like we're going to log something, but then there's this next level of detail of like getting into an approach to the idea of a log line. But it's sort of clear where this gets powerful. And I think actually like the incorporation of logs into hotel and the semantic conventions has. Like more to do with the eventual success of like unification efforts trying to serve ability than just to anything, right? You can automate the correlations in a principled way between these different slavers of statistical and event based. And I think that this particular milestone is just So it's a big deal, even if it doesn't seem that way and seems ho-hum. I think it allows for observability tooling to do things that people have been asking for for decades, you know? So I think it's very exciting. Yeah, and it also like finishes our original road map. So I'm excited to get this thing done because it means we can now move on to for one convenience, like you said, it can be like a real goal is not just to like make it all work and stable, but to make it make it easy for people to stand up and use so it can get in everywhere. So yeah, we can kind of go up the stack and it also means that like coordination at this level is so difficult, right? Because it's like you need to have a lot of collaboration and buy in from people who aren't like at cross purposes exactly, but they do have different needs and it's just consensus is hard, especially in open source. But I think having consensus at this level allows for a more parallelization of the development effort for hotel as well, which is just welcome, I think, from a philosophy standpoint. I will say defining like a standardized log format was way less fraught than defining the metrics. Yeah, why do you think that is? I don't disagree. Why do you think that is? So I have two reasons. One, the metrics issues, some of them are just like genuinely difficult, tricky nonsense, like stuff around like heuristics around things like histograms where you just have like off Y1 errors with your bucketing and just these like very subtle distinctions between the different various metric systems that are out there that are mathematically incompatible with each other, you know, that adds a huge amount of complexity. You have this like a push versus pull concept in metrics, which again, like both approaches are fine, but the tabs versus basis of observability. Yes, exactly. And we're like, fine, we'll do both. But that actually is like, like a big extra dollop of implementation complexity. So and on the lock side. Yes, there are different standards, but also it's like all the logging companies and backends, they're so used to just getting a fire hose of absolute crap set to them, you know, just like just blobs of strings that they have to parse. And like, it's almost like they're just like, yeah, as long as it's structured somehow, we don't care. Like, I think that's all correct. And the other thing I would say about metrics that I think make them particularly difficult is that I'll try to be brief about this, but metrics are all trying to encode some kind of query. And then you're actually you're the metric itself is like the output of a query, even the most basic example would be like a counter. Like you're doing essentially you're doing like a group by some query in a straining fashion in your code. And so you're counting how many times something happened. Okay, fine. It gets harder than when you're dealing with a query like a distribution query, which results in like a histogram output. And so like, that the metric is doing some kind of partial computation. And then the rest of the computation happens later on in some other system. And I think the reason that people have so much difficulty in collaborating or that's right, it's not the right work in achieving consensus around representations and stuff like that is that the query models actually kind of break if you change those representations. Whereas, I think for logging data, it's like you said, first of all, yeah, totally like people are people implementing logging systems are already dealing with an incredibly messy brownfield of heterogeneous data. So they're used to it. But also the data is basically event oriented, right? It's like it's much less common to see logs that are encoding the output of queries, like they're more like the input. So you don't have this additional level of like, people aren't thinking about it this way when they're debating these things, but for metrics, we're really debating we're debating the internal representation of a complex streaming query that's like midstream. So like no wonder that's a mess because like it breaks it breaks the computation if you change the representation. Yeah, that really happened with logs, I think. Yeah, it seems really seems like the fundamental axiomatic building blocks of metrics are more complicated than logging. Logging is just like here is an event of some kind. And as a telemetry system, we don't care what you do with it. But with metrics, like we can't get out of caring about what the function of this thing is. And that's in a world where people have been inventing functions for like 20 years, 30 years, longer, like it's messy. Yeah, for sure. Okay. Moving on, we got I think maybe two more items on our list. So now we're getting into the things that are like not on the open telemetry roadmap at all, but actually like interesting project. And one of them, speaking of EBPF is the Cilium project. So this is a new project that's been added to the CNCF focus on EBPF. I should add this is not an observability project. This is this is more of a efficient networking proxy kind of project that's really leveraging the fact that EBPF is a low level language for programming the kernel. So it's an efficient way to program your networking stack. But a side effect of that is you can use it for, you know, low level network observability. So I'm not sure how much you've followed this project or similar projects been, but what are your thoughts on this? Well, I followed it in the sense that I've like read the GitHub read me and stuff, you know, I've never tried to implement it or use it in a meaningful way. So I certainly don't want to be wrong about anything I would say I, I like a lot of things about this approach to using EBPF, my critique of EBPF based observability stuff in the past, which you've heard before, Ted, is is just that it, you know, a lot, but not all observability efforts are trying to bridge into the application layer. Oh, sorry. The nice, the best thing about EBPF is that you can integrate with it at the kernel. So assuming everything's on Linux at some level, great, like you have this integration point that will give you total coverage of everything happening in your containers or VMs or whatever. And that that's very appealing for sure. And I think, you know, coverage with observability and telemetry stuff in general is a huge, huge problem. And EBPF addresses the coverage issue in this fundamental, inarguable, very elegant way. So great, check plus. Now the trouble is EBPF is that a lot of observability efforts don't really actually work, at least in a full stack way, unless they go into the application layer. And if you're stuck in the kernel, that's really tough, especially given that most point to point communication is encrypted. So at the kernel level, it's just a bunch of just random noise, like there's no way to get, you know, layer seven data out if you're at EBPF, especially with the move towards, you know, TLS everywhere and stuff like that. So that's usually my critique. The thing I like about Cillian is that, you know, I mean, it is a bit more monolithic than say open telemetry rate, like it goes up the stack into delivering actual value props and stuff, like it's not just about gathering data, but a lot of the value props it's delivering actually do make a lot of sense with EBPF level data. I would be very, very surprised if you could use Cillian to completely replace, you know, application level distributed tracing thing that you have, whatever that is, like, because again, application level is sort of missing. And that's been my critique of having EBPF as like the be all end all to the open telemetry problem because open telemetry has a lot of focus and application layer stuff. I do think that Cillian has like an incredible insertion motion of just being, you know, in the VM. And then I also think that Cillian can solve for a lot of important use cases in a really encapsulated simple way. The trouble with it is that it's never going to be the only solution, right? Not like, you know, most complex enterprises can have one solution for stuff like this, but it's always going to run into a pretty hard brick wall. I think when it comes to application layer stuff. And that's just the, that's the thing I've never really gotten around to EBPF. I've seen a lot of people talk about it, but I've just never seen it done. So I know that they say it's L7 protocol where, but I think what they mean by that is that it can, it can like do point to point stuff at the L7 layer, but you still can't really inspect the application protocol at L7. And that's where it really would get powerful. If they can solve that problem, it would be amazing. Yeah. I mean, it's, it's kind of like the same issue with, I mean, it was literally the same issue with service mesh observability, right? Yes. Yes. Well, I mean, that's what I think actually service mesh has a little bit of an advantage because it is possible service mesh to be like post TLS. You still have this problem of like, you can't use service mesh to handle inbound to outbound propagation within a process. So it's analogous to service mesh, but at least service mesh, I think, has the advantage of being able to potentially see inside the decrypted packets. So yeah, so actually my one fun update on this, I was talking to the Sillian people about this, this issue, and they were proposing, there are two potential work of rounds around, you know, TLS encryption. One, of course, is you just give Sillian certs and, you know, it can dig in there. You know, there's some overhead with that. The more interesting future looking solution here is actually with the newest versions of Linux. They have, there has been a kernel level TLS project within Linux. So moving TLS down into the kernel, which one, efficiency gains, but two, it means that this EBPF approach, like it becomes a lot more feasible now to be doing this kind of stuff. So that's not available yet, from what I understand, but that's kind of like on the roadmap. Well, TIL, I mean, that's great. Honestly, I didn't know that. I'll just be honest, that's cool. Yeah, I think that there's still a lot to be done to go from that TLS level thing to like the fully demangled kind of HTTP stuff, but it's at least feasible, right? It's not like you're trying to like solve some sort of unsolvable math problem anymore. And I like the idea of moving TLS down the stack anyway, because I think we're having trouble. CPUs can't keep up to the network anymore. So this probably also opens the door to pushing that stuff down to the neck, which would be really helpful. So yeah. Yeah, yeah. Very exciting. Not available yet, from what I understand. Yeah, cool. Okay. So I think we're on to our last item here. So again, yeah, if there are questions in the Q&A yet, consider Q&M up. But another project, a CNCF project in this space is Fluentbit. Fluentbit is the C++ implementation of the FluentD protocol. It's, for people who don't know, it's originally started as kind of like a logging processor. But with V2, they have done two things. One, they have improved the efficiency and overhead of Fluentbit. And the other thing they have done is they have expanded beyond logging to also processing metrics and tracing, including OTLP, the Open Telemetry protocol, which also makes the Fluentbit daemon look an awful, awful lot like the Open Telemetry collector. I actually think this is fine and great, but I'm curious how you see this observability landscape unfolding. Yeah, I'm net positive on this existing. It is a little problematic in that, you know, it's just, I don't know, whatever. For end users, I think it makes it a little bit less obvious what to do. But overall, I mean, you've heard me say this a million times, Ted, but I'll just repeat, like I think it's extremely important for Hotel to remain decoupled. And the collector is a very popular and important aspect of Open Telemetry, which I'm certainly like enthusiastic about at this point, like I think it's good. But I worry sometimes that it's become such a dumping ground for like just like things that could happen at that layer that it runs the risk of like breaking some of the decoupling in ways that people wouldn't even notice because everyone's using the collector. So I like the idea of people pushing on alternatives to the hotel collector just to kind of keep Hotel honest actually about maintaining decoupling, right? So I think, especially if, you know, I understand it's fluent, but V2 has a lot of hotel kind of compatibility. And there's like OTLP support in various places and stuff like that. So it's a good test, I think, of Hotel's sort of resiliency from a design standpoint, that something like fluent but V2 can succeed and you know, and just work harmoniously with other parts of the hotel ecosystem. So like from that standpoint, I think it's great. Like whether someone should use that or hotel collector is kind of up to them. I mean, that's not as important to me, but I actually think it's really helpful to have projects like this exist. It is certainly off-road for OTL, like I don't as you know, have it here, but that's not to say that I think it's a bad idea. It's just like it's sort of helpfully divergent from the OTL way of doing things, which would be the collector. I guess put another way, I find it very hard to believe that you'd want to run OTL collector and fluent V2 in the same kind of, you know, like in the same environment, like that seems like they're sort of cross purposes. And that's fine. But it is off-road in that sense, but it's like helpful for the project as a check and a balance, I think. Yeah, yeah, I totally agree. I see this as, you know, for people who are in the fluent ecosystem, this means they can kind of stand Pat and just upgrade what they're currently doing and start gaining all the advantages of a collector without having to switch. So I think it's great. And I think you had a great point about keeping open telemetry honest. I haven't seen a step in this bear trap yet, but I do worry like you that if we start assuming that everyone using OTL is running a collector or that it's like no big deal to go to people and be like just run the collector, that we are going to create a problem for ourselves and for people. We haven't actually crossed that Rubicon yet, but I think having a good chunk of the open telemetry community using an alternative to the collector is good here. I also see it as kind of like this like go versus C++ thing. At the end of the day, you want this component to be as efficient as possible and we write everything and go, but I will be like a secret, I don't know, contrarian. This is a recorded call, Ted. I will go on the record and say like go is great, but are we going to hit some wall at some point where we're going to wish that this component was in C++ or Rust or something where you aren't fighting a garbage collector having written a lot of go code. Yeah, I think that's completely valid. And I mean, yeah, I was going to say Rust, but you said it first. I mean, that also seems like if you're trying to avoid the pitfalls of go, but don't want to run into the just monstrous C++ sort of anyway. I see that there's a question, hi, will there be a recording sent? Yes. Yes, there will. And it will be on the YouTube so you can watch it there too. Cool. Yeah. Yeah, we're at the top of the bottom of the hour. We're somewhere near the hour. And this is our last item on our KubeCon review. So it's time for questions or it's time to say aloha. Is aloha mean hello and goodbye, Ted? Hello, goodbye. I love you, man. It's just a really warm, fuzzy, general purpose word. Okay, that's good to know. Ted's from Hawaii, so we can ask him questions about Hawaii as well. Yeah, aloha is like a concept. It's like my aloha, like I have aloha. It's almost like a mana kind of thing. Is that right? Yeah, giving you my aloha. It's like a feeling, but like feelings are also like an energy, you know, you can share. It's like she or something. Yeah, a little bit. Yeah. Yeah. I was killing time to see if anyone was going to ask any questions. I'm disturbed when people don't ask any questions, as I'm sure that people disagreed with something we were saying, but it's fine. Yeah, it was being quiet. Cool beans. Well, it seems like we're good, so perhaps we should just sign off. Do you have any parting thoughts, Ben? I will just say that, you know, we didn't mention it, but it's really nice that KubeCon actually happened, you know, it's been a long three years, almost three years at this point. I know that KubeCon kind of happened before, but this one seemed like the veering back towards normality at some level. And it's also really nice the hotel has kind of gotten to the point that it is. The thing in the bottom right is critical, as you said as well, but now we announced Hotel KubeCon and Mia in 2019, or KubeCon EU, whatever, the one in Copenhagen or something. Yeah, I don't even remember what it was. But it's been a long time getting to this point, and it's just, it's really satisfying to have made it to that milestone. And now it's just all about ease of use and scope. It's an exciting time for an exciting project with a lot of things going on as evidenced by this slide and everything else. So yeah, I feel good about stuff. If people have other questions, please reach out to Ted or I on Elon Musk's personal Twitter thumbs down. Yeah, I hope this is useful for people. Yeah, I'm moving back to AOL chat this point over it. Yeah, well, always great talking, Ben, and yeah, we'll be doing open telemetry community days at every KubeCon, hopefully going forward. So everyone hope to see you there. Thank you so much, Ben and Ted, for your time today, and thank you everyone for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today. We hope you join us for future webinars. Have a wonderful day.