 Well, thanks for coming. I know that Thursday is very exciting, a lot of news, a lot of things. Who's here your first time at KubeCon? Can you raise your hand? Oh, really? And Fluent Bit users? Cool, half for half. Let's get 80% at the end of this year. Well, my name is Eduardo Silva, and together with Anurag Gupta, who's also my interior of Fluent Bit, we are really happy about this announcement. I know that you see this with some kind of news this morning, and the goal of this session is to give you a very quick intro to the project, because we don't have so much time, a quick update about what's going on, and a quick demo if we have enough time for that. So we represent ourselves, you can follow us on GitHub, Twitter, and, well, first of all, one-on-one, right? So Fluent Bit is part of the Fluendee family project, and both solve one of the problems that is move data from one place to the other. That's basically how it started, right? But in the middle, we can do processing, data reduction, and a bunch of things that maybe you are already aware of. Both projects are graduated with a CNCF, right? So a few projects are in that state for now. And as the name is called Fluent, it's going to be a Fluent ecosystem, as never the intent of Fluendee in the beginning or Fluendee Bit, be just the only solution that runs in your environment, right? Actually, in the beginnings with Fluendee, part of the story, we even integrate with other projects that are kind of competitors in the same space, right? Because the value is that the final end user, who are you, right, can choose between one solution or the other based on your own specific use cases. Okay, but moving forward, I think that when Fluendee we have followed the same pattern, right? No matter what is coming out in the industry, if there's a new standard, we are going to jump and support it. Why? Because the users need it, right? And yeah, and that's how we have evolved the project a lot with different type of integrations. And Fluendee is all about open source, but observability, and that is really important is to provide the avoiding the vendor locking for users, right? It's normal that when you go and you buy a solution because your goal is not to deploy an agent, right? Nobody likes to deploy an agent besides me, but besides that, right? You choose a splang, you choose elastic, whatever you want. But at some point, after one year, two years, or in the middle, you just want to use one backend and maybe switch to another and to gain more control over your data. And that's what we call the vendor lock-in. Vendor lock-in is when the vendor that you're paying for services locks you in into their own solution and it will be very complex for you to get out of it. Sometimes it takes more than a year, right? So the whole thing about Fluend and many other projects in the ecosystem is about giving you this freedom and flexibility to say, okay, today I'm using this vendor, but since I use this project like Fluend, I can decide where my data will go, okay? And it's project and product platform agnostic. Who use this? Well, if you go to GKE Google Cloud, deploy a Kubernetes cluster, or you go to EKS on Amazon, you will see that Fluend bit is running by default in there. There are millions and millions deployments of Fluend bit every single day. And where are we going? Everything started with Fluend D and we are from the same family, but Fluend D was really hard to make it scale even more from what was doing at the moment. Mostly the major technical challenge that's within Ruby is very flexible. It's very powerful. There's a thousand plugins in the market, but it's about to distribute the systems and be able to scale at a very high rate because companies and users are having almost 35% more data every year. You know, the agent is one side that is not scaling enough, and that's one of the reasons why Fluend bit was born. Well, originated for the embedded Linux, but then started evolving as a cloud native solution, and we are part of the same family, same project, and we say that most of the innovation is happening now in Fluend bit, and one of the trend is that cloud providers like Amazon, Microsoft, and Google has migrated from Fluend D to Fluend bit. In the highlights, well, production grade is really high performance. Maybe you don't know, but Fluend bit is really in C language. Yeah, unsafe memory. Yeah, you can say all of that, but it runs and it's scale. And that is the good thing. You can develop on any language. So in any language, you can do things slow or fast, right? Or buggy, right? The thing is that if you had the best practices to avoid situations like memory corruptions or, you know, having not free memory when it needs to be done, yeah, if you put all the best practices, you will be fine. And I think in the project over the years, we have learned about all these practices and there's a team, they're cloud providers contributing to the project. So every year it gets more strong. And of course, low resource consumption. Everybody, when develop applications now, nobody thinks about memory. And I think that that is a bad practice. Well, I'm kind of old school. And old school, I think about memory, how much memory this component is going to use. And maybe I can reuse this bunch of memory before trying it out to release it and reallocate it again. This kind of optimization exists in Fluend bit. And Fluend bit can be retrieved and consumed from different places. You can use, of course, follow the upstream releases. Fluend bit upstream, a new minor version is released every two weeks, right? But if you want to run in a more long term stable way, there are distributions of Fluend bit that are based on upstream, but they're supported like the AWS for Fluend bit, a Cality for Fluend bit, and Google Ops agent, which Google use for their own customers. And you can get the container images, packages for Deviant, Red Hat, Rocket Linux, or any kind of new fancy distribution around. Even CentOSix, I think. You will not imagine how many people is running CentOSix. Okay. So Fluend bit, as I said at the beginning, it's all about to move data from one place to the other. And you will think that the concepts that we have are sources or inputs, and in the other side are destinations. How Fluend bit connects everything is about the concept of plugins. So internally we implement a plugin to understand, for example, metadata from Kubernetes, how to scrape data from Prometheus, how to receive data from applications, sending open telemetry data, or just metrics from the Linux system that is running. So it's very versatile and flexible to get data locally or remotely. And the same concept applies for the output side. What I didn't mention here is about the filters. So in the middle, you can do filtering, which means a capacity is to modify the records or enrich your data or just drop data that follows a pattern that you are not interested in. And where we are today, this is crazy. And I say crazy because when you start a project and start getting a few docker hub pulls, well, this started in 2016, but we just started the container in 2019. And you see that during this year, we're getting more than 2 billion downloads. That's a lot. And what that means, it's a lot of responsibility for the maintainers and companies that contribute. But also you get more enhancement requests, you get more back requests, hey, we have more backs because you found many other kind of use cases. So the trend is like this year, everything is going up. And I don't know, in total, it's more than 3 billion since it just did a blog post about that. And well, we're really happy about this, but it's a huge responsibility. So I'm sure that everybody who's here will contribute to the project also to make it more healthy. In general investments in this year, we have done all the telemetry signal support. Fluentbit was originally just done for to support blocks, but then we expanded to support metrics and support traces, right? That means that we support the native payloads. We added TLS support for the input plugins. That means that you can use Fluentbit now securely as a collector or an aggregator, meaning a software that sits in the middle and can receive data over the network securely. We have a new cool feature called TAP, TAP. That solves one of the problems that many users have. They said Fluentbit is working and seeing my data, but sometimes the format of my data is not as I was expecting after I query Elastic or after I query Splank. So they said the only way to travel should that, sorry, is to stop Fluentbit, right? And try an STD out plugin and try to dig what is going on. And then I can start it again, on start, and they got into that cycle. So TAP is a future that now, if you enable, of course, over an HTTP request, you can tell Fluentbit, hey, open for me a TAP session and send me in a couple of seconds the data that is flowing through this input plugin. Just send me some samples and then that session gets destroyed. So you can inspect what's going on, you know, internally. Of course, you're not going to enable TAP to the public Internet. It will run inside your cluster securely, right? But it is up to you how you handle it. And all the requirements that we got is like Fluentbit has like a storage engine. It's not like a database, but it allows you to store data either in memory, either in file system, depending on how you want to handle the load. And Fluentbit internal metrics did not expose the storage metrics. Now you can float those metrics as part of the pipeline. For example, you can ship Fluentbit metrics through a Prometheus remote right endpoint or open telemetry metrics. Okay, so logs, metrics and traces. This is all about this release. And logs has been there since the beginning. We support unstructured data, structure is schema-less and we support processing like filtering and enrichment of data. You have seen this from the beginning. From the metrics side, we support the standards of open metrics and open telemetry, right? All of this is about defining the schema and we support different metrics types like countergages, histograms, summaries, whatever you have right now. And the good thing is that when we say support metrics, not just Prometheus and open telemetry, but the moment that Fluentbit gets metrics inside, we can even send those metrics to Splunk. We can send those metrics to InfluctsDB, right? So this is one of the values of this implementation that keeps being agnostic and provides you with flexibility to connect to different endpoints and end point types. In the Prometheus side, we try to replicate and integrate different specific futures. For example, I don't know if you're familiar with not exported metrics, which is a project from Prometheus. In conversation with the team, we come up with idea A, why we don't implement the same functionality but inside Fluentbit. So Fluentbit now has a plugin called not exported metrics that generates the same metrics that that tool that Prometheus does. So many people who was running Fluentbit and Prometheus not exported metrics, now it just can do the same with Fluentbit. So they just had one less agent to manage. We can scrape Prometheus endpoints and also we can expose that metrics data by using the Prometheus supported protocol or the Prometheus remote write. In open telemetry, today we are announcing with Fluentbit that all support for logs, metrics and traces. Fully supported. I know you will have a lot of questions at the end, so we'll be happy to answer all of that. And if we don't have enough time here, we can go to the hall and fight there. Okay, performance improvements. One of the challenges is like, you know, 35% more data every year. I'm tailing files, but I now have more load. And Fluentbit from the beginning was running just in a single process mode. But it was really efficient. But year over year, people say, hey, you know, I have like 96 cores in my machine and you're using one. Good enough. Okay. We implemented threading for the output plugins. So now when the output plugins are doing processing or encoding the data, all of that happens separate CPU cores. Now we did the same for input plugins, but you have to enable that future. Okay. So you can tell to tail, hey, just run your threading mode and that old tail will do run in a separate CPU core. And there's a bunch of performance optimizations like reduced memory allocation. We do a better handling for data encoding and so on. For us, performance is always first with security, of course. And we try to, you know, go up year after year. And for who's a developer, and you say, oh, why Fluentbit and developers? Not for who developed Fluentbit. But if you're in a company, one of the trends is that companies want to bring the business logic into the data pipeline. Meaning like if some data contains a pattern where there's a credit card transaction number or any kind of sensitive data, you might want to mask that information or there's any social security number and you want to perform some action if that data comes in. In Fluentbit, as of yesterday, we provided Lua filters. So filters with a Lua scripting language that you can write your own rules and take some decisions on that. And today we are expanding that with WebAssembly. If you're not familiar with WebAssembly, it's like a new trending thing that allows you to write a code in many languages like Rust or C++ or Python and generate some kind of byte code, unique specification that can run in Fluentbit. So you're not longer restricted to see, you're not longer restricted to Lua. You can write in your own language. And that applies for input plugins and filters. And for GoLang, now we support the GoLang to write open plugins in GoLang. Now you can write your input plugins in GoLang, too. Okay. I'm going to pass the microphone to Anrak. You want to connect your computer? Yeah, I'll plug in my computer. So, hey, everyone, what I'm going to do is show a demo of all of these things kind of in practice with the architecture. Looks like the other microphone is not. Yeah. So, yeah, what we're going to do is show you an architecture of how this all works together with Fluentbit 2.0. We're also going to show Yeager, Grafana for visualization, and then all of this running trace application underneath. So an application is generating a lot of good open telemetry data, great schema of firing off that data. We're collecting it with Fluentbit, and we're firing it off to the open telemetry collector, and we're firing it off to Yeager as well. So if I go ahead, I can just show you the Docker desktop that's running here. And this just shows you the containers and apps I have. All of this is open source, by the way, so you can try it out here as well. You'll see our Fluentbit demo app. We have Fluentbit. That's now running the 2.0 version. Yeager, Prometheus, the collector, Grafana, and Vivo, just so we can quickly visualize the data. So if I go ahead and go into a Grafana dashboard, the same experience that you might be used to and be using, we're having Fluentbit firing off this data from a metric side over to Prometheus, and here we are collecting those metrics and visualizing it with the dashboard. So what does that really start to enable is we don't want to have you continue switching all these tools. Why can't we just plug into all of the ecosystems, workflows, tools, and open source standards that you might already have in place? The next is with the Yeager side. We're just firing it off to Yeager with all the open telemetry support. That awesome team has been adding over there. We can quickly go ahead and take that same service, look at the spans. Again, this is a really simple, I think it's a JavaScript thing that's firing off telemetry spans over time. Now, this feature set is something where we've tried really hard also to make sure that the experience of setting it up doesn't mean that you have to have intense knowledge about the underlying protocols and how everything works. You really just set a port up and from there, depending on the data type that was being fired, we're able to grab that, take it for an internal conversion, and then fire it off to the expected output, whether it's Prometheus, Hotel, Yeager, et cetera. Now, the last thing I want to show is one of the kind of top features that have always been requested from our side of, hey, I'm running this on thousands and thousands of servers. I'm not sure what's happening. Data's not flowing. How can I go and make sure that the data that's being sent is correct? And this is a feature we call tap. Now, tap, I'm going to go ahead and open up my terminal here. And in the terminal, I'm just going to go ahead and increase the font size here. Awesome. Okay. And then what we're going to do is run Fluentbit v2 with this new option called dash Z. Now, what dash Z is going to do is it's going to allow us to make an HTTP request to the running Fluentbit to say, hey, Fluentbit, what's going on at this particular plugin? I don't know what's happening. Something's not functioning correctly. I want to start to debug and understand what's happening. Now, if we go into this other tab to go ahead and make a request, we can run a pretty simple curl command. And the plugin I'm running, it's called dummy. It just generates as you saw a message dummy every single second. I'm going to run this command. It's going to say, okay, Fluentbit says, okay, I've been instructed to now go and instrument all of those specific dummy filter plugins. Now, say you have 100 filters, you can actually choose the very specific filter or plugin that you want to go instrument and enable. And then we will, on the other screen, we'll print out all of that information for you so you can see the exact payload that is there. So you have to enable this. So it's not like this is enabled by default and anyone can access anything in your pipeline. Two, there are limitations or settings you can put in with how big a payload you want for a specific amount of time. And really quickly start to look at this information coming in live and say, ah, that's why I got an HTTP 400 error. That's why my filter is not showing up. A little bit easier to do things live, which I always am surprised with, right? Debug in production. And then last but not least, just kind of on the visualization side, if we want to look at what's happening, you can fire it off to say a live data viewer. And this is one of the open source projects where you just dump data in. You can start to visualize it. So with that, it's a kind of overview of Fluentbit v2 architecture of how we send data to all of the different open telemetry sources, logs, metrics, traces. We showed a little bit about TAP for more operational debugging. With that, I think we'll have plenty of time here for additional questions. But yeah, thank you so much. Hi. I have Fluentd. It is running well. No issues. Can you convince me to switch to Fluentbit? If it works, don't fix it. I think that's also, yeah, it's a very good, good question. If I'm running something, no challenges, what's the purpose of migrating if you're not getting incredible? But whether I should start considering it sometimes in the future when scale is higher or where is this line where we should start thinking about, is it about performance when Fluentd starts to fail due to high volume? I think three considerations, scale for sure. When running on Kubernetes, container density goes up, more logs, maybe network devices, security devices, generating more traffic. Great. That's one consideration when you might want to switch. Two is if you want to do some more of those enrichments, redactions, things with Wasm or Lua, there's a lot of inbuilt functionality there for shying and stuff like that. And last but not least is you start to have constraints with resources. Maybe your environment shrinks or things become a little noisy. Having that capability, those are the three draws that we typically see. But if it works right now and none of those things are hitting you, kind of just keep it. Thank you. Thank you guys. So we are running Fluentd, not Fluentbit yet. And like you said, no need to upgrade. I've seen like you, you are sending logs to Prometheus as well from Fluentbit. Right now we are running the NodeExporter. And I think it's Prometheus Blackbox. So what's an advantage of using Fluentbit instead of using just the Prometheus NodeExporter and the Blackbox? Great question. For, I'll say almost the same answer. If it's working with NodeExporter, no reason to switch it out. What we find is sometimes folks will say, I want to capture host-level metrics, but I don't want to run a NodeExporter. And that was the main draw of folks who were saying, hey, I'd love to scrape the data. I don't want to have to install another or manage multiple agents. Managing agents is not the fun thing of anyone's role. So if that's something we can enable and you get value from it, awesome. But if you have NodeExporter there, great. That's a big win, too. Something to add on NodeExporter in general is a great project. But we found that many users, for example, has a cluster like 100 nodes. And if not exported, is using 15 megabytes, 20 megabytes, multiplied by the amount of nodes, sometimes they say, hey, I don't want to waste that amount of memory from a cluster perspective versus some kilobytes in Fluentbit for the same functionality. That's one of the trade-offs. For both Splunk as well as for Prometheus? Or do we have to run two sets of things? So users ask to run Fluentbit where all the application sits. And it's a good challenge for us, because it pushes us to make it more lightweight. So yeah. Same Damon said, just add the Prometheus config, try it out. It will extract the logs, format, and all. Exactly. You don't have to do any additional Fluentbit instances. Thank you. You mentioned you added two languages support, WA and Golong. I'm not very familiar with the WA, but I'm a Golong developer. So I'm wondering that what dependencies, dependent packages that you natively support with Golong, can I add the dependent package as I want? Or what's the limitation of that language support, as well as with WA? Good question. So if you're going to use, so with wasm and go, if you're going to use specific libraries from either of those languages, you'll typically need those as a dependency. Similar to, like, say, Lua, we include the LuaJIT library, but if you're going to do some Lua XML or some other library, you need to have those as part of the initial package. With wasm and the go input plugins, those are the same dependencies that we have with the package today. So it's not anything additional. I think the biggest additional dependency you'll have with Fluentbit 2.0 is probably the YAML side. So we now have YAML-based configuration. So basically, if I understand your answer correctly, then you mean if I want another dependent package for Golong, then I have to have a Docker file, have those packages installed along with the Fluentbit. Okay. Thank you. The gentleman, you're right. So just to follow up on his question, as far as the node exporter feature, does that have, is it at parity with the Prometheus node exporter, or is there a subset? Where does that line get drawn? Yeah. Good question. It's a subset today. So probably about 80%. So you could take that same exact node exporter plugin, fire it to Prometheus, have that Grafana node exporter dashboard, and it'll probably fill out about 80% of the metrics. Do you guys have the 20% on your docs somewhere that isn't there? Yeah. There's a couple of GitHub issues. So folks in the community say, hey, we really want file system with this specific metric that we've skipped over. Okay, great. But I think those are the things that we're really looking for from the community also to say, hey, look, we want XYZ. Here's also what we want. Windows metrics, Windows exporter. And that really helps for us to build out a robust roadmap. And then you mentioned you guys have the Prometheus remote write API. Yep. So could I, I could deploy, I could end up basically deploying Fluentbit and not have to maybe deploy a local Prometheus cluster in my cluster? Yeah, you could. One typical use case is a lot of hosted providers have Prometheus remote write endpoints. So you just fire off the data over there. Awesome. Thank you. One other use case that came up is with all these open telemetries thing. Some users said, hey, I'm using this X provider or vendors with an open telemetry end point. But I want to collect my metrics with node exporter, right? With node exporter alone, you cannot do it because you might need the Prometheus node exporter plus open telemetry collector to do some processing and send it out. But if you have Fluentbit, you just can do node exporter style locally and fire off the data to open telemetry metrics directly. Hi. One of our lead engineers is very frustrated by how far AWS FireLens is behind the current Fluentbit. Do you have any insights into that? Do you have any knowledge of how long there it's going to take for them to catch up? I wish. No, we don't have any insights. There is a AWS Fluentbit maintainer. So that might be something we're talking to. We're trying to ship every two weeks, put as many cool features as we can. So unfortunately, I don't have any visibility into that. But we will pass the message. Yeah. Hey, so currently we have a log stash to scrape the metrics, scrape the Kafka event, and we send it to open search and elastic search. Do you foresee or do you have any, I mean, do you think the Fluentbit can support that kind of event scraping from Kafka and send it to any of the logging solutions? Yeah, good question. We've had a Kafka input issue for a while. We actually did a small POC with the single threaded input model, and it was a bit slow for what we feel comfortable with. With the new input threaded interface and V2, we can perform that at a much, much higher and faster cadence. So actually for Kafka, we use already Kafka out on the output side, and that works really, really well. There's a good presentation by LinkedIn on Monday at Open Observability Day. I think those sessions should be up about the Kafka output. For the input, yes, we've unlocked the pieces to make it so. Sorry, I just remembered my other question. Sure. On the tap functionality, enabling that to be a command line argument and then it's over HTTP, correct? That is correct. So there's two ways to enable via command line. You can also enable it in the container via service level parameter, and then to enable the actual tracing functionality, it's a HTTP request. And then can I provide an SSL cert or is that not a capability it has? I think for now the in TLS side is limited to input plugins, but that is something where they do borrow the same interface, so we can put it on top too. Okay, thank you. Okay, I think we're right out of time, but of course we'll be out in the hall if you have something more in-depth. I don't want to complain about any other vendors. We'll take it.