 Hey, welcome everyone. Thanks for coming in. I know it's a little packed, but thankfully these will be recorded and everyone can also see it otherwise as well. So I'm on a rug and I'm joined with Eduardo and we're our maintainers of the Fluent Bit project. And I guess some questions to kick it off. How many folks here know what Fluent Bit is? And keep your hand raised. Are you using it today? Okay, a couple less hands, good. Awesome. So what we'll do today, we're gonna go over the project a little bit and then we'll run through some of the updates that we've done, specifically talking about some of the integrations that have come in the latest version released yesterday. And then last but not least, we'll save some good time for a demo here. So again, yeah, I'm one of the maintainers on a rug and then I'm joined with Eduardo. We both work for the company, Calyptia. And so we've been here doing Fluent D, Fluent Bit and very much in the space of telemetry, logs, metrics, traces for quite a while. So really excited to get to chat with everyone here. And kind of starting at a very high level, what is the complexity of telemetry data? The idea behind all of this data that we're getting, it's really so we can go do some data analytics, data analysis, what's going wrong, troubleshooting, have a really nice understanding of what's happening within the environment. And ideally, when you understand what's happening, you're able to make better decisions, whether those are business related, whether those are for operational purposes. And to do this, really we look at the challenges that occur with this. You have multiple sources, everyone here might have a different logging library, you might be logging XYZ, you might be instrumenting your application in a certain way, there's different languages. And we have tons and tons of different data formats. So the more that we continue to produce and create the software, we are continually expounding and increasing the challenges in order to understand what's going on. And a lot of folks might say, oh, there's profiling, there's others, but really if we focus on the three major verticals of telemetry data, you have your logs, things that have been there for long, long periods of time, metrics as well, things that kind of detail how a specific application might be running, could be custom metrics about something within the business. And more recently you have the tracing vertical. So things about your application, how a response might be going. And if we look at how these three types of telemetry have evolved over time, really it's, from a logging level, you're getting things like your raw text logs, your sys logs, your system D, you're getting all of these kernel logs and it's trying to tell you, as the word log implies, of what is happening within that system. Thankfully, we're starting to get some more structure with things like open telemetry, which we'll touch on in a bit, but things like JSON and other types of formats have kind of helped us to understand how these logs are evolving over time. Now from a metrics side, this has been, again, something that's evolved over the past few years, right? We're looking at two types of things. One is what is happening from maybe a performance standpoint, what's happening from a business perspective, how many users are signing up, how many users are leveraging the platform. And those helped us to build things like alerting. And typically when we're doing these type of telemetry observability scenarios, we're starting with that alert, we're looking at what happened and then deriving the metrics, looking at the logs, looking at tracing information. And we have a lot of different types of these metrics. So folks who might remember all the stuff going on with CPU metrics back in the day, you have things going on with stats D. Really, we've kind of cornered into the four common data types around metrics, your counters, gauges, histograms, summaries, things that are very well known within that Prometheus and again, hotel ecosystem. Traces, so now as we continue to build applications that are distributed, not just on a single server, really we want to understand how things are flowing through a system, service mapping, how long steps are taking, are we spending a lot of time talking to a database, spending a lot of time talking to a particular end application. And within that implementation, we're seeing a lot of awesome things around spans, attributes, and even things with extended Berkeley packet filter, EBPF, where now you don't even have to instrument some of that code, it can be done for you or looking at that kernel level. So all in all, we have a ton of stuff where we're continually building all these different data sets, data pieces. And if we look at what's out there, many of these tools are gonna look familiar. You have Fluentbit, FluentD, R-SysLog, things that are just there standard out of the box. On the metric side, of course, we have Prometheus, OpenMetrics, Nagios. And then tracing side, we have OpenTelemetry, OpenTracing, and everyone here has come to the Cloud Native Computing Foundation, KubeCon event. It's really great to see we have such awesome communities that are building on top of each of those different pillars of observability, even others, and really trying to address each of those problems from a best of breed type of scenario. Now, with that, the challenges that we're starting to see, or at least what we've seen is folks, again, multiple sources, multiple formats, data's growing year over year, and not all of it, to be honest, is it should be treated the same way. Sometimes I'll accidentally put a console logger for hello or something just to see that it's working or hitting that path, but that's not useful for someone else debugging the application. And if I accidentally leave it on, it can generate tons and tons of expensive data. Back pressure, where you're in an environment where we're continuously sending all of this data, how do we handle the fact that some of this data is being generated faster than we can even send it out? And then from a Kubernetes standpoint, we're introducing all these different architectures. We need to get more and more data off the machine. We're packing up these Kubernetes nodes with even more containers and pods, resource utilization, speed, different architectures, whether it's ARM, x86, these things start to come into mind with making sure that we're leveraging all of this to the full extent. So again, at a high level, that's where all of this telemetry sits. Now, from a fluent bit side, what we've been looking at trying to address a lot of these challenges, this project created about 2015, and initially it was built for embedded Linux, where Kubernetes had just kind of been open sourced. IoT was the big thing back then and said, great, let's make something for embedded devices. IoT, let's get that data. The resource utilization should be super small, less than a megabyte. Well, it turns out that embedded Linux is a great environment similar to a container. Low resource utilization, super lightweight, something that you want to have that's not taking up too much, not noisiness, and that's really where Fluent Bit evolved. And from there, it's just kind of taken off. We've seen so much usage throughout containerized environments, Kubernetes, even older environments like your Red Hats or embedded Linux, Red Hat 6 specifically. Wow, I didn't expect to keep seeing that out there, but hey, it's prevalent and running. So that's where this project has kind of come into play. And as it has, we've been continually expanding it to meet those telemetry use cases. So the inception actually wasn't logs. Everyone thinks, yeah, Fluent Bit started with logs. It was actually with metrics. We did JSON-based metrics to show things like heat, kernel-level metrics, CPU, memory, and that was for your IoT devices. Are they performing well? Are they overheating? They were log-based metrics, but then all of a sudden folks said, hey, this lightweight package is great. Can we tail the log file? Can we actually get that system information? And so logs was added. And then from a metrics side, Prometheus started exploding. Well, we wanna make sure we integrate really well with that ecosystem. So we added Prometheus scraping. We added node exporter metrics, windows exporter metrics, the ability to remote write to a Prometheus endpoint or even export those metrics over an HTTP endpoint if you already have scrapers and places. And now with traces and kind of the open telemetry wave that's going on where we're finally getting some really nice protocols and schemas for all of these data signals, we've added that support. Make sure that, hey, from a tracing standpoint, you're gonna be able to do open telemetry input, logs metrics traces, output log metrics traces. So again, this kind of cohesive way of adding and combining all these different ecosystems. Again, from a background perspective, a lot of folks are leveraging fluent bit all over the place, spin up ECS, EKS, GKE. If you're using Azure, they're networking service, Azure Mariner, which is open source is using it. And then again, a ton of other companies within different verticals. And so this is where we've been focusing from a project standpoint. How can we continue to address the challenges of increasing telemetry data, connect those ecosystems that might be a little different that have these different sources, different destinations, and make sure that from your perspective, it might sound anti what we're showing here, but you don't really have to think about an agent. You're thinking about how can we make better analytics, better business decisions? How can we make sure that we're servicing our end users with the highest grade of software operations? So we're in the background, we're in the infrastructure level. We just wanna be that easy kind of click and usage side. Now from the ecosystem side, again, we're really trying to plug into what's out there and what folks are using. We're trying to make sure that you're not locked in. So if you think you're using X backend today and you wanna use Y backend tomorrow, excellent. Let's make sure that that can work really well. And then work on top of these different products, on top of these different platforms, and in top of those different projects. So with that, let me hand it off to Eduardo to talk very specifically about what's in 2.1 and how that helps kinda keep that story alive with all the telemetry. Thank you. It's exciting times. I'm most excited after lunch, right? I know that we're getting tired. All rooms has been packed. So let's go into do some quick updates around the project and then try to do some demos so you can see what is new, mostly for people who's aware about the project. So today we're launching FluentBate 2.1, right? The major release 2 was around November last year and now 2.1. At the end of the year we're planning to have 3.0. Now, one of the biggest features that people was asking for a long time, I would say four years, five years ago, from the beginning is that hot reload support. And now FluentBate finally has a hot reload support so you can do a trigger through a Sihap signal or just through the H2DP endpoint. Yeah, security guys, yeah, you have to enable this in the configuration. This is not there by default, of course. No, but you don't want that somebody just make a request and restart your agent, right? One of the problems that people had is like, hey, I have this application that is shipping a metric but as a log. Maybe it's shipping a JSON that says, my counter equals five, then six. But they wanted to expose this in Grafana, right? So you wanted to ingest this into Prometheus as a metric and we didn't have that way. Now with this new filter, you can convert any type of log to a metric. It could be support counters, gauges, histograms. It's pretty flexible. You can define the metric name, the description, and so on. For Windows lovers or companies who loves Windows, also we extended the support of a Windows metrics. I think that we already have a few agents around that is exposing Windows metrics. And this is getting a lot of traction recently. We're excited. I'm not excited about Windows, but I'm excited that people can get this type of functionality. And now the different collectors and scrapers for Windows can be configured at different intervals of time. And yeah, we are supporting Windows on ARM64. Yeah, this is kind of new, but it's something that's going to hit in the next year, at least for servers. And I think I'd skip one slide. Oh no, I'm good. So I don't know if you're aware about Potman. Potman is a project started by Red Hat to run containers. It's pretty compatible with Docker. But same as Docker, Potman exposed metrics. So how do you monitor your containers that are running under Potman? So now Fluentbit has a plugin that you can scrape natively the Potman metrics and expose them through Prometheus, Remote Ride, Open Telemetry Metrics, or any kind of format that we support for the metrics output side. Also, Fluentbit used to just have the knowledge of every record or event contains a timestamp when this was generated or when it happened, plus the content of the record or of the event. But recently people are starting to think on microservices and microservices, what is important is the concept of context, right? And how do we represent context in the data that is being generated is called metadata, right? Usually a context for people who's, I don't know, configuring something could be a label that says color equals blue, right? Or any other type. Since Open Telemetry was released and they released, this is a real story, they released the spec for logs, right? They separated metadata from the content. So, and that lead us to adapt our new internal format without breaking changes to support metadata inside the log. So right now, for example, Fluentbit can receive Open Telemetry logs, right? From the input side and send it out and you're not going to lose any data. Metadata will be there and the content will be there. And one of the use cases of Fluentbit, this is quite crazy, right? So we have a JavaScript application sending Open Telemetry Metrics and trace it to Fluentbit, which is sending to Jager, but also we have a Prometheus script and made it from Fluentbit, which also shipping this information out to an Open Telemetry collector, right? So Fluentbit is quite versatile on these scenarios. As I always said, it's not a drop-in replacement for something that you have in your environment. Actually, it helps you to augment and extract more value from it. At the end of the day, I see, I've never seen a production environment that just has one tool or one database. I don't think so. If I MySQL, PostgreSQL, Redis, MainCache, whatever, right? In this space, it's the same. You won't have one tool, but you have one common problem. How do you correlate information? How do you collect all the information together? And this is one tool to solve that problem. It can collect, but also can aggregate. Okay, and with the updates of 2.1, you know that Fluentbit can collect data, process data, and send the data out. The process, we used to do that with filters. And the thing is that from a global perspective, you can see that the data flows through an input on a source, inside Fluentbit. It goes through a filtering process and then go to the output destination. And we support between input, output, and filters more than 100 built-in plugins. Now, if we go to more technical terms, the input plugins and the output plugins can run in separate threads. Because, of course, nowadays, your computer might have more than one CPU core and you want to take advantage of that, right? So since we have this, you might see that all the filtering was happening in the engine, right? So the engine, at some point, if you're doing so much filtering, can become a bottleneck. We have seen use cases like with 30 filters, 40 filters, and I'm not kidding, in customer's environment. And it was, wow, that is huge, right? So we need to think about in a way how to optimize this. And we come up with the concept of processors. Processors are kind of filters but that runs attached to the input plugins and runs attached to the output lines, too. So in the input, nobody what you generate, it's a log matrix of traces. You can do any kind of processing. The output of that can optionally go to the filtering phase, go to the destination, but it will be reprocessed again. Now, you can say, but why do I need to process in the output again? Because the thing is that we support multiple outputs at the same time, and sometimes you will like to filter out certain information for certain destination. And you don't want to do it in the input. You want it to do it in the output, most advanced use cases. And this is reflected in the configuration. I started showing the new YAML configuration. So just as simple, for example, for logs, you can run a Lua script. We support scripting in Fluent Bed, but also for NoExporter, which is scripting metrics, we can add labels. And now I'm going to do a quick demo. But after that, we are happy to announce that we got a little traction in the latest years. And today we are announcing that we hit $6.3 billion of Fluent Bed in a few years. So this is a lot. Well, thank you. I'm just concerned that some point Docker have is going to send us the bill, right? But anyway, hands on. I think that you just got your Fluent Bed T-shirt, right? So we're good. Let's jump into the terminal. OK, I think that we have a couple of minutes left. So I'm going to do this in a very, I would try to do it in a very quick way. Can you read the screen? Yes or no? Or maybe. Oh, oh, oh, oh. People in the last line in the row, can you see it? You have good glasses. You don't have glasses, man. OK, I'm going to show how this works from a configuration perspective. I'm going to open my new terminal that is here. Oh, this is too small. Let me close this one, and maybe do do do do do. OK, how this works. Fluent Bed defines a configuration that has a service section, and then you define a pipeline. Same as you have one pipeline, you can have two on multiple. But you need to be careful about routing. But we're not going to touch that part right now. In this example, what I'm going to do, have you used Node Explorer from Prometheus? Have you had the need to scrape metrics from a file before? OK, because you're using new technology. But if you're going to talk to other 5,000 folks around, they have been scraping metrics from a text file because their application ships metrics to a file. So one of the new additions in Fluent Bed 2.1 is the ability in the node-exported metrics plug-in that we have, which mimics the Prometheus one. Full transparency, we copy-paste the source code and go in C, in the Fluent Bed. And this was a joint work with the Prometheus folks. That's the best part. Right now, so we added the collector for text file. So if you go here, I'm going to my example number one, the Prometheus format is pretty straightforward. It's like a text file that defines help, a type, the name of the metric, and then some value. This, of course, is a counter. So what I'm going to do in this first example, my input is node-exported metrics. I'm going to scrape a text file. I'm going to disable everything else. And this is the name of the file. In the output side, I'm sending the information out to three places. I'm going to expose it as a Prometheus exporter. Maybe we can skip that part because of time to a standard output. But also, I'm going to forward send over the network this metrics to a place over the forward protocol. And that place, I'm going to take the time to announce a new open source project that we have in the company that is called Vivo. Do you know what Vivo means in Spanish or in Italian? Vivo means alive. I don't know if you've got this problem sometimes that you want to see your data, but you don't want to deploy elastic Rafa and Prometheus, blah, blah, blah. Well, maybe you weren't in our same foot. Vivo is a project that you just deploy. You send the data and just runs in memory. It doesn't have a storage, but you can see the data that is available. So I'm going to take the opportunity to use it for this demo. We just released the version today. So the UI might have some glitches in there. But what I'm going to do is send data to Vivo. So Vivo, actually, what it does, we don't have a slide for that, but it runs flow and bit. So it's used flow and bit to collect the data. And then we have a web UI that's created the data from flow and bit and show it in this React application. So running this, I'm going to create the metrics. I'm going to run here. And there we go. The information should be here on metrics. It refresh after five to seven seconds. There we go. So the metrics that we have in Vivo, we use and we represent as a JSON, but we use the internal flow and bit representation for a metric. Yeah, might not be ideal. I know that. Hey, what are you using Prometheus or other? Yeah, I think we're going to enhance that. But you can see that the metrics is flowing and the data is here. So we just scrape from a file, take the metrics, convert it, sorry, a metric from a file, convert it, and send it out to Vivo as a native metric. One of the cases that we get also, people said, I don't know if you're familiar with metrics, but if you're using a metric backend or a vendor, you have the concept of dimensions, indexing. As many dimensions you have, more money you pay. Are you using Datadoc? Are you facing this problem? You are facing this problem. Yeah, if I go to US and it's the same question, everybody will raise their hands, right? It's a market thing. But the problem is that when you have more dimension, it's more expensive to run queries because you have to do more indexing across different endpoints. And in metrics, dimensions are given by labels. But guess what? Developers love to add labels. Color, car, model, year, and you don't need all of those but you end up paying for those. So in labels, also in the opposite problem, it's like you don't have labels and you want to add dimensions. So you want to play with this modification. So in Fluentbit 2.1, in the processors, we have processors for metrics and we have a processor that is called labels that pretty much can update, delete, or update a label. The absurd, you know what is absurd? What absurd does? SQL guys. Update or create, update or insert, really good. Awesome. So we are going to do in this second demo, it takes the same metrics but enrich it with this label. If I go here, because it's in this repeatedly, you will find that the labels are not there, right? So what I'm going to do is to run this second example. I have the same metrics and I'm going to do something that's refresh because it's new, right? We don't want any kind of messy data here. And after refreshing the data, you will see that we had to wait a couple of seconds. The new metrics entry that we got here will have the color blue in place in the labels. Oh, this was not the one. There one. Let me close. Yeah, this should not be here. It should be the other side. Let me here, here. I don't know if it's adding the record at the beginning or at the bottom. It seems and at the bottom. And if it's not working here, my apologize, so we will have to fix view. But as usual, also we have our standard output that should never fail. Name, STD out, match, everything. Labels color blue. So if we take a look at the STD output, we will see that the labels should be there. And here we go. The label for that dimension is there that was not there in the file, so this is a back in vivo. We have the first back. You know, that's the risk of live demos, but it's good because we get feedback. Now, when processing looks, also we got the same problem. It's like how we can reach data. Are you using a Splunk or any kind of expensive backend? Yeah. I don't want to ask how much you're paying for Splunk because I don't want to feel you bad. But how much data are you doing just per day? Gigabytes or terabytes? Gigabytes. Are you using all those gigabytes? No. Actually, there is a normal standard that says from the 100% that you ingest into Splunk, usually you query 20% of it, but you pay for the 100. And why this happens? Because people just send data. Developers send data. They back, hello world, and there you go and you ended up paying the bill. And but the problem is with logs is like, if you don't have a way to control the flow of data, you're getting a problem because every year every company is generating 20 to 30% more logs. That's it. Because yeah, you have more microservices, more applications. So having control of this is really important. And for example, in this example, for logs we have a Lua script. Maybe you are familiar with Nginx, you have done some Lua scripting in the configuration, but if not, this is a really simple language that runs in the configuration or in a file where you can define arrays, maps, well, this is called Lua tables. Lua is a Brazilian language which is really easy to go with. So basically what this is doing, I'm going to show the file that is called test.log. Let me close this previous example. And test.log is, it says, we have a JSON map. Maybe we can see here. You know, it's conference, KubeCon, where the year and the status by now. Now, if we go and run this example, and we go to logs, we're going to see the record. But now we see that it's not by now, it's sold out, same as this room. And why it got sold out? Because we run a processing rule inside the input that changes the name, sorry, the value of status. And if you recall here, status, what was by now? Same as we can modify content in traces, in metrics, we can do it also in logs. Now, when this is more useful, oh, you're talking about cost reduction, but you are also modifying the data. You can drop data. So you can say if something contains debug, if something has something that doesn't have any important meaning for the company, you just can drop it. Or maybe even better, you can route the data that you don't care today to a cheaper storage. Amazon S3, Kafka or any kind of fancy database that you can have in your environment. And I don't know if we have time because how many minutes? We have three minutes for Q&A? Okay, so I think we can use three minutes for Q&A because we'll deserve. We got a ton of slides of content here. You know, next time we will ask for a block of two hours with a room for a thousand people. Is this a hello, hello? Any questions? Anyone have questions? Yeah? It's regarding authentication. With Fluent D, it's supported that you can authenticate to elastic search using certificates. Is that something that could come to Fluent Bit? So you can pre-sign the certificate and then you can authenticate to elastic search or open search. Yeah, so from a certificate side, we support TLS Basic Auth, AWS Auth. And I believe there's a CA key as well. But maybe, yeah, we might need to get into the specific weeds of it, but yeah, happy to take a look. We've made a lot of improvements from Fluent D and Fluent Bit, so folks who've wanted to migrate over for the lesser resource utilization, the stuff like Hotel and Prometheus. So we knocked out the biggest one, which was input TLS. And so that was knocked out last October. So if there's more, let's go tackle those as well. Any other questions folks have? Happy to come to you. Okay, we'll be up here. Great, thanks so much. We appreciate everything. Thank you.