 Unified layer for logs, meters, and traces. Like six, seven years ago, we started pitching how to do a unified logging layer. That's how, well, maybe 10 years ago, how do we start it in this journey? And one of the challenges is when you're dealing with telemetry data, who is called today, it's really hard to concentrate and handle the volume. Even more when data comes from different sources and destinations. First of all, I would like to know who are your first timers using FluentBit or FluentD. Maybe you can raise your hand so I can get an idea. Oh, just a few ones. So I'm going to get more back reports or similar things. So I assume that most of you have some knowledge about what FluentBit does. But first of all, it's good to clarify where we came from. You know, CNCF as projects is a neutral place for all the projects around cloud native space, starting for Kubernetes, Prometheus, Jager, and so on. FluentD, our parent project, was graduated around 2019, if I'm not wrong. And FluentBit is part of that journey. Both projects are graduated. And while others are still in this phase of incubation, we are expecting to also get open telemetry graduated at the end of the year. So it's really interesting to see how the projects evolve from one stage to the other. And what is really important is adoption. It's adoption not just because somebody is using it, it's because they are using it, they are reporting issues, and their people contributing back to the project, which at the end of the day is more important. You don't want to deploy something that nobody is going to maintain it over time. You want to deploy something like this and forget about it, right? And when you get security issues, somebody will issue a fix. But at the end of the day, what is important is that this is a very vendor-neutral place. Despite their company like us on Chronosphere Calypia or Amazon using this project's development code, you will make sure that whatever you have here will be there for many, many years. So let's get started with locks, methods, and traces. In a nutshell, FluentBit as an agent, as you might know, can handle all of the telemetry types of data. And to be honest, it was not designed for that. It was designed for locks. But the way, and this was lack, the way that we designed how to handle locks was in a very agnostic way. We don't care how the data looks like, we're going to serialize it and provide some processing and be able to ship it out to any type of backend storage. And that's how FluentBit was evolving for handling just locked to metrics and now traces at the same time. What is, we talk about metrics support, we're fully compatible with open metrics and open telemetry. Also, for example, if you have Prometheus services exposing metrics, you can use FluentBit to scrape those metrics. Also, FluentBit itself generates metrics and you can expose them as Prometheus exporter or you can send them over remote write. And also, that's not enough, we can send that metrics to Splank so we know how to format metrics to Splank and send to InflexDB and other formats around. And when you want to do metrics with Prometheus world, so as I said, it's not just also how to, you can collect metrics but also FluentBit can, well collect remotely, I mean, can also get them from the host file system that you are running in. That means, are you using node exporter? By the way, in your systems from Prometheus? Okay, some years ago, and I think that this idea came from Europe, we always get really good ideas from here, from this version of KubeCon, I don't know why, but they told us, hey, I have node exporter from Prometheus and I'm using FluentBit. So is there a way that in FluentBit we can replicate the functionality of node exporter? Meaning like generate the metrics from the host? Yeah, because we're reading profile system, we're doing bunch of stuff. So we ended up replicating the functionality. So today if you're running node exporter and you have FluentBit, you can use FluentBit to replace node exporter, because we have a plugin that copy paste the same metrics, same levels, dimensions, everything. And node exporter works in our version support for Linux, Mac OS, and Windows. Yes, Windows is everywhere. And destinations for the Prometheus world, well, we support the two primary formats. In open telemetry side, as an agent, we support OTLP in the input, OTLP in the output. What that means? We are open telemetry compatible. So that means that in this journey where now you might think or instrument your applications to move to open telemetry, because it's a standard that all of us we're believing that will solve many problems and also for the vendors. So you can trust that FluentBit is having one of the best implementations also with the open telemetry tools that they are providing in the input and in the output. So with FluentBit, if you are using, you understand that you can collect data from many sources, many destinations. Maybe some of you are just using it in Kubernetes, but in the story, FluentBit was originally designed to collect metrics from embedded Linux devices. That means that it was designed to be very lightweight, to use a very low CPU, very low memory. Somehow ended up being used in containers like Tracy. And when you say this, it's important because before Kubernetes, everybody was running VMs, bare metal machines, and we have collectors for system D, for files to receive data over TCP, UDP. Many people collect syslogging messages from firewalls by using FluentBit outside of Kubernetes. So I just wanted to give you a bit of context. And the FluentEco system, as I said, it not tries to be a drop-in replacement. Actually, if you go to talk to any company that has any product, everything, they will say, hey, replace A with B. This is better because X, Y, Z, right? But in reality, in technology, and if you look at your production environments, you will notice that you don't have just one type of application that you need to replace. In our space observability, you have logs, metrics, and traces, plus syslog plus messages from system D, plus X, Y, Z. And you need a unified way to collect this information because your goal is to do data analysis. If you cannot have something that can be kind of plug-and-play to receive this information, it will be a mess. So in the FluentEco system, we always, that's what it's called Fluent, right? And actually, the principal project, it was called Fluent D because a Fluent name was taken. So everything, I don't know if you remember, everything was MySQL D, or something D for services. That was Fluent D. But the organization name is Fluent because I need to be fluent in talking to protocols, fluent to integrating with other projects, even competitors. And pretty much, you can create what is called telemetry pipelines. As I said, one family, one origin, and both are graduated projects. And the problem that we solve is pretty much this. We fit in the middle. We connect multiple sources to multiple destinations and we are able to adapt to every single use case from embedded Linux use cases to burn metal machines in the cloud, as Google, Amazon, and others use. And the way to deploy this agent is that you just set it on the VM, in your container, in the burn metal, and you think that there are some experiments if I'm not wrong as a serverless. So Fluent Bit is a C program, a C language program. It starts really fast so you don't have any delays, so which is a really good solution for something that needs to compute data pretty fast in a matter of less than one second. And when you collect data, you can not only send the data to one destination, we support routing, we support buffering, if something goes wrong, the data can be retrieved from that safe place that is called the buffer, the file system buffer, and you can send data to multiple locations if that is desired. But also at some point you would like to do some, what we call in our word, aggregation, sometimes called now collection, right? But Fluent Bit can be deployed as an aggregator or what is used to be called as a forwarder on the edge or a processor in the edge. So what that means, and when you are going to try to do something like this, imagine that, do we have elastic users? Elastic search users? Okay, please don't blame on me, but if we look at elastic search three years ago and you have like a few hundred nodes sending data to elastic, what happens? Boom. The JVM explodes because it cannot handle the load, right? Well, it can handle by text time because also it's doing indexing and many things behind the scenes. And one way to solve those problems was to put something that controls the data flow in the middle. So you will find that in the past, many elastic users were using even LogStash or Fluent D in the middle, same as this, in order to control the data that is being flowed from the edge or from the agents. And same as it can be used with Fluent Bit, you can receive data from OpenTelemetry or you can scrape meters from Prometheus and then send that to your own destination. And I'm going to do a really quick recap on how Fluent works in the Kubernetes login world. So, you know, Kubernetes is just a group of computers, right, a cluster, right, and you have two primary roles. One is called the master server or API server, and then you have nodes. A node is pretty much a physical or a VM machine that runs the operating system instance. And when you deploy an application, you deploy what is called a POT. A POT runs a container. And the way that login works in Kubernetes, in most of the application and kind of best practices, it does make your application write any type of login message to a standard output interface or standard error. What happened next is that the container engine, when it's running and it has it's piping these interfaces and receive those messages that are being generated, it writes them back to the file system. Yeah, maybe you can bypass the file system, I use a network driver, but normally what it does, it's writing the messages. And there's a typo because it says, Apache is writing NGINX, I just noticed that, but we will fix it. So every container has its own container log. But that is not enough, right? So when the data gets written to the file, the file gets encoded inside a JSON message because it has some metadata from the container engine that is the stream where it comes from and also at the date or the time where this data was generated. And this looks okay, school, but also our message gets encapsulated in JSON and it gets escaped because JSON say you have to escape your messages. But that is not all. When you deploy an application, you don't want to say, oh, show me your logs from NGINX from Apache. You might have a thousand, but the way that you deploy this thing is like you said, I'm going to set labels and going to add some context to this pod that is running. And those are labels and annotations. But that information does not leave where the application is running. It leaves in the API server, in the master server. So if you want to do login, you have to take data from two places, make them one, and then you will be able to ship it. So something like this, that is a simple message at the end of the day, gets expanded with a lot of metadata like this. So, yeah, login is really expensive. You might imagine that writing this to the x-ray from this decoding and coding is really expensive. And in our world in fluent, we try to do as more cash as possible, right? And reduce the number of API calls to API server, but also we can talk to Cubelet that if you're familiar with architecture, Cubelet runs on every single node and that has a copy of the metadata, so you can optimize. Just a tip in case you have access to the API of Cubelet, yeah, you can point Fluembed to Cubelet to avoid all your nodes talking to the API server. Now I'm going to introduce you a new concept that we recently were talking about, about filters and processors. As you know, historically, what filters does in Fluembed is like, imagine that you have a pipeline and that pipeline goes from left to right. In the left, you have all the source where you are reading the information that is coming from your files over the network. But after that, you have one face that is called filtering because maybe you want to drop data, you want to enrich data, meaning like you want to add some keys, remove some stuff, and then you have the output, meaning like a way to decode that information and send it out. What you have here in the screen is the architecture of the current Fluembed version on how do they work with filters. Imagine that you have thread one, which is a principal thread, that is the main event loop, what is doing buffering, scheduling, handling retries if something goes wrong, but also run filters. Meaning like if thread two is on the left, read information somewhere and code events for the main event loop, it gets buffered, it gets buffered, then it gets filtered if there's any filtering, and then it's passed to the other thread, which does only handle destinations. Handle destination means get the data, decode the event, format the data to the expected backend, and try to deliver. And I say try to deliver because things fails and when it fails, you have to retry. And if you cannot retry, you have to delete, you have to release the space. And this sounds, oh, yeah, it makes sense. This is the new version, so the current version from one year and a half ago because we used to have only have one single thread, Fluent Bit was single thread with asynchronous IO and just one event loop. Now we have three event loops per thread and we have different mechanisms around. Do you see any problem here? Or potential problem? Yeah, it works. Now, what about this? Who can tell me what will happen here? Come on, don't be shy. This is what we had before. We were doing nothing, not much filtering, but now we have more filters. And the truth is that we have found users and customers were running 15 to 20 filters. Insane. I want this, like kids with a candy, I want that, that, that, that. Awesome. At the end of the day, you had too much sugar in the body, right? And the same happens here. When one thread is ingesting data and data is being filtered, that's delaying things. As you can see the main event loop, what it does, buffering, skating on a red trice. But what is the priority now? If you add more filters, you are adding more computing time and you are overloading the main thread. That means that you're not able to ship data fast enough, you're not able to read more because you need space for that. You don't want to blow up memory. We don't want to be a JVM or logstash. So this is what happens, contention. Meaning we are sending data but we have to wait until the whole filters finish so we can continue operating. And this has been a problem, but a recent problem from the last eight months because before that this architecture was pretty fine. Everybody was processing, I don't know, 40,000 messages per second, 50,000. But now we are in 2024, workloads are increasing and you will notice that in your environment. And this is what happened, you are more filters and we are not able to process the data fast enough and the things gets messy and you get contention. And the solution for this has been processors. Processor is a new interface we just shipped a few months ago that has a really interesting value. It's, as a concept, processors are a single chain of processing units. Processors can work over different telemetry types. For example, they are processor for logs, for metrics and for traces. They can run in a separate thread and process can do pretty much whatever you were able to do with filters. And this is the way how it works. Yeah, we give the problem to the other thread, right? But he's a guilty, right? If the user is trying to process more data on that thread, that's fine. But we are not blocking the main thread. Now you might imagine, okay, oh, I'm using filters so I should not using them anymore. And no, actually you can run filters as processors today. We made it in a way that we didn't want to break compatibility, right? So all the filters that you have to modify to the Kubernetes filter or there's some Amazon filters to enrich your metadata with the instance information can run as processors. And I'm going to run a typical FAQ and this question comes from this week at KubeCon because I had two, three questions here and that people was continuously asking, asking, asking. Why should I use FluentD? When should I use FluentVet? And I wanted to give it just a simple, simple summary of FluentD, it's written in Ruby language. Which it works fine, but it's a single thread. It can scale until certain level, right? FluentVet is written in C. And the difference that FluentVet has been designed in the latest years to be a multi-thread program, so nobody longer has one CPU in his computer, right? Her computer. Even if your phone's running more than one core. So why we're going to just use one. And we always keep the same philosophy to run with very low CPU and memory footprint. The other is that when you think about extending your pipeline, because everybody at scale want to bring the business logic into the pipeline, they say, oh, I need a new plugin, a new way to add my own behavior to the pipeline. In FluentD, you have to do it with Ruby, but in FluentVet, you can write in C. Yeah, people don't like it, I get it. But now you can do it in Wasm. So you can do it in Python, you can write in Rust, you can write input and output plugins in Golang, or you can do Lua scripting. If you're familiar with Lua or Nginx scripting that is Lua, so it will be something really simple that you can add to your config maps to add your own logic into the pipeline. And when operating with other ecosystem, as I said again, I want to be very clear on this, we try to integrate the best way possible and give the best solution available in the market for every single user. Every FluentVet is being used everywhere. So that means that whatever you go with any type of protocol implementation, we should be able to solve the problem on that ecosystem. And the question was, oh, can I use OpenTelemetry? How this overlap? OpenTelemetry is a protocol, OpenTelemetry is a specification. But also we have SDKs to instrument your application that they are able to ship telemetry data, right? Now, as you can see here, the overlap that exists is with the OpenTelemetry collector. Not with OpenTelemetry, that is a big difference. But that is fine. I think that it's good that the user must have choices for different use cases. HotelCollector is really good, for example, for tracing, for sampling. And they are getting into there with other features for metrics and logs. We're really good at logs, we're getting into metrics. Yeah, and we are new in traces. Yes, this adds some confusion where we can get started, but always my suggestion is use what works for you and will work over time. And I think that I like to do that emphasis. And same as you can receive from one to one, yes. You can use Flow and Bet as a main arrayator or collector for all your telemetry type of data. Not just OpenTelemetry, but for everything that you have. And as of today, we are 13 billion downloads. And this is insane. Maybe I say the same thing every year, because I'm amazed at CubeCon Amsterdam last year, we were 6.3 billion, right? So now we are two X of that. That means it's not because, personally I think it's not because the project is, oh, it's so great, we're going to deploy thousands. It's like, hey, people is moving to Kubernetes. People is, I don't know, using more cluster, the adoption is growing on the CNCF ecosystem. And now they need an agent to manage logs by default. And it's Flow and Bet. And that's why, and we are really thankful for our users. And this week, we released Flow and Bet version three. So a few things. So every time that we do a major release, and actually the last major release has been here at CubeCon Europe. And somehow we like that stuff. We do a new t-shirt, a new design. We try to give these t-shirts for everybody. But next time I will owe you one, as I explained you before. But let me tell you about what is new. So first of all, Flow and Bet is a small team, right? At least from our company, Caliptia, now Stronosphere, we are like four, five developers, right? Leonardo wrote one of our maintainers, wrote the HTTP to support for Flow and Bet. So thanks so much for that. What that means, that now, if you rely on any type of instrumentation or collect data from application that wants to use HTTP V2, now you can do it. There's no big change from the user's perspective, but you will notice that you will be more performant. Because with HTTP V2 was made for concurrency, something that didn't exist for HTTP V1. Also we get other internal changes on how to manage the data salarization that I'm going to talk to you in a few minutes. Also when thinking how to modify the data, because users always want to modify the data because of business needs, we said, okay, we have a bunch of filters, modify, modifier, grep, I don't know, we have like 20 or more. And we say, hey, most of them are kind of overlapping and functionality are doing pretty much similar things. So we are chipping today one more to just try to reduce the complexity that is called content modifier. But of course, this is a processor, not a filter. The good thing about this processor is that it operates on top of logs and traces. For logs, it operates on top of metadata or attributes plus the body content and in traces, of course, on top of the spans. And the operations are supported, insert, absurd, delete, rename, hash, extract, and so on. Somebody asked me this week, hey, why do you have a hash function? And in the product that we offer, also we got the same question, why you need to hash? There's many use cases where even in Europe with GDPR, you are collecting information but you're not able to send all the information in plain if you have sensitive data. For example, social security number, user password, but you have to send the key and some value. Hash works for that. Of course, the idea of hashing is like it cannot be a reverse process and so on. And the next one is a metrics selector. As I told you, flow embed can extract a metrics, it can receive metrics over the network, but at some point you would like to exclude certain metrics that you are not interested in. Why you're going to send everything to Prometheus or everything to your metrics backend? You don't need all of that. So the metrics selector allows you to filter, to include or exclude certain metrics by its name and shortly by labels. That means that it could be really easy to manage. And the last processor that we implemented before in this release is a SQL processor. You will say, oh, why SQL? Are you using SQL Lite or my SQL unit? No. The thing is that we found that what users want to do with data is basically sometimes select certain keys and sometimes are thinking about expressions in their mind. Hey, I just want to hold the data that this value is greater than 10. And that's it. And our option that we have available before was to, hey, write a Lua script for that. I don't know Lua, I don't have time. Oh, maybe you can use some filters and this person waits five to six hours. Right, or more. So we said, okay, why we don't implement a SQL processor? We used to have a stream processor in FluentVid. It's still there, but it works different and it has its own issues at scale. So we said, hey, let's implement a new SQL processor that allows to select for locks primary specific keys, rename them and put conditioners on it. So with this new SQL processor, you can do that. And now I know this question is coming. Oh, does it support aggregation like count, max, mean, group by? Yes, that will be available in two weeks. Not today, in two weeks. So yeah, this is a new SQL processor. And one other part of the news is like, when you're thinking in an open telemetry, for example, in the ideal case of this, this works perfect, but what happens in reality, if you go to production environments, it's like on the left, you don't have open telemetry. You have syslog, you have UDP, you have TCP and any kind of protocol that you can imagine. And the structure of that data, it was not meant to follow an open telemetry schema. But you need that as an open telemetry schema. So one of the important updates in our open telemetry output connector is that allows you to customize in a very granular way the message that's going to be sent as open telemetry format. For example, you are getting a log message though you have five keys, but you know that two of them conceptually or logically are metadata or attributes. And you want to represent that in open telemetry. Hey, these two keys are metadata, everything else is part of the body content. So now you have that type of control. And before ending, I wanted to show you a little example of how this works and the new processors content modify. Can you read the screen or so small? Oh, you need glasses like me. That's better? Yes, okay. This is my sheet sheet here. Don't take a look at that, those are comments. So we have here, yeah, if you're looking at this, you are not familiar, we ship a new YAML format some months ago. So the classic configuration mode of Fluentbit works, but also we have a YAML compatible version. Okay, we encourage everybody to switch to YAML because there's more tooling for validation. Okay, so we have defined a pipeline here and now we have defined the input plugins or the input sources where the data will be received or generated from. The first plugin that I'm loading is called Dummy. Dummy, what it does is just send the same message over and over because we are doing a test and just to explain the concept, we're going to generate a JSON message like this that it has one, two, three, four keys with different values. Key one, key two, message and HTTP URL. And then we have attached a new processor. As you can see, the processor is under the plugin. Now you cannot use processors in classic mode. If you're using the other version of Fluentbit, you're using classic mode, you cannot use processors. Okay? And we want that everybody moves to YAML, that's right. So yeah, we have to force the user somehow, right? And well, here we're loading the content modifier processor and the action that we are taking is going to insert a new key on the record. Right? The key name, the key value. And what we're doing is to send this information to the standard output. I'm going to run this locally in my terminal so I'm not running Kubernetes, nothing fancy here. Very basic. Can you see it? Let me see if this doesn't break. As you can see every second, it's printing the same message but you will see that at the end there's a new key and that key is called new key, new value. And let me help you here. Maybe you can use JQ. This is really good for debugging so you can learn something about how to debug this stuff. So there you can see that a new key is being generated. Okay, it's not a big deal. But internally, this new processor, I don't know if you remember that I mentioned that we have a new serialization format, a new way to work with the API. Right here, new internal API for data manipulation. This new processor is using that, which is a new mechanism, a more performant to perform all these modifications on the records. Now, okay, and for example, imagine that now we want to hash message. Well, it's a value new, but anyways, you can hash it. I'm going to break, I'm going to start again. Let me close this one. As you can see the message now value has been hashed. And now we can do something more interesting. We can operate in an action that is called extract. Sometimes you have data that in your mind means something. You know that if you read an address, you will see, oh, this is a protocol, this is a domain, this is a subdirectory or path. And sometimes you need that information to be extracted in different ways. You used to write a Lua script for that or run a filter or a regular expression, but now this processor supports that. And that means that what we're going to do right now is to extract the value of that key by using this regular expression. If you don't like records, that's fine, nobody like it. And, but there, the good thing is that here you can have the group names and the values will be assigned it automatically. So the goal is to just get this key and expand its value as a key value per. So let me save this one. Let's go again. And now you will see that we have, oh, let me stop it so you can read. And now you can see that we have the HTTP URL has been split, split, oh, new key, well, this is an order matter, has been split in four new keys, HTTP protocol, HTTP domain, HTTP path, whatever. Everybody who's using a Splank or Elastic sometimes is running these queries when the data has been ingested. The value here is that you can do it before. And what is the gain with that? It's not the same running a query when you have ingested thousand and thousand of messages than doing this processing in parallel while the data is being collected and ingested. And the CPU cycles are very, very low. So we are always encouraging and also seeing some users that all these analytics queries or processing are moving from the left now, sorry, from the right to the left. They're really interesting. And finally, if you don't want, for example, in this example, this key that was just extracted, you just run a new action and you just delete it. So you can play with logs like with anything, right? It's pretty simple. And let's talk about SQL, the new SQL processor. Now we have a small version, right? Of the same, well, it's a small version of the same dummy message where we have a key, we have the HTTP address. And we're going to do first is to enable the processor and we're going to extract that key as a multiple key value pairs. But as a, what our boss asked us, hey, give me a list of all domains that are being scraped. I don't care about anything else. Well, at the end of the day, it could be IP addresses or anything that's more useful than just the domain. So we're going to do it in one round. So the first processor is going to extract, right? The message. And the second is what is it going to do? It's going to select the new key that does not exist here, but we can, oh, this is a type of, we can, okay, key one, we're going to select key one, but I'm going to rename it as key. And then we are going to select HTTP domain from stream. From stream means like it's coming from here. We don't have tables. Everything happens in memory, okay? And then we're just going to switch the example. And there you go. So every one second is creating a new message. So this is how you can extract messages. You can mix content modifier, running a SQL query. And of course, and today we've got the first question. Hey, is it possible to run SQL over metrics? I was like, not yet, but that's interesting. And then we were thinking like, hey, fluent bit ship internal metrics will be good to provide a SQL interface for the internal metrics. Interesting. So there are many things that can be done here. And lastly, maybe I can show you the not exported metrics. So I have one VM that is running here. It's a Linux VM. And they're going to run fluent bit from the command line. So when you run a fluent bit, and you put minus H, you will get a ton of information around the input plugins that are built in. Everything is built in. You can have external plugins, but by default we ship ton of good things. Then you have the processors available. Oh, we have a labels processor to play with labels. And then we got filters. They look old, but that's fine. They can be run as processors. And then they all put connector for Azure, or I call Google, forward, H2PM points, whatever you can think. So we're going to run what is called the not exported metrics. I'm doing it on Linux because we have more metrics collected on Linux than on Mac OS. Yeah, and I'm going to just, oh, we can expose it to, okay, let's send it to Prometheus exported. Let me see if I'm not wrong. Oops, I put something wrong. Red permissions from, oh, I should not, oh, it's a thermal zone. Okay, so we are running the Prometheus exported on 2021. What that means, fluent bit right now is gathering metrics from the file system and exporting in a buffer into the Prometheus exported metrics. So if you do curl H2TPs, 2021, and you put metrics, I just did a scrape of the metrics that fluent bit is exporting. That means that fluent bit can behave as a Prometheus exported. Okay. And if you can start seeing something similar, like I'm going to send these to the exporter, but also I'm going to send the data to STD out. So I'm going to see the data on both sides. I'm going to get a match rule. You will start seeing that on one side, the data is being generated. Not generated, oh, the buffer is ready. So from one side, we are sending the metrics to standard output while on the right, we're just querying this by using curl. This is a simple concept before you can integrate the things around telemetry, playing with logs and metrics. So I owe you the traces example because we just run out of time. But anyways, I would like to invite you to sign it up for the book, for the webinar and give some space for questions. Thank you. Just raise your hand. Please. The question is, if we support our own logs as JSON, yeah, internally in FlanVid, we don't do JSON. We do message pack, which is a binary version of JSON. But if we can ship our own logs as JSON, no. Yeah, it sounds like, yeah, we don't do it. Most of people don't need it. But if you are interested in that, into that, please open a GitHub issue and we'll be happy to support. Well, if you have more questions, oh, we have a more question here. Oh, you want to know about the architecture? Oh, he's asking if we can go back and see the comparison of that. Okay, I didn't get the last part. Sorry. Why do you stick to just press rights for this instead of, for instance, having a workflow for the filters? And so I was kind of wondering, where does the coroutine sits in those press rights? Oh, where the coroutines? Yeah, I mentioned that. Yeah, okay. Yeah, for the background. So we're going to go more technical. Yeah, Fluentbit, when it was created, it was a single process, right? A single thread. And in order to process data, and without blocking, right? You have to do something with the operating system. For example, when you are doing computing or doing something, there are always two contexts in the process, right? There's user space and there's kernel space. And sometimes through system calls, you want to achieve certain things. For example, I asked the kernel A, give me a socket, connect to this endpoint. But while I'm connecting or doing a DNS lookup, I send out a DNS request. I need to wait for the response. That will take some time. And coroutines help us to say, A, you know what, you have an event loop. A coroutine is like a lightweight thread where it says, we have an API for that that says, okay, I'm going to do a DNS lookup and I'm going to suspend. While I'm suspended, the event loop was still working, scheduling, reading data. And at some point, the kernel is going to signal back the event loop and says, hey, you got a response. You are ready. And that coroutine, that context, now can resume and continue operating. Okay, this was when we had a single thread. Now, in the model that you see in the screen, now we use coroutines in the output. So every output thread that you see can spawn 1,000 threads. Sorry, 1,000 coroutines. And everything looks really smooth. Let me check if I have, let's take a look at the code. Yeah, if you are here, it's because you have time, right? Okay, yes, but let me jump into this because we won't get it online. So this is the HTTP output plugin. This is called, okay, pretty quickly. Okay, plugins registered with a structure. We define callbacks when the plugin gets initialized set, when the plugin exits, and when the plugin is going, needs to flash information. Okay, the plugin doesn't manage any logic. It tries to get the data, process, and deliver. So if we go to the flash callback, oops, oh, oops. Let me check this. There you go, flash callback. Let's focus on this important part. Oh, I choose the more complex plugin for this. Sorry about that. This has changed so much. Okay, we have a function that is called HTTP post that is here. Okay, take a look at this code. It says FLB, which is a prefix for Fluent Bit API. AppString context get. Do you know what does behind the scenes? It does a DNS lookup when it knows the IP, it perform a TCP connection, and if it's a SSL connection, it will do all the TLS handshake. And once it's ready, it will return here. But imagine that the remote endpoint takes five seconds to respond. The program doesn't block. It runs under a coroutine, the flash callback under a coroutine. It just send the request, wait for the kernel for all the step that it needs while Fluent Bit keeps working on that output thread that we have keeps operating in other coroutines. I don't know if you remember if you are old enough as me when you were running Windows 95 or Windows 3.11, you used to have one CPU core. And how was possible that I can move the mouse, click and open Windows? They're running some kind of lightweight threads. You have one CPU core, how that's possible because what you're doing is doing a micro execution of code under a very small single unit of time. And with coroutine and the scheduler and the event loop, we do the same. I don't know if I answered your question, but no. Okay, sorry about that. Hi. So, when we're using processor to hash some GDPR data, can we hash it with some R-provident key to decrypt in some cases like legal requests, et cetera, et cetera in our backend for future? Today we support SHA-256 and it's hard-coded, but if there's any good use case to say I would like to show my hashing function. Since we use OpenSSL, we have a bunch of them to choose from. It would be great to know what would be these cases. Just open a GitHub issue, we can implement it. Okay, to provide some custom private key. Or to sign the messages. To decrypt it for some legal requests. Okay, so what we're doing in the hashing algorithms, hashing is not in reverse mode. So it's not encoding, what you're talking about is encoding. Code and be able to decode. So with a GPG, for example. So you want to sign a mess, encode it, sign it, and decode it. Yeah, we don't support that at the moment, but it sounds really interesting and just open a GitHub issue. Okay, thanks. Thank you. If I can do manipulation with Wasm plugins. Yes, but you might have to implement like a filter that does that custom hashing. You can do it in Lua too. Yeah, Lua runs, same, Lua originally is a filter, but can run as a processor because they're compatible. Yeah, you just write a function that takes a value, applies sign it with a key, import your Lua packages, and. Hi, thanks for the presentation. Is there a plan to add a Microsoft Sentinel output? Just like we have Forced Plunk, for example. It's possible? Yes. So you might open a request on GitHub, but it looks very enterprise use case. So I don't know if the product side has something about it, but I never use it. So I would not say, yeah, we'll be ready in two weeks. No, it's not like that. Maybe you can open a GitHub issue. The way that it works is like, most of the stuff that exists today in FluentBate has been because users feedback. And sometimes users don't get it, but it's real. You open a GitHub issue and you get a lot of people saying, hey, this is interesting for me for XYZ, I will need to use it for these reasons. It's a better way for us to prioritize it as a project. Thank you. And when you say it was first single-threaded, do you refer to FluentBate 1.x? I think until 1.x, yeah, I think that after two, we introduced threading, but in a very optional way. Now, most of plugins are by default running a separate thread. So every output plugin that you create in the configuration automatically runs in a separate thread. Okay, thank you. Yeah. Hello, as it is a new major version, are there breaking chains for the actual config in previous version? There's no breaking changes on configuration, on this new version. Okay, thanks. Okay. Okay, thank you so much, everybody. I hope you have a safe return to home. One.