 Hello, good morning one of the as Well, I hope that you're doing well, and you had a safe trade to come here to keep comes so welcome to Valencia My name is Eduardo Silva and one of the maintainers of the fluent ecosystem And I created the project fluent bit now maintaining board more companies more people around and Well, thanks for joining this fluent con This is a really small collocated session that we found that has to get a lot of value Along the years if I'm not wrong. This is our third time doing this event and We are really anxious to know more also about your journey. It's not just about a Maintainers and developers sharing knowledge, but also is important to us to have this interaction with you After each session to get received some feedback and understand what are the challenges that you have in observability Right, so just take it that in cone that everybody who's here is really open for networking Understand a what is your journey in observability and how we can help in different angles So we call that the fluent ecosystem is like the Swiss knife for observability and One of the main differentiators along the years thinking about that fluent D as a project it started like more than 10 years ago It's like we never intend a as a project to be a dropping replacement We always intended to be something that you can plug in your architecture and solve your problems When it's about to move data from one point to the other and now when we think that in That mentality we think that the fluent is more than a tool. It's more than fluent D. It's more than fluent bed It's a concept that allows you to integrate With different systems different architectures, but also other projects around Pretty much a part of the history is like from the beginning. We aim to even connect with competitors projects now talking a more about from the ecosystem we try to connect with Everything that is around because the problem is not just a moving logs from one place to the other There's traces. There's metrics, but also you don't want to remove your your infrastructure agents, right? moving infrastructure agents a for example Anyone that you can have might take you up like a year We have a users and customers who deploy this in a scale of a hundred thousand servers, right? So when it's about to switch technology, it's really expensive Because it takes a lot of time a lot of planning Maintainance and so on so for us fluent is a full ecosystem that allows you to get into this journey on how to connect the dots between Different a source of information different destination and nobody what is the data? We try to be diagnostic today is lost matrix and traces maybe tomorrow another kind of Critical information might exist and you will be sure that we will support that one of the values is like we think that Everything in observability must be open source, right? And not just because I say this because we are an open source conference I say this because of the value of contribution across different vendors Companies and individuals of this but also is really important that you don't get into this common vendor lock-in Because when you go into observability, even if you think from the management perspective you think that what you need to get is What we would you need to accomplish is data analysis you need to centralize your all your information in a central point You can call it the last day. You can call it a splunk. You can call it Prometheus. That's not a What is relevant for us our values to be able to provide you the tool that you can choose your own back end or Backends or cloud services in a smooth way, right? And as I said, we try to be agnostic in all ways possible in terms of project products and platforms Now where is fluent right now? We found that when the cloud providers, you know botched for a technology every other company mid-sized or small-sized follows the path and Nowadays Amazon Google cloud Microsoft Azure plus another hundred of companies are using the fluent technology in their own Infrastructure, so I would say that this has been production grade for a long long time Now thinking about the ecosystem is always to year over year start thinking Okay, where are we roots where we are now and where we want to go and there are many areas to To think about that for example when flu and he was created the problem Well at that time was how do I move my login information to a cloud service, right? And of course the technology that was created to solve that problem was flu and D and actually part of the story is like You know many people complain that flu and D is written in Ruby language with some components in C And one of the fun part is that the POC Was meant to build in Ruby But the real solution was aimed to be built in C++. I'm talking 10 years ago, right and it worked out pretty well And so this POC just stayed in Ruby and starting evolving evolving and nowadays is still used in widely Short after fluent D when we got fluent D in production already in thousand of servers We decided what we needed to have a kind of solution that was more a Lightweight more performant with low memory footprint at that time It was in better Linux, right? that was a target and then at the time that fluent D was around Kubernetes was around right, but also start to explode all these a technology of containers right, I'm talking about Docker at that time and fluent bit For somehow got like a better fit for this new area of microservices and containers Why because it was reading in C. It was optimized it to consume low CPU low memory But have a very high throughput when it's about to move data from one place to the other so naturally Users starting Migrating from one project to the other in the initial intent Was the idea was to have fluent bit like something natural for fluent D users Where fluent bit couldn't work like a suck forward there forwarded in our context is something that just takes data and forward it out and Have fluent D as an aggregator that just receive all the data and take care of all the all the processing But our users starting asking for more features in fluent bit Like we didn't have filtering at the beginning, right? We didn't support Kubernetes even we didn't support Taining files one of the first plugins available at least in fluent bit where to gather kernel log messages CPU metrics memory metrics those plugins are still there And this is really really fun because the trend now is that most of users at Switching from fluent D to fluent bit because they're following a journey Where now they're moving to Kubernetes or they're moving to a different kind of architecture and when it's about to move data We got a problem Right five years ago. I don't know people were processing a few terabytes. Maybe per month sometimes now is terabytes per day, right so Every year every company every architecture is moving more data. Well, actually it's generating more data Right, so if you're generating more data You need to be able that your agent or the tool that you have in your infrastructure Be able to move this data in a scalable way Right and that's why fluent D is starting to struggle on this new era of containers Because it was not designed for this kind of workloads But fluent bit was designed for that and that's why we are seeing this pattern of a fluent D users moving more to fluent bit Now talking as a maintainer and as a founder of a co-founder as a company We are investing in both projects fluent D. It might be around for another 10 years fluent bit Maybe more but all the innovation and the train is happening more in the fluent bit side and One of the things about fluent bit is that production grade system is very high-performance and consume Very low resources. Of course. It's not the same process a thousand messages per second That ten or twenty thousand of course we are going to need more competing resources for that, right? But our challenge as maintainers and developers that every year we need to optimize more We need to find okay how to reduce memory allocation What kind of trading strategy we can use how we can optimize in data? Celerization before moving the data even how we optimize when it's about to encode the data into Jason Which is a really expensive task And as you might guess every destination waits for a Jason payload So no matters if in the agent we have a binary representation of the data, right? We need to convert that back to Jason And now a Which is interesting Same as the pattern that happened with Linux same as the pattern that happens with Kubernetes We started to get a distributions of fluent bit the third distribution Well, of course is the upstream distribution where we publish packages container images And then what followed after that was AWS when they said Amazon told to talk to us and well AWS are maintainers of the project to and they said hey We have our own specific backend services and we want to provide You know a first-class connectors for our users by using fluent bit, right? So we started this synergy or working together and they created their own a Plugins initially in go now in C and they say we need to package this for our customers So they are using AWS for fluent bit AWS for fluent bit is just a fluent bit distribution that contains I'm extra Amazon and connectors now you might ask why AWS for using the forward and not AWS from bed It's a just a foundation policy until we got some conformance committed about we aim to change that shortly and Calypthia the company that are representing Also, we are launching our own a fluent bit version, but this is not is upstream version, but kind of LTS We got many customers that said a okay I want to run fluent bit, but fluent bit upstream is released every two weeks Right and sometimes they're breaking changes. We try to improve in testing and stuff, but sometimes things happens So Calypthia for fluent bit is an LTS version that will last for 12 to 18 months And you can run safely on most of environments And also we have Google who's investing heavily also fluent bit Where they are creating their own agent which is kind of wrapper that a Google ops agent use open telemetry for traces and Use fluent bit for all the login management So if you're running Google cloud, likely a Google ops agent will be available for you If you deploy for example a Kubernetes cluster in Google cloud You will get fluent bit as a default bot running So as a sorry as a default demons and there to collect all the logs and ship all these to stack driver so cloud providers in in this aspect from a project and community base is really important because you see that a Technology is has been growing and everybody now is contributing back to have a more general value and fluent bit and I Missed to connect the dots between all the sources available all the destinations So this is not new actually. This is the the first button that we envision when fluent D was created as part of the community We are seeing a huge traction on how many docker hub pools. We're getting every year if you compare from 2019 when we started publishing. This is just containers public containers. We started just a few hundred right now Oh, this is the order of millions. Sorry, but I just forget the unit Right. So on 2021 last year we hit 600 million deployments and This year and if you think we're just in May, we're just approaching that value Actually, CnCF just reported that fluent bit was deployed more than a billion times Right and that is insane. It's insane because it's a huge adoption Right and now with more adoption you get more back reports more enhancement requests And this is like a snowball that continue growing and it's really interesting because there are more challenges You might think that an agent that solves a data problem will be done But actually every year at least in the count in this cloud environment Is more heavily every year's more challenges more stuff that we need to do or innovate into the space As part of the investments from my tenors and other companies We have been working on how to make the fluent bit project a stronger, right? How to avoid regressions how to in increase our coverage of issues So we incremented all the overseas CICD pipeline on github So every pull request everything that has been merged go to a very strict Sanitization process and making sure that there's no problem and Also, it's no problem running in different architectures, right? We aim to support the four basic architectures x86 in two flavors same as arm and also making sure that there are no regressions As a part of security is like okay fluent bit is reading in C and there's always this This fear. Okay. What about memory safety? What about crashes like folds? So from a language perspective, there's no much that we can do besides best practices Right. We try to enforce the best practices a run memories and it is a sanitization and making sure that there are no leaks around But also we start in working a with a security company that we have a presentation about that in a few minutes And that integrated all the Google open source fast mechanism to influence it What this means fasting is a security mechanism where you take any Component of the project that receives an input, right? And it tries to put a similar input or a random input or a modified input that aims to make the project to crash It just send trash data And what is the interesting thing in the first day that we run this we found many bugs many many many many and In the last 12 months all of that has been fixed They are not relevant backs that you might face in a daily in a daily basis But if your service is being attacked you might face this kind of problems But Google OS faster the good thing is that the runs 24 by seven So every what we are doing is that increasing the coverage of the fosters for every interface in fluent bit And the result is you have a more stable agent as more stable product And also as a project perspective you might know that fluent D was created for locks fluent bit the same thing But as of last year we started extending the scope of the project and now we support Matrix and this year we're releasing traces support So and this lead us to the next topic locks matrix and traces and you know might be thinking about Prometheus open telemetry And this is really interesting It's like for the login. I think that we don't need to elaborate too much We support on structure structure messages We do all of these in a schema less fashion and we do all the processing filtering Enrichment of data and one of the good things that we allow in fluent bit to bring your business logic into the data pipeline So even if you want to do some scripting and says if this record contains Certain key with certain value you can take some decision of that modify the content or also making sure that data that a Matches certain criteria can be sent to a different destination and this is usually common for security and alerting Now the matrix journey is really interesting because most of a I don't know which I think was a coupon Europe some years ago and they told us a I'm using what was an example that moment I'm using a Prometheus not exporter. I'm using fluent bit not exporter for matrix fluent bit for locks Can you replicate the same functionality in some fluent bit? And it was not just one person many many people were asking about that So we say okay now maybe we can do metrics right actually fluent bit does metrics from the beginning but as locks But we never formalized at that So we start in extending our our own core engine to support native metrics payloads Right, but following our philosophy on being bend diagnostic and be able to connect on the different platforms and ecosystems And now it's compatible with open metrics, right, which is a Prometheus standard and open telemetry metrics Now from a plug in perspective what we did was and you know our pipeline has inputs We are sources that are plugins that can read data from some place or receive it And we have a outputs right. How do we connect to a destination in the input side? We implemented now a Prometheus scraper so fluent bit can scrape your Prometheus endpoints If your application is running exposing some metrics We just can connect fluent bit scrape those metrics and get it get them into the pipeline Also, we have another plugin which is the mimic of not exported from Prometheus that runs Locally and it's create the same metrics that not exported us We support a coverage around 60% of what the real no exporter does, but I think that that covers 85% of majority of use cases That's why we get the data into the pipeline now in the output side You can expose this data with Prometheus exporter plugin that we have or remote, right most of a services that are Supporting Prometheus remote, right like ball new railing Google you can connect to them by using these connectors So you can put the fluent bit in the middle of something no matter what you have as logs metrics You can use it to get this data and ship this data to your Most famous destination And the same is happening with open telemetry the way that the fluent process open telemetry We say it like it's where the industry want to get a standard for example from if you go to any environment The start the industry standard for metrics everybody will say Prometheus, right? Everybody is Prometheus that is perfect Open telemetry if you ask for example, what is the industry standard for traces? Everybody will say now it's open telemetry But as you can see the standards always are switching or evolving, right? But at some point you are overlapping features You are overlapping protocols and you need something that runs in the middle that allows you to connect the dots Right and that is fluent bit and in the open telemetry side. We are just starting our experimental a Open telemetry input and open telemetry output a Plugins that now for now support just metrics payload and we are extending to traces in a few weeks But we don't aim to replicate for example the open telemetry collector We don't aim to implement for example traces processing I think that that is open telemetry job, right or Bendos job Our job is to be able to move the data, but in the matrix space We are also we are going to allow to do some kind of metrics processing and Yeah, this will be launched in Q4 2022 Another stuff that is really important is investment in performance as I said at the beginning of the presentation, you know You and your environments are always getting more data more data more data, and we need to be able to Handle that load. So we have another presentation from Amazon where they contributed a new mechanism For a the event loop that we have now you can have priority cues and all the events run smoothly There there's no even loss there The agent does not get stuck because the way that is being tested from it now It's not the way that it was tested Some time ago, and we did a lot of optimizations on trading input support and in court on how do we manage a memory and all that stuff From a developer experience Okay, those are the users now developers is how do we extend this agent? How do I put my business logic? How do we implement a for example? I'm a company and they want to provide a new way that I take advantage of fluent bid users so they can connect to my services So we from some time ago we support goal and in the open side of fluent bid But one of the missing pieces was the input side And now we are launching shortly this year goal and input plugins. So you might have your own Fund service and you want to expose on logs and you want that fluent bid pick up those logs or that information You can write your own goal and plug in for that and make it a available So with that we we conclude the presentation so the invitation is like be open for Observability always be fluent like water as Master Bruce Lee is to say so Yeah, thanks for attending and if you have any questions I think that we have one or two minutes for that. Thank you any questions Don't be shy first day You can make the question in Spanish if you feel more comfortable We just have like one minute. So you can make it you can talk in private if you want if you prefer I think we have next speaker coming up. Okay. Thank you