 Hello everyone, this is Feynman Zhou from Beijing. We're glad to be joining this open source summit hosted by the Linux Foundation. Today, we're gonna tell you about the topic under the hood with FluneBit operator, which is a Kubernetes native log processor. Yeah, my name is Feynman Zhou and I am a CNCF Ambassador and CubeSphere Community Manager. And I'm also one of the contributor and a member of Flune's community. Hi, I'm Dhruv, I'm a software engineer with the marketplace team at DigitalOcean. I'm also a member of the Fluent team and a contributor to the FluneBit operator. Okay, thanks Dhruv. In this session, we're gonna walk you through the challenges of logging in Kubernetes. And next, we're gonna introduce what is FluneBit and the challenges with FluneBit. And next, we're gonna tell you why we create the FluneBit operator and we're gonna introduce more about its architecture and workflow. And as a final, we're gonna introduce more about their use cases. You know, the FluneBit operator in DigitalOcean, APB platform and the CubeSphere logging system. I think you guys may know the Game of Thrones. This is the screenshot from the Game of Thrones. And you know, the white workers tell us that we have a lot of challenges of logging in Kubernetes. You know, we have multiple and different data sources. We have different data formats. And the white workers King said that we need lightweight, security and resilience. And we also get the challenges of, we have multiple outputs and destinations. Okay, let's see what the details about the challenges. You know, we have different logs from multiple places. You know, we have metal or VMs or we have the embedded devices and edge devices, especially in the distributed environments, you know, the Kubernetes environment, we have containers or ports or even in the networking environment, we have the TCP or UDP logs. We are also facing the different data formats, you know, the JSON, Apache, engines logs, Docker and CLI. In some typical scenarios, we want to shift the logs from point A to point B. So we might get the challenges of the different outputs and destinations. So typically we have, we have multiple output plugins in Flonbid operator. You know, the elastic search, loggy, Splunk, MongoDB and S3. Yeah, our clients also told that we need to collect and process the logs in a high-performance and efficient way. You know, in the multi-tenancy environment, we still need to handle the data security and reliability. So those challenges are difficult to deal. You guys may know how to debug your workloads or containers in Kubernetes. Yeah, you can try to get the Kubernetes logs from this command, yeah, kubikato logs, pod and containers. Yeah, this is a native method that we used to retrieve the logs in the Kubernetes ecosystem. So, you know, the cloud providers also provide a lot of logging solutions. Maybe you are using the Stack Drivers or the CloudWatch provided by AWS. Yeah, you know, the SVs also provide a lot of logging solutions, you know, the Splunk, some logic, the log and the song. Yeah, we still have some open source choices, you know, the EFK stack or loggy. Yeah, there are a bunch of choices you can choose if you want to debug and retrieve the logs from your Kubernetes ecosystem. Yeah, I also list a table of their solutions and give their pros and cons. Okay, for the Stack Driver and CloudWatch, if they are easy to use and low maintenance efforts, but they have, you know, the vendor locking or it has high codes for their solutions. As for the SV solutions, you know, this Splunk, Sumo Logic or Datalog, Datalog Story, they are also easy to use and you are still facing the same challenges when you use them for your environment. Okay, you know, the EFK is a standard, we say the defector logging solutions in some of the enterprises, you know, they are most popular logging solutions and they are very mature, but they require the higher resource consumption in some scenarios. So if you want to retrieve the logs in the edge devices or the embedded or even the, you know, the K3S clusters, EFK maybe is a little bit, you know, the high resource usage. Okay, you know, the logging is a promising logging solution in the cloud native ecosystem, but it has changed its license a couple of months ago. In this session, we're gonna deep dive into the FlunBit and FlunBit operator. So FlunBit is a lightweight and high performance log processor. So you can use FlunBit to process forward and distribute logs to different places. So I think FlunBit is a Swiss knife for logging processing. This project was founded in 2015, created by Treadedata. Now it is a sensitive sub-project under the umbrella of the ecosystem. Due to this project written in C-language, so it is very lightweight and zero dependencies has around 70 plugins available. So, you know, the input filter, the output and the router plugins in its ecosystem. So you can also write your own plugins in the FlunBit ecosystem. So, you know, FlunBit has only 640 kb, so this is very low CPU and memory usage. This is the general workflow for the FlunBit in Kubernetes ecosystem. So you can use FlunBit to ship the logs from Kubernetes pause to the destinations, you know, the Kafka elastic search and MongoDB. This is the general login pipeline with FlunBit and their plugins ecosystem. Generally speaking, we have the workflow from input to the routing, and they will route the logs from filter plugins to the different outputs. So this is the popular plugins in the FlunBit ecosystem. So as for the input plugin, which is used to gather information from different data sources. For the parser plugin, which is used to convert the unstructured data to structured data. Yeah, as for the filter plugin, which is used to match, exclude, and then reach logs with some specific metadata. So it has the grep, lua, modify, and Kubernetes filter plugins. So you can leverage them to filter out and then reach your logs in this login pipeline. Okay, if you want to ship your logs from different sources to the different destinations, so we can use output plugins, you know, the file, forward, Kafka, loggy and elastic search. You still have more plugins in the output, so you can choose or you can custom your own plugins if you want. Yeah, you will find FlunBit is not a silver bullet if you have been using FlunBit for a long time. Since you may be facing the challenges of FlunBit, you know, they have some typical issues which is described there, you know, the restart mechanism. You know, FlunBit has not been support the dynamic restart gracefully. So if you want to modify the configuration about FlunBit, you have to edit their demo set and you need to reload and restart the FlunBit itself. So this is not convenient for folks if you want to modify your log in workflow. So you have find that there are some typical issues in the FlunBit repository, yeah. And you will find the time was, you know, the three years ago. So this is why we create the FlunBit operator which is used to facilitate the management of FlunBit. The core functionalities of FlunBit operator brings to the FlunBit ecosystem. We can say, you know, the FlunBit lifecycle management it will help you to deploy and destroy FlunBit automatically. And it also provides the custom configuration. You can select different plugins, you know, the input, filter and output plugins while the labels in Kubernetes. You know, the most typical feature that provided by the FlunBit operator, it helps you to handle with the dynamic reloading. So you can update the FlunBit configuration result rebooting the FlunBit pause itself. So we can say FlunBit operator helps us to facilitate the deployment of FlunBit and it provides great flexibility in building a login layer. So in DigitalOcean and KubeSphere team, we use the FlunBit operator to deploy and manage our login system and to retrieve the logs from different resources. I think Drew will help you guys to deep dive into the operator pattern and the FlunBit operators architecture and their design. Thanks. So before we actually dive deeper into the FlunBit operator workflow and architecture, I wanted to take two minutes to address what is an operator, what is a Kubernetes operator? A Kubernetes operator is a method of packaging, deploying and managing a Kubernetes application. It essentially can be broken down into two parts. CRDs and the controller. CRDs stand for Custom Resource Definitions. Custom Resource Definitions are a way to extend the Kubernetes API with your own set of APIs. So for example, by default, the Kubernetes API can be works against pods, deployments, services, et cetera. But through CRDs, you can extend it towards your own application. The second part is the controller. The controller takes care of three things. And the first part it does is watches for changes in application state. So if there's an application state which you're interested in, it keeps watching it and it keeps tabs on it, essentially. It then looks for differences between the desired state and what the actual state of the application is. And if there is a difference, it acts upon it and makes sure it reconsigns and the actual state is equal into the desired state. So yeah, the next question is like, why do we need an operator? Why can't Kubernetes handle this by itself? So Kubernetes could actually handle it and manage applications by itself if they were stateless. Stateless applications such as web apps, mobile backend, API services, any application which doesn't require any additional knowledge about how you should be running these applications. But when it comes to stateful applications, for example, a MySQL instance or monitoring systems, databases, it requires additional domain-specific knowledge that Kubernetes doesn't have because every, let's say, instance of MySQL has its own identity. It needs the knowledge in order to scale, upgrade and reconfigure these applications. Kubernetes operators essentially encode the specific domain knowledge into Kubernetes extensions so that it can manage and automate an application's life cycle. Okay, QBuilder is a set of tooling and opinions which helps you scaffold your operator and helps you translate your code into your CRD file. Controller runtime is a part of the controller aspect of the QBuilder, which helps you build anything to do with the controller, for example, watching and reconciling. So, FluentBit operator defines six CRDs. The first one is the FluentBit, DemonZ and Config. This FluentBit Config CRD essentially just selects the input filter output plugin and generates the final config into a secret. And the input pass or filter output CRDs define the respective config sections. So, each input filter output represents a config section which is selected by the FluentBit config via label selectors. The operator then watches these objects and constructs the final config and stores it as a secret. The secret is then mounted on with the FluentBit DemonZ. To enable FluentBit to pick up these updated secrets and making sure those changes are reflected in FluentBit, we've created a wrapper called the FluentBit Watcher which essentially looks for changes and if there are any changes, it restarts FluentBit. FluentBit is FluentBit operator at Digital Ocean and for the Digital Ocean app platform. This solution app platform is a fully managed platform as a service that lets customers build, deploy, manage and scale all kinds of different apps. The FluentBit operator in the app platform is used to solve the problem of logging and displaying logs to the end user. One of the main challenges when the context is the past is aggregating and forwarding logs reliably in a dynamic environment with hundreds of nodes, thousands of workloads. It is absolutely essential to make sure that all of these things are happening, all the logs are being delivered in real time and reliably to our end users. So how do we use FluentBit operator in the Digital Ocean app platform? So we have our customer workloads running on docs which is the managed Kubernetes service by Digital Ocean and they are continuously generating logs. So we use the FluentBit operator to make sure that FluentBit is capturing the latest logs as soon as there are any changes that happen. We make use of the go client which the operator provides to filter out and parse the logs which are essential and are in a structured manner so that we can actually return it back to our end users. Yeah, thanks for Drew's introduction to the use case of FluentBit operator in Digital Ocean app platform. Yeah, actually we see Digital Ocean team has made a lot of great contributions to FluentBit operator. You know, they have add some great features and enhancement at FluentBit operator and also they brings more plugins into the plugin system of FluentBit operator. Yeah. So for the next, I would like to introduce how we build the locking system with FluentBit operator. Yeah, this is the built-in locking system in KubeSphere. KubeSphere is an open source container platform built on Kubernetes. If we say Kubernetes is the Linux kernel, so KubeSphere is similar to the Ubuntu, a user desktop for the operating system but KubeSphere is designed for the distributed operating system, you know, the data center. So we can use KubeSphere to deploy and manage your cloud native applications and to build the cloud native observability on Kubernetes. So KubeSphere locking system integrate FluentBit operator to deploy and manage the life cycle of FluentBit. So as you can see the general architecture of FluentBit, the locking system in KubeSphere. So we have the FluentBit operator which is used to set up the FluentBit demo set on each Kubernetes node. So you can also leverage the output plugins to send the data and metrics to different destinations, you know, the Kafka elastic search or even FluentD. Okay, so we run the FluentBit on each node instead of FluentD. We provide a centralized log query which is used to search and to retrieve the users and the infrastructure's logs in a unified platform. We build the Kubernetes native way, you can search the logs by namespaces, workloads and pods. So if you want to leverage this platform in a multi-tenant environment, so imagine you have different tenants in this platform. So you will require the users can only view their logs under their own users namespace or workspace. So, and next let's jump into the demo of KubeSphere locking system. All right, as we mentioned before, KubeSphere locking system integrates the FluentBit operator as the underlying logging solution. So we are going to showcase a live demo to deploy the FluentBit operator into Kubernetes use KubeSphere console. Okay, let's jump into the live demo. So as we can see that we have a sample YAML which is used to deploy the FluentBit system to deploy the FluentBit with dummy as input and the standout as output. So if we successfully set up this demo, you will observe the message from the FluentBit port. Yeah, you will get the logs like these lines. Okay, we can have a quick look at this quick start YAML. So we have defined the tag with the dummy in the input and output section. So we just need to copy it and paste it to our WebCube cut-off. Yeah, as we have already created this YAML file locally, so we just need to apply it. Okay, it may takes a couple of seconds to create these resources. So I will cut the waiting time until the resources being created. This is very strange that it takes around more than 10 seconds to create those resources. Since we have already set up the FluentBit and FluentBit operator in my environment, yeah, we can get logging system. Okay, so now we have successfully created the demo manifest in Kubernetes. So we can retrieve the logs from the FluentBit port. So we can use the centralized logging console to retrieve the logs from the FluentBit port. So as we can see that we can type the project name, yeah, logging system. Yeah, for now we have seen the output from this FluentBit port, yeah. Let's drill into it and to find the standout container logs from this panel. So for now we have seen the same result with this quick start. Yeah, that means the FluentBit operator works properly and we have seen the dummy logs was output. So if we want to get more metrics and events from this container, we can drill into its detailed page to see what happens into this container, yeah. Okay, this is all for my demo. So we've added some more resources for you to check out if you want to dive deep into the world of operators and I hope you find them useful. Thank you. Okay, thanks everyone. If you want to learn more and ask anything about FluentBit and FluentBit operator, find us on Twitter. So we have personal Twitter accounts. So feel free to ask us anything. Okay, thanks for your time. Bye bye. Bye bye.