 So I think it's about time to start. Hello, everyone. Thank you for joining me today. I know this is the last day of the conference, and it's nearly lunchtime. But I will try to make this half-hour well worth for you. So my name is Sjandar Guba. I'm co-founder and developer at Banzai Cloud, acquired by Cisco. I'm working for more than five years with Kubernetes and the Fluent Ecosystem. I'm a huge observability fund. And the title is Show Me Your Label. I will tell you who you are, which may sound a catchphrase, but actually I will show you how to exploit labels for your logging subsystem. Let's start with the basics. Kubernetes logging is not that difficult. The container runtime puts the containers STD out into the host file system. And the Kubernetes service access to these logs and provide it to users through the Kubernetes API. It's that simple. It's nothing much you can configure about. What's the problem with this? The problem is that these logs are only stored on the host file system. And eventually, they will get rotated or fill the node file system. And the other part is that CubeCitya logs is not always the best way to consume your logs. So what you can do, you want to collect these logs and transport to your log analyzation system. But there are other problems with this as well. You need privileged access to access the host file system. And there is no separation of logs on the host file system. So all kind of pods logs in different native spaces are just stored on the same folder. But let's see how this system would look like on Kubernetes. And you need a collector to collect the log files from the systems and push them out to an output. Eventually, you will need an aggregator that will collect the logs from these agents and push them forward to another log analyzation system or whatever destination you want. And basically, this was the reason behind Logging Operator that it was born. Because if you are doing such kind of system, you will end up creating tons of configuration files with hundreds of lines of configuration code. And Logging Operator, for this architecture, under whose use is Fluent Bit as the demon set, we collect all the logs from the node level and we transport it to a Fluent D stateful set. The heart of the Logging Operator is a labor router plugin developed at Bonsai Cloud, which helps you to distinguish logs based on Kubernetes metadata. So you can create different log flows. And if you identify the log flow, it's really easy to after just apply any filter, like parser filter or enhancing with GUIP data. And at the end of the log flow, you can post your logs to any kind of output that Fluent D supports. If we take a look from the other direction, how a user would use a Logging Operator. Obviously, as an operator, it will use Kubernetes custom resources. There are two main types of these custom resources, flows and outputs. So let's take a closer look on these resources. Flow can be a flow or a cluster flow. A flow works within a namespace. A cluster flow works across namespaces. You can use selectors to distinguish your log flow. So it works like Kubernetes label selectors, like when you get in Kubernetes pods from your cluster. You can apply filters on those log flows, like nginx parser or Prometheus exporter filter, or any type you want. At the end of the flow, you have to specify outputs. Let's see outputs. Similarly to flow and cluster flow, there is output and cluster output. And the same thing applies to here that outputs works within a namespace and cluster output works across namespaces. In the output configuration, you can use Kubernetes secret references to configure the credentials or sensitive information that you need for this to work. I don't have enough time to dig deeper in the features of Logging Operator. But since the first release of August in 2018, we grew a thriving community. On GitHub, we have about 1,000 GitHub stars. We have more than 1,500 people on our Slack channel. And I'm really proud that Rancher's official backend for logging is actually Logging Operator. There are a lot of great feature that's built on top of Fluendi, like scaling and draining the Fluendi instances, applying configuration checks before the configuration goes to live. And it offers secret and certificate management. But you can read about all the features on the documentation. So as we have a broad experience with Logging Operator now, we should go and try it out. We have a friend, Bob. Bob is an infrastructure engineer, and he has to operate an application on Kubernetes cluster. For the sake of example, this application will produce EngineX access logs. And he has three requirements. One is that he needs to extract metrics from those logs and create an alert if the 500 and 400 error codes reach 40% of the total request. He needs to create a word map and visualize the original IP addresses of the request. And for example, a security team wants to store all the logs in raw format in an object store, for example, in AWS S3. So we know Logging Operator. We can create three different flows for the three different use cases. One to put the logs on S3. One to unreach the log data with GUIP metadata. And one to extract the response codes and create an alert for them. So I will be doing a live demo. So I prepared some software on my cluster. And I hope it will work. So on my cluster, I pre-installed Logging Operator, obviously. I installed Prometheus Operator for metrics. And I installed OpenSearch for the GUIP dashboard. And I installed MComUI, which is a Cisco product on top of Logging Operator. But we won't use any of the extended feature. But it has a really nice web UI to help you follow what happens on the cluster. This is just the default log-in flow. You can agree with it for now. And I have a cheat sheet. So I will be really fast. But please try to hang with me. So I already installed the log generator on my cluster. We can see that if you are tailing the logs from this cluster, they will be standard Nginx access logs in raw format. So the first requirement that we fulfill is to actually install the S3 output. I won't show to you guys because this is my AWS credential. And I don't have that credit card to support you, all of you. But believe me, this is the secrets inside this secret. And after that, I'm applying an S3 output. And this is how simple is to just create an S3 output. You just need to specify the bucket, the region, and the secrets for AWS account. I will apply this to the cluster. And to make this work, I need to create also a cluster flow. I haven't mentioned that I used a cluster output for S3 to be able to use a cluster flow to gather all the logs from the cluster itself. There is a name, the all log. I applied one filter, the record transformer, to remove the annotations because annotations are really verbose on Kubernetes. I have an empty selector, which will select all the pods from the cluster. And I referenced the previously created output, the S3 output. So if I apply this on the UI, we can see that the custom resources I created is visible inside the system. And eventually, after reconciling and fluent the config reload, the logs tries to flow to the S3. For speeding up things, I will cheat a bit and apply the last step of the next example because this way, we will have some outputs already during the presentation. But I will show you how you create a new flow from scratch. So the first I created null output, which is basically, and I applied again, so which is basically a DevNull device inside Fluendy. So I will just drop all the logs that comes into Fluendy. And I need that because I need an output, but I don't need the logs itself. I just want to extract the Prometheus data from it. So first of all, my first step is to create a flow, is to have a name of the flow. I will reference the null output. And I have a selector, which is a label selector. So this label, the app.kubernetes.io per name, equals log generator, is the label on my pod that I want to collect the logs from. So I have the raw logs. What's next? Next, I want to parse those logs. So I know that these are Nginx access logs. So I will apply the Nginx parser on top of it. And I will set the types, like size and code, to numbers, which is really useful if you are ingesting into OpenSearch. And you want to make aggregation on top of these metrics as well. So I have the raw logs. I've parsed them. The next step is to actually apply the Kubernetes filter on top of it. So because I have named fields from the pars, I can create a matrix filter with a type counter. This will count all the logs that's flowing in. And I can apply labels, like the code label, and get the value of the code label from the log itself. So I have matrix HTTP response total with each different code that's are happening inside my Nginx access log. And more over that, I will apply the pod and namespace labels as well, because this way, I can identify which pods actually produce the logs. Because in the Prometheus point of view, it will see only the Fluent D pods. But with this trick, you can actually have a source of the pods. And as I mentioned, I already applied this. So if everything goes according to plan, I will have matrix, yes. I'm getting matrix really fast from Fluent D. And these are the 200s, 400s, 300s, et cetera. And there was one another task that I should create an alert for it. And it's just as easy as creating any alert for Prometheus, because it's actually Prometheus matrix. So it may seem complex, because I have some label magic here, but it's just the simple Prometheus that you would use any time. I apply to this cluster. And I should see in the alerts, yeah, it was already applied. Sorry, I've prepared the demo. And it's appending alert on these logs that I'm outputting now. So I have one thing left. I need to do the GeoIP flow. Actually, I can reuse the flow, because you remember we didn't output the log anywhere. One thing is that you should do before ingesting into Elasticsearch, you want to make sure that the mapping is proper for the data that you want to inject, especially if it's like GeoIP data, because Elasticsearch need to know what kind of fields you are ingesting. So I applied it. And let's see what I configured. The first is really simple. It's just an open search. Elasticsearch open search, I will use the name interchangeably. But in the open search, we have the secret to the credentials. And we have other log-stage format and other configuration. But the interesting stuff is we have updated GeoIP filter to our previous flow. It is here. And it will automatically translate IP data to GeoIP data. And the most important part is the location RI, which are actually coordinate pairs. So we will use this field to plot the origin IP addresses on a word map. And if everything goes according to plan, I will just quickly port forward to inside my open search dashboard. I will just deal the credentials. I will log into open search. And because I have the mapping already in place, I can go to the visualization, create a new visualization in the coordinate map. I have the anginix previously applied index mapping. I can create a bucket with the coordinates. I can use the GeoIP format. And the location are all filled to plot actual request origin IP addresses. And as you can see, it was successful. And we are already counting the request numbers and the origin IPs. So let's get back to the presentation. And with this solution, we actually fulfilled all the requirements. We have the S3 bucket filling the logs. We have the GeoIP filter enabled. And we have the other things in place. And moreover, with the Kubernetes labels attached to logs, metrics, and everything, it's really easy to use an attitude for correlation, metrics, traces, logs, et cetera. And are we happy with that? Bob is not happy. Bob has a question. What if he wants to tail those logs? Of course, he can use the kubectl logs to tail the logs. But there is two problem with this. The flow is a much more complex filtering mechanism than a kubectl log. So it's not just one pod or more pod. It can be spanned across namespaces and a lot of other stuff. And we do a lot of work on our logs, so transformation, enhancing with GeoIP data, parsing. And we won't want to lose this. And I wanted to mention that I want to show that how easy it is to build on top of logging operator for your own use cases. So if you want to do such things like tailing the logs, we need CLI on our computer that actually print the logs on the terminal. We need a service, we will call it Logsocket, that can function as a flu and the output. And we need a way that these two things is communicating. And do we want to add more things to do it? And I would say, why not? Let's make it more interesting and add access control to log flows as well. So imagine a situation where you have a bunch of logs, bunch of namespaces, and you have flows in place because you want to output your logs in your preferred logging system, but you want to give access to different persons to actually tail those logs. But we want to specify who can access to which logs from the log stream. And to do this, there can be several different solutions, but I would say use labels. And a really good example of using labels to actually reach an airbag situation is the Prometheus label proxy, which is doing exactly this, but in front of Prometheus. So we can define a label structure like airbag per namespace underscore service account name, and the way you can be a policy. For the sake of example, we will use Kubernetes service account tokens for authentication because they are there, and it's really easy to use. You just need to send a bearer token with your request. And an example of this role would be airbag per default underscore alias allow, which means if your port has this label, it will be able to tail those traffic. And if it's denied, she's denied to tail this traffic. And we can define a default policy like airbag per policy, and all will deny if you want to have ports that everyone can tail. I know it may sound a bit complex at first, but let's go through some of the architecture. So we have a CLI tool that's connected to a log socket. After that, the log socket authenticates against Kubernetes service account. It uses the token and the Kubernetes API to get a username out of it. And on the CLI, we should have a specify a flow name. And the log socket will create an output. It's a logging operator custom resource. And patch the output reference into the flow that we decided to want to tail. And after a reconciliation cycle, the logging operator will pull the information from the custom resources generating the flu and the configuration. And eventually, the log starts to flow to the CLI tool through the log socket service. So let's see this in action. I need to go back to my cheat sheet. And I will do the same trick that I will apply a bunch of command. And after that, I will start to describing what I've done with those commands. So first, I will need to create the service accounts for the two user, Alice and Bob. And I will use some Helm upgrade command to just prepare the stage. And I will use part of this command to actually start flowing the logs. So this is not going the wrong directory, but that's not a problem. So what have I done here? The first, we created a service account. And now Kubernetes automatically created token for it alert. In 1.24, this won't work. So actually, the latest Kubernetes work a bit differently. We got this token out into an environment variable for later use. And we will do the same for Bob. And what I've done is to update this full service, the log generator, to have these extra labels. Like I described before, I have this Airbus per default underscore Alice a lot. So Alice will allow to tell the full service. And I have bar service, which will tell us that Bob is allowed to tell these logs. And Alice is allowed as well, just we can see it. And the last command is the Kubernetes tier command, which is our command that we just wrote. We specify if we want to tell a flow or a cluster flow. We specify the namespace and the name of the flow. We give the token. And we start to tailing those logs. And as you can see, in my terminal, it's already started to tailing those logs. And compared to what I showed earlier, these are parsed logs, unridged, and in a nice JSON format. On Bob's side, I did some tweaking. And I only print out what logs I have permission. Because if I don't have permission, I have a permission denied error. So I'm getting logs from the bar service, but I have a permission denied on the full service. And if I want to change the outcome of this, I just need to upgrade the service. So I will upgrade the service to deny a list to tell the bar service. And I will use the same syntax like Bob. So what we have to see now is that, actually, each of our users can just access to one service. There is a sum of latency between actually upgrading the pod labels and getting the permissions right. And until that, I just wanted to show that if we are back on our UI, we can see that what we created before is actually in place. So we can see the as-free cluster flow. We have the GUIP flow down here. And you can see that, actually, the Logsocket patched inside to this GUIP flow and have this output working for it. And another benefit of using labels is, actually, you can tell who has permission to access the pod, because they are on the pod labels. So if I just use a simple kubectl command and check what pods at least can access, I just can print it out. Because we are using labels, it's really straightforward to just know what you are doing in your cluster. And, yeah, they upgrade finished. And because the home chart reconciled the deployment on the pod, we will see that both Bob and Alice will have permission tonight for the service that they can access and they can tell us for what they can access. And just to summarize what we achieved today, we created complex logging flows. And we can edit those logging flows dynamically, adding new filters or outputs or anything. And it's done automatically by the logging operator. We can tell flows on Daemon with the benefits of don't have to specify the selectors and have the unreached parse log for us to tell. And on top of that, we added access control on these logs, which is pretty cool if you think about compared to what we have just by using Qubectl logs. And the fun fact that one year before, I had a presentation at FluentCon. And one of the things that I wanted to achieve with Logging Operator is to actually add Arabacruz to different log stream. And one year later, here we are, and I'm talking about this. My last slide is about the useful stuff that helped us to make this demo. The Logging Operator, the Laxacut, the demo will be published on GitHub, so you can try this at home, as well as Laxacut. This is open source. I don't recommend it. It's used in production, but we will get there as well. So thank you for listening me. And if you have questions, there is a mic in the center of the room. And please ask. Yeah, please use the microphone on the middle of the room. I wanted to ask about outputs and IAM roles for service accounts. Can we use those instead of access keys? Can we use instead of? Instead of access keys in outputs, can we use service accounts and IAM roles? Yes. So what kind of the outputs is basically FluentD Outputs. So if the FluentD Outputs supports the IAM roles, which are they supporting, for example, for AWS, SREO, and things like this, yes, absolutely, you can use them. OK, thank you. And also, you talked about outputs and flows, but there is also a third object, which is logging. Yeah, that's true. And we use one logging in our setup currently, but when would you recommend to use more than one? I don't really recommend to use more than one logging. It is possible, but it will duplicate the demo set and duplicate the FluentD as well. If you need hard separation, for example, there was a case when the cluster had hybrid clusters with windows and Linux nodes. So in that way, you want to use two different logging or two different agent configuration. It depends on what you want. And you can use namespace selectors or other tints to not duplicate the demo set, but actually specify which nodes you want which agents. OK, thank you. Welcome. If you don't have any more questions, thank you for listening again. Have a nice keep coming.