 Hey everyone, welcome to FluentCon and thanks for being here. I am Aaditya, a senior here at CAC undergrad from India and I worked with open telemetry a couple of months ago as part of the LFX mentorship program where I created the FluentBit exporter for open telemetry which is one of the few things that I will be talking about today. So the goal of this presentation is to give you a brief overview of open telemetry and how you can use FluentBit along with open telemetry to export your telemetry data from your application to the backend of your choice. I will also be giving you a brief demo of an example application that I had worked on for an experimental use case that allowed users to use FluentBit to export not only logs but also trace data from the application to the open telemetry collector. So before we start talking about open telemetry itself I think it is important to understand what observability is and what are the few pain points of open telemetry is trying to address. So observability is basically a property of your system or application that allows you to determine what's going on in your system and you do this by collecting data from the system and these data are mainly of three types, logs, metrics and traces. Logs are timestamped records that denote the events that occur in your system. Traces are... Traces basically trace the request flow through your system and metrics are numerical representation of the help of your system. So once you get all these data, you need to correlate them so that you can get rich insights into your session. We need to do all this because systems now are more complex than they used to be because of the distributed nature of the service that we design and with the adoption of micro-services it has become difficult to find out what's wrong when something breaks. So if we implement observability correctly it becomes very easy to find out what's wrong and fix it. So open telemetry itself is a mojo between two projects called open-senses and open-tracing. Open-senses and open-tracing both were projects that gave a standard to allow you to integrate distributed tracing into your project. Open-tracing just provided to you with a tracing API while open-senses had a standard for both traces and metrics. So the two projects understood that observability is a key aspect in modern software systems and decided to merge to create a single set of APIs, SDKs, toolings and integrations so that users would not have to choose between the two projects and could easily create and manage their telemetry data. Open telemetry is also vendor-neutral and is currently, if I'm not wrong, the second most contributed to projects behind Kubernetes itself and it has contributors from over 40 different companies and this ensures that no company has a say in the direction which the project takes. So let's take a look at the components of open telemetry. So the main components are the specification, the open telemetry collector and the instrumentation libraries. Open telemetry itself has a really probable architecture so you can create your own components if you really need to and you can add extensions or modify the existing ones if you need to. So the API allows you to instrument your applications, libraries and frameworks and the SDK implements those APIs semantic conventions are directly naming conventions for traces, attributes, etc. to ensure consistency across the various language implementations. The open telemetry collector is a component that receives telemetry data, processes it and exports it to the back end of your choice. So the request data flow usually looks something like this. So we have an application in which we use the open telemetry API to instrument our application which is implemented by the SDK and the instrumentation that is generated is exported using the exporters. Now the exported telemetry is sent to an open telemetry collector where we usually collect the data from various applications and we have it in a single place. The collector enriches this data and passes it through various processors and then sends it to the exporter to send it to various back ends. So let's have a look at the open telemetry collector. The collector is by far one of the most important components of open telemetry because it allows you to receive, process and export telemetry data. So it does this by using config files to define the behavior and a major advantage of this is that it allows you to decouple sampling, QN and retry logic from your application code itself and by providing a simple config file you won't have to change your application code every time when you need to make some changes in the collector. So the open telemetry collector has units called pipelines that are mainly of three types based on the data that they deal with. So these are traces, metrics and logs. So the basic logic of the collector is it will take telemetry data from the various receivers. It will pass them to the processors and the process after it has been processed it will send it to the exporter that will export it to various back ends. So the cool thing about this is that open telemetry converts all the data to an internal data format so that you can convert it into whatever back end you want. So let's say I have traces coming in from the OTLP protocol I will be able to convert this into an internal format and then I can convert it into the format that the EGAR requires or the format that SIPCAN requires. So by doing this I can not only send the telemetry data to one back end but as many back ends that I want to. So along with DC components the collector also has additional extensions that are not part of the core collector but provide additional functionality for example the health extension which responds to health messages. We can also have a look at the config file so we first define all the components that we have so we have various receivers, exporters and processors also extensions and then we define the pipelines that we want so here we can see that for the traces pipeline I am receiving the telemetry data using the OTLP protocol I am passing it to the batch processor and from the batch processor I am sending it to the login and EGAR exporter so the login exporter will log the data to the console while the EGAR exporter will export it to the EGAR back end Similarly for Metrics we do the same thing but instead of sending it to EGAR because EGAR does not support Metrics we use Prometheus so the Metrics are sent to our Prometheus back end so you might be wondering what Fluentbit has to do with all this and the approach that Open Telemetry has taken with Flux has done that it has taken with traces and Metrics so for traces and Metrics Open Telemetry has defined a new API and language implementation for both of those but for logs it makes use of third party agents to collect and process logs and Open Telemetry only focuses on addressing the data so if you are already using a different log model what Open Telemetry does is it takes all the log data and converts it into its recommended format based on the log data model that has been defined in the specifications report and from there we can export it to any back end that we want to so usually what we do is we use Open Telemetry, ABI for Metrics and Traces and use a third party agent like Fluentbit to collect the logs and send it to the collector so now I want to talk about how you deploy an Open Telemetry collector and how you can use Fluentbit to export traces as well as logs so we can see here that we generally have an application that has an Open Telemetry library embedded in it which sends the instrumentation data to an Open Telemetry collector that is deployed as an agent from where it is sent to the back ends of your storage or an Open Telemetry collector so the logic behind this is that Open Telemetry collector can be deployed as an agent or a standalone server so typically when you deploy it as an agent you deploy it as close to your application so that like I said before you don't have to worry about things like retrieve logic or queuing or sampling logic and you can just add in the config file if you wanted to chain something and while you in case you want to deploy it as a service you can also do that so that this is usually deployed on a different machine that does not have resource constraints so it can collect data from multiple hosts and you can have all your telemetry data in a single place so now that we understand this we can imagine a scenario in which we are not generating a lot of trace data and we already have deployed a fluent bit order fluent d agent along with the application because you always need to deploy it to collect logs since Open Telemetry does not do that so in this case if you don't want to deploy the Open Telemetry collector what you can do is you can use a fluent bit exporter which you can install directly inside your application that will send data to the Open Telemetry collector using the fluent bit service so thereby reducing eliminating the need for deploying an additional collector agent so that's the application that I had that is the application that I had worked on and we do this by making use of the fluent forward plugins input and output plugins which follow the fluent d forward protocol and this protocol defines the format in which you send data to it and if you send data as a message pack array in this specific format it will recognize it and pass it and send it to the output plugins that you have used so what we do is we are using fluent bit or fluent bit to collect logs as well as traces so what we will do is we create a... we reserve a special tag name called data-was-pan to denote that ok this is a span so pass it accordingly and when we do this we effectively can tell the Open Telemetry collector that logs need to be routed to the logs pipeline and the spans need to be routed to the traces pipeline so let's take a look at how the data flow looks like so we have an application and we have installed a fluent forward exporter inside this fluent forward exporter will send the telemetry data that is traces to the fluent bit engines that can also be reading logs from any file or any input plugin and this fluent bit engines will forward the data that it receives to the collector on the collector side we have the fluent forward receiver running which sees the tag of the span and if it sees that the tag is data.span it will tell the collector that ok this is a span and it passes it on to the traces pipeline and if it is not then it routes it to a logs pipeline so once you do this the processes that you have defined in the collector config will take less in that order and it will be sent to the exporter where it can be exported to labor or whatever exporter that you are configured with this sort of query I think we can take a look at a demo application so let's first take a look at the fluent config file so this input plugin basically reads the log files and this is the forward plugin that I was talking about which listens on this port for trace data so our exporter that we had configured in our application will send the traces to this port and we can also see that there is an output plugin that logs the data to the console and one that forwards uses the forward plugin to send the data to the open telemetry collector where the fluent forward receiver is listening for data using the fluent forward protocol so now we can look at the config for the open telemetry collector and here we have defined the receivers the processors and the exporters and for the pipelines we say that both the traces and the logs pipelines will make you the same receiver and pass the data to the batch processor from which it will be exported to the back end or log to the console if it is a log so with this done we can start the collector and take a look at the application that we have written so this application basically first initializes a trace provider a trace provider is basically a component that gives you access to a tracer and tracers are responsible for creating spans so we need to initialize a trace provider first which is globally set and once we do that we can access tracers based from that so we create a tracer and using that tracer we create a span and add an event to it and let's say we had a service that this particular service was calling so we simulate that using a sub operation which is just a function called so inside that function we again create a tracer and using that tracer we start a span which will be a soft span of the original parent so we can start the flinted service and we can already see that the flint bit instance has read the log file and forwarded the data to the collector which we can verify in the collector which shows that okay I got a log message from some application now if I run our example application sorry for that so we can see that the flint bit receives a message pack array and has the tag data dot span so it will signify to the flint forward receiver on the collector side that okay this is a span and we can verify that in the collector logs and it shows that okay I am receiving two spans here so we can also open up a lever in our local host find traces and like I said we can find the span here so this is the message that we had added on the span as an event and this is a soft span where we also added an event so that's the demo for me and I believe that's the end of my time you can also scan this QR code to see the slides or where you are looking so thanks for your time and that will be it from my side