 Hello, everyone. We are late by five minutes. I will try to rush it through. So I will be talking about the opam. What is it? Why do we need it? And I will walk you through a small demo, which shows very specific use case based about passing the logs. A little bit about myself. I work at Cygnus. I was an end user of OpenTelemetry about four years ago. And then I became a contributor and then maintainer. Now I'm mainly focused on the opam, on the Python SDK. Outside work, I do cycling and I enjoy board games. Colin on Twitter.com says, using anything else for gathering and shipping the telemetry data after using OpenTelemetry collector just feels wrong. This reflects the growing sentiment among the people who adopt OpenTelemetry collector. And yeah, it's a great testimonial. A big shout out to everybody who's involved. There's some maintenance and approach here. And also everybody who is involved in making this happen. With the growing use of OpenTelemetry collector, as it gets more widely adopted, there's also a need for ability to effectively use it and monitor and being able to control it. This is precisely where opam comes into picture. Let's jump into opam. So if I have to say in a one sentence about opam, it's a vendor-neutral protocol for effectively managing large fleets of agents. There are mainly two things that you need to be familiar with. There's a protocol on specification, and there's a reference implementation. The protocol supports the functionality. The protocol has all things involved that enables one to effectively manage the whole lifecycle of the agent management. It supports the functionality of remote configuration of the agents. The term agent, when I say here, it does not specifically about the OpenTelemetry collector as an agent. It's about anything. It could be your OpenTelemetry SDK. It could be proprietary agent. It could be anything as long as it supports the protocol. And it supports reporting the agent's own telemetry because it's important that you understand it and reporting the status, such as version, the platform that it's running on, et cetera. And coming to the security aspects of it, there's a connection management, credential management, auto upgrades. And most recently, for anything outside of this main lifecycle management, there's a support for custom messages for the ad hoc requirements. You can check out the reference implementation of the client and server. The link here, it also contains the generated protocols. So this diagram shows a very typical architecture of how it would look in practice. So you have this agent, and client is running alongside with the agent. And it works with the server. Client sent, like the agent, it sends the data to OTLP receiver or a number of destinations. The client sends the data, like works with the, sends the data to server on behalf of the agent. So as you see here, there's a communication between the bi-directional communication between the client and server. There are mainly two modes of transportations. One is WebSocket and the other is HTTP. The client sends the agent to server message to the server, and server sends the server to agent message to the client. Let's take a look at what the message contains. If you see in detail, so the client agent to server message has agent description that contains the version of the agents that's running on the platform, et cetera. There's an effective config field which reports what's the current running effective config of the collector, and then remote config status, package status, et cetera. And the server agent contains the, it encapsulates the number of things that server can offer to the client. So it has the, what it can offer, like remote config is there, connection settings is there, things that it can offer to the agent. And there's also a command that it can send to the agent that the agent should perform. Yeah, let's look at the, what can be an open client? There's no specific, there's no explicit way defined what can be an open client. It can be anything. It can be a separate process that's controlling the agent. It can be fully integrated into the agent code itself, or it can be a simple plugin. So the example where it can be a separate process, this is how it looks like, where the client runs as a part of the supervisor process. The agent is managed outside of the, like the client is outside of the agent process and uses the system signals to work with the agent, like restarting it and managing it. This is the where your client is fully embedded into the agent code. So where you see the client is part of the agent code itself. So let's look at the agent as a part of the agent into the server to agent message if you have seen we were offering remote config. So let's specifically look at the part of the message which offers the remote config. Here you can see the server can offer a map of agent configs where it shows you what's the name of the name of it and the body contents of the agent remote configuration. So in conversations with the people who are using the open calametry, the key things that they, they mentioned that they're doing on regular basis and then are like one is passing logs. There are logs that come out of from different systems from some containers. Some are coming from the legacy systems. So to be able to convert them to the OTLP logs is one common thing that they do. And other thing that they're interested in is the sampling rate and the other problem that comes up very often is that hey, there's a lot of data that I'm sending but it's little to noise. How do I drop it? How do I sample it? So I will talk about the passing logs part and I'll show the demo where we see the logs coming from a container and then apply the remote configuration and then get them to a more structured data. So here I, I'm not sure if this is visible but here this shows the logs from a Docker container. If you notice, like if you see there are like number of parts to it, there's a log level, there's a location, there's a message and then there's a, the number of extra attributes associated with the log record. So this is a raw log record. This on its own won't be able to effectively fit into the OTLP model. So we need to perform some sort of parsing to map the log record from the log, log level and message and the attribute like the just instinctive attributes into the attributes. So there are a number of parses in open telemetry that helps with being able to effectively pass this log and convert them into structured OTLP logs. Yeah, let me show a demo, little demo. So I hope this is visible. If you look at this log line, it has the log level, it has the location, it has the message of the body and it has the number of JSON stringified attributes. And this is just a raw log, there's nothing more there. But this is not very helpful on its own. You won't be able to look at, you won't be able to do some aggregates on top of the attributes that you have here or you won't be able to map the exact log level to the severity text of the OTLP log model. So I will create a one dynamic pipeline where we will take this log record, we will do number of step-by-step process and get it to some structure format where this JSON string is extracted to the attributes of the log record. This info is mapped to the severity text of the OTLP log model. We go here. So here I will select the hot rod as the container logs that I'm interested in parsing them. And if you look at this, you can see that there's no attributes here and the severity text is not mapped yet. So let's create this. First thing I will do, I will add a RegEx processor. So we will add RegEx processor. So I have created, I have had a, created a handmade RegEx, so I'll pass it, I'll copy paste it here and I'll create this. Now if we, if I go here and do the simulation of the process. So here what you just saw is what would be the log record look like if it were to go through the RegEx processor. So now if I show what the log record looks like, you can see here this, now there are, you can see here there's a location as a part of the attributes. There's a log level as a part of the attributes and there's a attributes JSON, the stringified JSON of the log record. Next I will add another processor, the JSON processor, JSON parser so that we get the attributes out of the JSON stringified. So now I will add JSON processor, we'll pass it from, and if we look at now, we have two parsers as a part of the pipeline and now if we look at what we will see, we will see more number of attributes as a part of the model. So now you can see that along with the, along with that we now have service, we now have span ID, we now have trace ID as a part of the attributes of the log record. This is, while this is helpful, it would be good to have, good to get this span ID and trace ID mapped to the original log record of the span ID trace ID instead of them being part of the attributes. So I'll go ahead and add another trace parser here. If I do the simulation of the processing again and we wish it should be able to map those, if you see here, the trace ID span ID from the log record are now mapped to the trace ID and the span ID of the original log model, we now have the attributes then part of it. This looks good. We now have, we have, we went through this step by step of getting the raw log model to a shape where the trace and the span ID are now part of the hotel top level log records. I will go ahead and apply this change so that what we saw earlier logs were not, were very useful. Now if I go and apply this configuration, let's see if it's applied or not. Yeah, it's now applied. I will go ahead and look at the explorer again after this change. Let's wait for a moment. Now let's look at the hot rod logs of the container again. Now if we look at it, so this is no, this, the logs are now, no longer are just the simple raw logs. Now they have the, they have the trace ID, trace context passed appropriately. They have the location as a part of the attributes, the method as a part of the attributes, the log level. Now let's go ahead and apply the one more change where we map the log level to a log level to a severity text. Let's go ahead and apply one more process, severity process and it's part of the attributes with the name log level. So I will give it, let's apply it here and simulate it. It should now map the severity text as well. Sorry, let me see again. Okay, I'm not sure what went wrong there, but yeah. So the idea is we can take the configuration, apply the configuration, get it applied dynamically and see the result in action. So this was about the, like throughout this demo, what we have done is we had some unstructured logs, they were not really useful. We applied the, we applied the some modified configuration which had the req spacer, JSON spacer and trace context spacer and now the logs are more helpful than they were before. This is about the demo. Let me go back to the slides again. All right, so while doing this, there were a number of challenges that came up and one of them is misconfiguration. So like whenever you dynamically apply configuration to the collector, what would, it should be that the configuration could contain errors that can lead to telemetry not working properly or it could be that the configuration itself is invalid. So they should be gracefully handled. Other thing is while applying the remote configuration, the service, the open telemetry service must be restarted. So it could be possible that for a number of reasons that there might be a data loss challenge due to the restarts. And the other thing is that there's a movement in terms of how people manage config. So there's more inclination towards using Git as a source of truth and relying the Git as the making the config as a part of the Git ops. So the couple of solutions around the data loss are open telemetry offers a persistent queue where the data gets buffered through the disk. It has file storage extension, which is very popular choice. It helps in these scenarios. It also offers a Kafka receiver and exporter message bus option where one can write it to the Kafka topic and rely on the Kafka semantics so that there's no data loss. So to give us some status on what's happening, there are a number of components in open telemetry related to open. There's an extension. It's almost like fully developed. Not fully developed, it's ready for the alpha stage. There's a supervisor. It's actively being developed and appreciate any contributions from people here. There's also operator bridge, experimental work that one can use. Yeah, thanks. We also have an observatory session on Thursday, 1300. People can come there and give the feedback or ask any questions specifically about the open. Thanks everyone.