 Welcome back from lunch, everyone. I'm Mike Kelly, and we're gonna kick things off talking about managing open telemetry with Op-Amp. And that's a new open telemetry effort around managing agents and defining the protocol for managing those agents. So first little about me, I'm Mike, as I mentioned, I'm the CEO of ObserveIQ. Been working on observability for the past 10 years, primarily on the agents, the integrations, and the pipelines. So everything about telemetry and getting data in. And at ObserveIQ, we've been entirely focused on open source telemetry for the past couple of years. Started with developing an open source log agent, stanza that was eventually contributed to open telemetry. And then more recently released a bind plane OPL, talk about it in a second, which is an open source observability pipeline agent management system. And it uses open telemetry at really at the core. But let me jump in and talk about Op-Amp. And I'm gonna apologize to all the electrical engineers in the room, because Op-Amp is an overloaded term. This is not operational amplifiers. This is open agent management protocol. And it is a network protocol for remote management of large fleets of data collection agents. And it's worth noting, I think that it starts with large fleets, like that was a key consideration of this from the very beginning. It's a protocol definition. It's part of the open telemetry project and currently consists of that protocol implementation and go and then applications that are utilizing this. And one thing I'll note here is, just because it's part of open telemetry, this really isn't, and I'll show an example of it managing open telemetry collectors. It's intended as a general solution, a generic solution. So the idea is that this should be used for any agent, proprietary, open source. The idea was to get folks that have done this before come together, like we do with a lot of things in open telemetry, and define how we want to manage agents and what that protocol should be. So what does it include? The short answer is everything that you need to manage the life cycle of agents. So that includes things like definitions for remote configuration of agents, obviously things like status reporting, agents owned telemetry, so what's going on in the internals of that agent. Agent upgrade, so allows for packages to be deployed. And then security considerations like connection credential management and secure auto update capabilities. And again, that was part of the focus of, making sure that this is something that can be used by the largest organizations. And I'm gonna go through a little bit about the communication model and a little bit more about the protocol, but a lot of the hard work goes into defining the protocol, but it can be more interesting seeing it in an actual application demo, so I'll do a short demo as well. So when it comes to the communication model, there's a management layer, and this is where most of the definition comes in. There's an op amp client. This can be embedded in an agent, I'll talk about other ways that it can operate, and an op amp server. Communication is typically via WebSocket, but also supports HTTP, and it's fully asynchronous if you're using WebSockets. All the payloads are binary serialized, protobuf messages, and then the, as you might imagine, the op amp protocol is well defined for all the scenarios I just showed. But then it layers on an additional component, which is owned telemetry for the agent. So as you might imagine, folks that are involved in open telemetry, resistant to reinventing OTLP and definitions for telemetry signals being sent to another service. So the way this operates is op amp will configure an OTLP exporter in the agent, and that will be sent then to the OTLP receiver in the agent management service. In the little box in the bottom that I neglected to mention the first time, other clients, that's really everywhere the agent would typically be sending data. So whatever your destination is, this is really just concerned with the configuration in the management of that agent itself. This was one other model that, you know, the specification describes, it doesn't provide full specs for what this would be, but a common scenario is we have an agent that's maybe it's proprietary, maybe it's something that you don't wanna open up the internals and embed a new library in that. And so in this case, we have separate supervisor process that is intended to be an op amp client and also to manage agents, whether they're proprietary, they could be open source, they could be agents that you just don't want to deal with changing the internals of that around. So it allows for some more flexibility. With particularly with some proprietary agents and just gives another model for this to operate. So that was a little bit about the protocol. We just have a couple of minutes. So I wanted to jump into a demo and see this in practice and I'll walk through all the different scenarios we just talked about, but first this is bind plane OP. It's an open source project started fairly recently. It implements an op amp server, right? And it also works with an open telemetry collector, specifically the Observe IQ Distro for Rotel, which includes an embedded op amp client and some other functionality. So as we look at this, you'll see that there is already an embedded op amp client in those collectors. But it takes advantage of the op amp protocol to provide full lifecycle management and configuration of your observability pipelines. Okay, so this is bind plane OP and what you're looking at here is several hundred agents that are under management. So this is a demo environment. It could very easily be several thousand agents, but you can see all of these, some details about them, the status of those agents, the versions of configuration, and even the amount of data flowing through them. It's just a simple example of installing an agent. What this is going to do is it's going to deploy or download and deploy the Observe IQ Distro for open telemetry. It'll give it an endpoint to connect to and as soon as this comes up, it's connecting to that server and pulling down the latest configuration that's available for it and is immediately under management. And so you can see that the demo system is now available. If we go in and get more details about that, you can see the configuration details but also other metadata about that agent. And it's easy to change. So it had a predefined configuration but I also have a demo configuration that we can switch it to easily to show off some of the other capabilities. So error, you know, since you can report status, you can then get in-depth error messages. So if I introduce a simple error to this file, all right, something that happens very often when we're trying to configure complex agents like this, you know, it shows us exactly what's going on, it shows us exactly how to resolve that. That's all standard defined in the op-amp protocol. So I go through and resolve this so we don't have any more red on the screen and then you can see it pushes that new configuration and we get the immediate status response. And part of that is, you know, with WebSockets, it is very fast immediate response communication. But a lot of this, we're talking about scale. So we'll look at this. It's a configuration that was predefined and we wanna apply this to a large fleet of agents. So if I go and look at all my production instances, I'm gonna select all of those again. It's 130, it could be 1,000 or 2,000. As soon as I select those, it already has connections and it can deploy that at scale very quickly but then they're also managed so we know what's going on with those individual agents. And one of the things, you know, I think you can pull it all together in a second here when we talk about what you can do with this, right? So now we see that there's, you know, configuration that's been applied across a wide range of systems. We know that now since they're all managed, we can go in and upgrade those as well and upgrade them individually and this is using the protocol to push down packages, deploy them and we get the status as that agent is being updated. And so we go through a cycle of disconnecting, reconnecting and then validating the configuration is available. But one of the key pieces here, you know, and miss one spot. So Agent Update also supports, as you might imagine, doing this in mass, doing it across all the agents or also automatically updating agents. But then there's own telemetry piece which is, you know, we have visibility into the agents and what's happening at every stage along the pipeline. And so what you can do with that is, you know, we have sources here, this defines everything that's happening in the configuration file, all the data that's flowing from the source through the processors to the destination. And so when you have that level of detail, you can make changes in real time that are gonna impact the flow of data through your system and give you a lot of control over that data. So for example, here we saw, there are a lot of logs coming through, we can add in a simple severity filter to filter out anything that's beneath the warning, apply that and get real time feedback on the data reduction that we're seeing through the pipeline. And you can do that at every stage, you can do it on an individual basis, you can do it across hundreds or thousands of agents and see the changes in real time. So I think I'm at time. But yeah, there's a lot of power in this. I wanted to share a couple of resources that are available here. It's in the slide deck online. But any questions, I'd love to chat with you about anything observability. And thanks and have a great afternoon.