 Hi again everyone. I'm Constance Caramelis and today I will be talking to you about the open telemetry collector and how it empowers end users. Now you're probably wondering what is open telemetry? It is a merger of open sensors and open tracing from early 2019. These were two projects, successful projects that were working on very similar goals and the decision was done to merge these into one project to unify resources. It is a collection of tools, APIs and SDKs. And yes, that actually means that there are support in several languages, actual libraries that you can use and adopt in your applications today. For more details in terms of the instrumentation side, today we'll be focusing on tooling. I highly suggest that you watch last year's keynote with Liz Fung Jones and Sarah Novotny by introducing open telemetry and also some Kupkan EU talks. Now, why open telemetry now? The moment of observability is here. As we build more complex systems, we need to better understand our systems. And so it is the time for a project that focuses on merging multiple telemetry formats, metrics, traces, eventually logs. And it is all about prioritizing end users. And as you'll see in our landing page, we talk about being vendor agnostic because unfortunately when it comes to telemetry and the resulting back ends, vendor lock-in has happened. It depends where you are in your cloud native journey. If you are lucky or an early adopter of cloud native, vendor lock-in might not be a strong story for you. But I can guarantee that for many of you watching today, vendor lock-in is a pain. And this actually hinders your ability to choose the proper back ends for you because vendors will provide better tooling and better instrumentation that works better with their back end. But this isn't the way of the future. That is not the point of cloud native. And so open telemetry is strongly addressing that by making sure to provide vendor agnostic instrumentation and tooling. Now, you're probably thinking, great, another migration. And I don't blame you. It is annoying to think about another migration. You're probably thinking about, do I job service mesh? Do I Kubernetes? What about containers? Do I security? So many things working there. And so I deserve this face. Now, I am here to tell you that in terms of adopting and migrating open telemetry, it isn't an all or nothing. There is a way for you to start adopting open telemetry and specifically the collector without changing much of your existing infrastructure. And so the path forward is the open telemetry collector. Now, you're probably wondering, what is it? It's an executable that receives telemetry data, transforms it, and sends the data along. It works on metrics and traces. It supports several popular open source protocols. I want to call this out because open telemetry does have its own protocol for metrics and traces, but we do acknowledge that there have been other protocols for metrics and traces before. And so to successfully adopt open telemetry doesn't mean that you have to use our protocol right away. You can still use your existing protocols, as I'll show in a little bit. And it's a pluggable architecture. If you look at the codebase, it's pretty nice and clean. They're very clear interfaces, so you can implement either new ways to get data in, transform the data, and export it out. There are two ways you can deploy it. One as an agent side cart or as a standalone binary instance. Examples for why you might want to run it as an agent is that if you want to do sampling decisions or if you have a high throughput service and you want to offload the telemetry as soon as possible to some other binary so you don't create a ball and look in the application, that's one way. Or you can just have it as a standalone instance because maybe you're adding some extra information to your metrics and traces afterwards before forward and long to back in. Either one. It is all the same binary. Now the internals of the collector, we've said it before, but there's three parts to it. Getting data in and the formal term is the receivers. And so what receivers do is that once you get something within, once you get telemetry within a certain protocol, it gets translated into internal format. And this internal format, the processors consume this internal format and then transforms the data optionally. Little spoilers in a bit. And then at the end, it forwards the data along to whatever format you want. Now, when you're thinking of migrations, there's a few complex things about it to think about. One is that you can have multiple telemetry workflows. Different teams might have used different protocols. They might be sending things with different back ends. It's a little complicated there. Depending when the telemetry is added, the quality of the telemetry can differ. And so you want to standardize this and then controlling where the data goes. It's great that you generate all this telemetry, but controlling where it goes in the end is so important. Now let's tackle the first thing, managing multiple workflows. This is naturally simplified service application depiction. And so all these services use only one metrics format and send things to Prometheus back end. And now they're actually two different tracing protocols, Zipkin and Yeager in this case. And within these services, they might have these values hard coded. What makes it really hard is say if you're an observability team is how do you manage all of these? If you don't provide a central way for teams to send their traces to some data collection, yes, the collector, it's hard to audit where things are going. So what I'm suggesting to you today is that you can use a collector to maintain all of your workflows. Now there's a few ways you can do it. One is that you can use a collector to create, in this case, since we had three different protocols, one for metrics and two for traces, three different pipelines, one for Prometheus, one for Zipkin, sorry, go back, and one for Yeager. But also if you wanted to say you know what, we're tired of having two different back ends for traces, we're going to want to merge them. What's really cool about the collector, where it was hinted at in the pipeline's diagram, is that since it supports multiple receivers, it can say on one pipeline, you can say, I want to support Zipkin and Yeager. And then it'll convert both of those into the internal data format. And then after once it goes through the rest of the pipeline, export it in whatever format you want it to be. This allows for you to have multiple protocols for telemetry data and send it to one backend, or maybe something even cooler at the end. Now, how does this work out if you have these protocols and a new service comes along, and these service owners are super excited about open telemetry? And they want to use the open telemetry format. Now, kind of like what we did with the Zipkin and Yeager trace pipeline, we can do the exact same thing. We can add the hotel protocol to the receivers for the metric and traces and just send that data along. This means that your service owners don't need to change protocols. So when you can control the data, yes, it does require changing where the data is, you know, exporting to. But it also means too that you don't have to force everyone to upgrade to the latest instrumentation, right, to the latest library or protocol. This gives you power to choose, hey, maybe we only upgrade the latest services, right? Only new services use this latest protocol or, you know, this one service, it needs to be rewritten. Let's do that one there. This gives you power to choose when you adopt a new protocol. Now let's talk about standardizing the data. We have, you know, now we can, we get all our data working through. But now we're seeing that the quality of data isn't so great, right? And a few causes for this different source, different quality of data can be different languages, you know, naming for that. Maybe earlier data doesn't isn't as rich as newer data because we've asked as we play around with systems for longer, we learn to ask better questions. Meaning of all a term that we might have used four or five years ago doesn't make sense anymore, right? Maybe you've added more regions and before there were no regions, so you used default. And accidents happen as best as we try. Sometimes, you know, like PII leaks, right? Something like that. And we need a way to catch that. Yes. As you all know, the collector does all of these things. So it's called the attribute processor. And in this example, I'm going to be operating on traces. And so say I'm missing an attribute, right? I have an earlier version of the application that wasn't setting environment before we just had one environment. Now we actually have, you know, testing, staging production. And so using this processor, I can say, let me add this attribute environment product. And now my data looks similar, you know, from newer stuff. And then I realized looking at this a little bit more. Oh, oops. You know what? I got into a little joke saying that the West Coast is the best coast. And so I set region to best coast. And it didn't make, it wasn't as much of an issue back then because we're only deployed on the West Coast. But now we actually have the East Coast. And yes, the East Coast is really cool too. You know, it can be the best coast. So the bestest coast together. So I want to change the best coast to West Coast. And you know, I'm looking at this data a little bit more. And oops. Sorry, Steven. I leaked your name in logging, you know, in setting these attributes. And so that's not okay. So I'm actually going to delete anything with username for my trace to look like, my span to look like this. Now this is giving you just a few examples of how you can tweak existing data. And this is all done in the collector. I haven't given any YAML files here because I think we're all a little bit tired of YAML at the moment. But this is all done with a few lines of YAML. This doesn't require your application to change anything. This allows, you know, say the central observability team to control the quality of the data without pushing changes onto the application. And the last part, forwarding your data along, right? We're getting, we've got data in multiple receivers. We've enriched it, standardized the data. And now we want to send it on outward. And so right now, in this example here, we only have, we're only sending it to one backend. But what happens maybe if you had a hackathon and two teams came up with two different backends and you want to compare these backends side by side? Well, this is great. Is that with, similar to the receivers, you have multiple receivers, we support multiple exporters. And so with a few lines, once again, a few lines of YAML code that I'm saving your eyes from at the moment, is that you can tell the collector, hey, send everything, you know, send all these traces to backend one, two and three. And this has absolutely no implication on your application. Yes. Implication of your application. Absolutely none. Now, the statement I want to make, I want to make an even stronger statement is that this gives you the power to choose whatever backend you want, right? This is partially targeting the vendor lock-in. Is that if you can control how you collect all your data, you can also then control where your data goes. And this actually holds, I know it's funny coming from me, because I do work for a vendor, but this holds us vendor accountable to providing a great backend for you, right? If you have the power to choose where your data goes, we have a responsibility to you to make sure that you have the best experience with your data. And so that's why I think that this is actually the coolest thing about open telemetry collector is that this puts a power in your hands. So to reiterate some of the things we saw today, from complex operations to easier operations, we can go from, instead of very disjointed ways to managing multiple telemetry workflows, the collector, you can have multiple pipelines and multiple protocol support. So you can do that with one binary, one deployment, few lines of YAML. You can standardize the quality of telemetry using the attributes processor, right? This is once again, no code changes. And you get to control where your telemetry goes. This is leveraging multiple exporters and multiple protocol support. Now, a few other takeaways is that it's not a requirement for you to use the open telemetry protocols to leverage the collector, right? There are a lot of really cool features within the collector that I highly suggest you're looking into if you're trying to collect and manage all of your telemetry. This also provides a way for you to adopt new telemetry protocols and only change what is needed, right? By, say, having one pipeline that accepts multiple protocols, you can then say, I keep everything, you know, anything before a certain date, only using protocol, you know, legacy protocol and everything going forward using open telemetry protocol. This gives you more power to choose. It's easily extendable, right? Anything that is missing, add it, right? If you have a custom protocol, create, you know, it's an easy to add and you receive an exporter to support your custom protocols. And processors do more than modify attributes. They can do sampling, memory limiting, which protects the collector from ooming because you don't want to lose any of your telemetry, batching, retrying, cued, like, standing with Kubernetes metadata, like, standing with Kubernetes metadata, there are quite a few processors that are able to enrich the data and also control how the data flows through. So I'm hoping that you're a little bit less annoyed with the call to migration and that this maybe provides a path forward for you to look into open telemetry, so the collector and also instrumentation without thinking, without it having to be a huge migration and only potentially updating small things that need to be updated. So maybe you're a little happier with this concept. We are working towards GA and you can find the collector repository here, collector and contrib there, and there's a meeting every second Wednesday. Thank you, everyone. Keep cloud native connected everywhere. Bye.