 Alright. Hello everyone. I am Gautam. I'm a senior software engineer at Grafana Labs and I've been a Prometheus maintainer for more than five years now. And I've done a lot of Prometheus, a lot of hosted Prometheus with Cortex and for the past year or so I've been looking into OpenTelemetry and contributing there. Yeah, today I'm going to talk about leveraging OpenTelemetry for your Prometheus pipelines. And funnily enough, kind of bits and pieces of my talk have already been covered in previous talks. But there's still content, I promise. So yeah, we're going to talk a bunch about the collector, about the receivers and exporters. And specifically I will talk about some of the use cases you can use OpenTelemetry and Prometheus together. Cool. First, let's talk about Prometheus. So Prometheus is a monitoring and alerting system. It's matrix based and this is typically the architecture for Prometheus. So you have your applications and the expose matrix on a HTTP endpoint that Prometheus creates. So it's a pull-based system. And then you can point your Grafana to your Prometheus and Prometheus also runs alerting and sends it off to an alert manager. And recently, and it's becoming more and more common, you don't want your data just in Prometheus. You kind of are also writing it to a long term store somewhere. It could be your own on-prem. It could be a cloud solution with remote write. So that's about Prometheus. Let's talk about OpenTelemetry. So there's a lot of text on this slide, but the thing that resonates with a lot of people, including me, is one of the main compelling reasons why OpenTelemetry exists. Let's say you're an enterprise and you decided to invest in one vendor. You did an evaluation. You said, I want observability. Let me pick this vendor. And then you tell your developers, hey, here's the SDK for this vendor if you want observability. Instrument your applications with this. And then two years later, this vendor gets acquired and the prices go up. Or you need to move. You found a really nice open source solution and you want to move to this open source solution. Now, if you go to your developers and tell them, hey, now you have to instrument, re-instrument your thousands of lines of code with the new SDKs, that's just not going to happen. It's going to take a year at least to migrate from vendor one to vendor two. And this migration is the reason why in many enterprises, there's like six different observability vendors. And this is lock-in. And that's kind of where OpenTelemetry steps in. So Open Telemetry is a set of standards, SDKs, and middleware. So the idea is you instrument your applications with OpenTelemetry once. You send OpenTelemetry data to the OpenTelemetry collector. And from there on, you can basically send to whatever vendor you want, which means if you're an enterprise, if you're a platform team and you're evaluating a new vendor, you don't need to change anything in the applications. You don't even need to tell the developers anything. You can just say, okay, I'm going to change the collector configuration, which I control. I'm going to send it to vendor one and vendor two, and I'm going to compare performance. Does it really work? Are we getting the same value out of it? So that's the kind of key value of OpenTelemetry, I think. And today I'm going to focus on the middleware side of things, which basically contains the OpenTelemetry collector. And as you can see on the slide, it's a conduit that allows collecting sources to destinations. What does that mean? So zooming into that. So that's the OpenTelemetry collector. It's a big blue box. But inside it, so it has, again, in the light and talk, we learned that it has receivers, processors, and exporters. The receivers are used to collect and receive data. In this case, OpenTelemetry supports a wide range of receivers, not just for metrics, but also for traces and logs. And I'm just focusing on the metric side of things here. And here it can ingest OTLP format, which is the OpenTelemetry format. It can ingest stats, the CloudWatch. It can also scrape metrics from Prometheus. And then you have the processors where you can modify this data, add extra labels, or do some aggregations, like run recording rules, kind of, like reduce the number of labels, reduce the number of cardinality. And once you're happy with modifying your data, you can basically send it off to a bunch of different sources, like, you know, you can send it off to all the vendors, like all the major vendors have integrations with the OpenTelemetry collector. You can also send to Prometheus via two different methods. And in this talk, I will be focusing on the Prometheus components, on the Prometheus receivers and the Prometheus exporters. So the receivers. So we have the Prometheus receiver, and then we have the simple Prometheus receiver. And then we also have the Prometheus exec receiver, which is currently deprecated. So let's look at the Prometheus receiver. So this is different ways you can ingest data from Prometheus. So the Prometheus receiver is basically just a wrapper over the Prometheus config. So if you have a Prometheus that you're using to collect data, you can just copy that YAML out and paste it into this Prometheus receiver. All the parts, remote right, everything you can just like copy and paste it into this, we just support the scrape config and the global labels. So if you want to migrate from an existing Prometheus installation and use the collector, use the Prometheus receiver. Also, if you are running a serious Prometheus workload and you want to monitor it, I would also suggest using the Prometheus receiver. It works really, really well. It is a wrapper over the same Prometheus scraping code. So it's exactly the same data that you can kind of scrape with Prometheus. And it works quite well. There were a bunch of compatibility issues, but slowly, every one of them is getting fixed. You know, our friends at eBay have also kind of pointed some out some of those out. And they're all more or less fixed right now. And then we have the simple Prometheus receiver. Why does this exist? So if you look at this, there's this Kubernetes Service Discovery configs, and they look pretty scary. Now, if you have a single application that you want to monitor, and you want to say, hey, here's a Prometheus, here's an endpoint that exposes Prometheus, I want metrics from it, you don't want to write a lot of complicated code, you want to have something very simple. And that's where the simple Prometheus receiver comes in. So you can basically say, hey, use a simple Prometheus receiver, here's the endpoint, and here's the collection interval. And there's a few other features you can add, but that's basically it. Now, on the flip side, you cannot do services covering with this, or it doesn't make sense to do services covering with this. So for serious Prometheus use cases, I would still use the Prometheus receiver. But when you have a single one off system that you want to monitor, I would use a simple Prometheus receiver. In the end, no matter what you use, the simple Prometheus receiver converts it, it's a wrapper over the normal Prometheus receiver, so you get the same data, so that's fine. And then you have the Prometheus receiver. So the reason this exists is because for Prometheus, for you to collect Prometheus data, you need to scrape a dash matrix, a HTTP endpoint to get the matrix. And not all services expose this data. For example, MySQL doesn't have a slash matrix endpoint. So you can, you have to run a different service that basically talks to the MySQL or MySQL API, converts that matrix into Prometheus format and gives it for you to scrape. Now, this means if you want to monitor MySQL, you have to run the collector, the MySQL exporter, both. And this is a simple wrapper that kind of lets you run just one service and say, hey, run a sub process. And then I'll scrape and scrape this up process for the data. There are some security implications for this, which is why it is deprecated. I would not recommend this. I would run this separate process normally. Cool. So that's the Prometheus exec receiver. And so I mean, the processes that convert from different matrix format to the Prometheus matrix format, they're called exporters. And here you can see there's the Apache exporter, Postgres exporter, MySQL exporter. Funnily enough, I'm going to talk about exporters, but nothing to do with Prometheus exporters. I'm going to talk about the exporters in the Prometheus collector. Even more confusingly, the first exporter I'm going to talk about is the Prometheus exporter. So yeah, so now you have your collector, you're ingesting a lot of data and you want to kind of use this and send it to a Prometheus or Prometheus compatible system. So there's a multiple ways to do this. One of them is the Prometheus exporter. And the other one is the Prometheus remote right exporter. There's also the Google managed Prometheus remote right exporter, which you can basically use to send data to Google managed Prometheus. There used to be when I proposed this talk, the AWS Prometheus remote right exporter, which needed a special way of doing authentication to send to AWS, but we now support it in Prometheus core. So that has been deprecated and even removed. Funnily enough, I spent 30 minutes trying to find the commit that removed it. I could not. My good skills were not good enough. Yeah. So let's talk about the Prometheus exporter. So the Prometheus exporter basically takes all your metrics and exposes them on a slash metrics endpoint. Now you can point your Prometheus at the open telemetry collector and say, Hey, Prometheus, go and scrape the open telemetry collector. It's very, it's very simple. Like it just takes all the metrics, converts them into the Prometheus formats and put some on metrics. It has a couple of interesting things. One is send timestamps and metric expiration. So in Prometheus, we kind of expect the metric, like even though the metric has not been updated for a long time, we expect it to be scraped. Even if you didn't update it, we still want to scrape it. We just want to show the same graph. But typically when you do that forever, you're just metrics, the list of metrics that you have grows. So you have to control it with metric expiration. In this config, it says, you know, after three hours, if you don't see any updates to this metric, remove this. The other thing is send timestamps true. So when Prometheus scrapes your application, it says, okay, the value at this timestamp that I'm scraping, time dot now, is the value that you see here. Five minutes later, Prometheus scrapes it, it's like, okay. At the timestamp five minutes later, it's also, the value is the same. But when you're collecting data, the data has a timestamp when it comes in. And when you're exposing it, you kind of, you have to decide, should I use the original timestamp or should I just let Prometheus assign time dot now. So for some use cases, it makes sense to send the timestamp. For some use cases, it's fine to not send the timestamps. So that's one option. Having said that, I would not recommend this exporter for serious users. Because the moment you're talking about any decent set of scale, all your metrics are controlled by one scrape. So when Prometheus scrapes it, it basically scrapes hundreds of thousands of metrics. And one of the reasons Prometheus can scale to a lot of samples per second is because there's some form of load balancing happening in the form of thousands of scrapes, not just one scrape sending all the data. So if you have a scrape interval of one minute, you basically, and you have like a total of hundred thousand series, you can say okay per second, I'm going to ingest on average hundred thousand divided by 60. But if it's all the hundred thousand are exposed on one endpoint, there's no load balancing, there's no spread of load there. So that's one issue. The other issue is, so in the Prometheus format, whenever you expose, you expose the metric. First you expose the metric type and the metric help text and then all the data about that metric. Now you can have the same metric with different help text across different teams. And the moment you have something like that, this exporter just breaks. It just doesn't work. That's another thing, another reason I don't want to encourage the use of this exporter. Having said that, if you have a small amount of data, it's the easiest thing you can do. The other is the Prometheus remote write exporter where you basically give the remote write endpoint and it just writes all of that data. Sorry about that. My throat is going dry. I don't know why. Yeah, so you just give it an end point and it just automatically remote writes all of that data. It scales, it works really well. I would recommend using it even against the Prometheus. So if you have your applications and if you have Prometheus and you still want to use the collector in the middle, I would still recommend remote writing to Prometheus rather than using the slash matrix endpoint. So yeah, that's the different receivers and exporters for Prometheus. There's some gotchas when converting from OTLP to Prometheus. One of them is that dots are not supported in Prometheus. We just don't support dots. But OTLP supports dots, which means when you have dots, they're converted to underscores. Which means if you convert from OTLP to Prometheus to OTLP, your data is different. That's kind of okay. But the other bigger issue is if you have OTLP and you're ingesting logs, metrics and traces, your metrics labels will be different from your logs and traces labels just because the dot is not supported. So jumping from one signal to another with Prometheus is going to be a little more difficult. So my recommendation, don't use dots. Dot suck. No, I'm kidding. So we're trying to, I mean, at least it's in the, it's part of the next dev summit to allow dots in Prometheus. Let's see if the Prometheus community accepts it. I will be trying to push for it. That's one gotcha. The other gotcha is that resource labels are sent to a single metric. For example, in Open Telemetry, you can add several resource labels. For example, the pod, the region, the namespace. All of these are resource labels. Oh, wow. Thank you. That's helpful. Ideally, people expect all of these resource labels to end up in all the metrics, but that's not the case. So the service namespace and service name label end up in the job resource, in the job label for Prometheus metrics, and service.instance.id ends up in the instance label. But every other resource label ends up in a special metric called target info. So now, if you want to use resource labels in your queries, you need to do a really ugly join in promcule to kind of add this resource label to the metrics that are, that have the same job and instance labels. It's not pretty, but it can be done. And typically that's, this is something that also confuses people. And the other thing is exponential histograms don't work yet. So Prometheus, I mean, open telemetry has parsed histograms and Prometheus has, or will have exponential histograms. They are fully compatible now, but while open telemetry histograms exist, Prometheus histograms have not yet merged. Having said that, a pull request was opened 12 days ago, and it should be, sorry, it should be merged in a day or two, hopefully sometime in the next five days. And once that is done, I will work on adding basically sparse histogram to exponential histogram conversion. So even that will work, but that's something that doesn't work yet. So if you are looking to leverage that in open telemetry, just a gotcha that it might not work as well in Prometheus. Cool. Going to some of the use cases or how you can basically add the collector to the Prometheus pipeline. So one thing is if you have a Prometheus pipeline that looks like this and you're like, I love open telemetry, I want to adopt the SDKs, I don't want to have a vendor lock-in, you can do that. You can add open telemetry SDKs to your application and every single open telemetry SDK already supports a slash matrix endpoint. You don't need to change any of your pipeline, you can happily modify your applications to use open telemetry SDK. That works. The other common way, I mean this is also getting more and more common where you have Prometheus to just access the collection agent. It's doing all the remote writing to a long-term store and everyone queries and relies on the long-term store for alerting and you can use the open telemetry collector here. Now why would you do this? The thing is when you centralize on a long-term store, you kind of also want to allow developers to maybe use open telemetry. So instead of just using Prometheus, you can also add more sources and use open telemetry collector to get matrix from all these sources and write to this single Prometheus long-term store. You can't do that with Prometheus. Having said that, I typically give a gotcha with this approach. Here if you look at it, the long-term store does the alerting which means your one collecting data and then sending it over network to a distributed system and then expecting that distributed system to do alerting. There could be network issues which means the data could be delayed which means your alerts might not work and typically it's less robust than asking Prometheus to do the alerting. But it's also very robust and a lot of people are okay with that level of robustness. They don't need to run alerting as close to the applications as possible. So it depends. Like you have to pick what you want. And another cool thing that you can do is basically get all the data from Prometheus and send it to a different vendor. Send it to LightStep or send it to New Relic or like send it to a different vendor and you can do this without any changes to the application. And that is something you can do with Open Elementary Collector. And again you can also add more and more data sources not just Prometheus into your pipeline. You can experiment, you can see and this kind of liberates you to do some experimentation and basically lets you avoid locking. And I think that's really really powerful. Yeah, this is something I want to mention. If you in the end, if you do want to insert the Open Elementary Collector in between your applications and Prometheus, use remote write, do not use metrics for all the reasons that I mentioned. This is very important. Yeah, that's all the content I have any questions. Okay, I see questions there. Maybe thank you, applause first. Great dog. Okay, who wanted to ask question? Is it okay if I ask two? I've got an easy one. Are you planning on writing a remote write receiver anytime? Yes, so in Trafana we have the Open Telemetry Squad we're hiring by the way if you're interested. And that's one of the things that we're exploring next quarter. We do want to have a remote write receiver into the Collector. Also with that how you dealing with the lost help text or metadata? Are you just using naming conventions or are you updating the remote write spec? So as far as I know, I think the remote write today supports metadata. It does. So maybe we'll use that. But like remote write today kind of allows you to also send metadata. Cool. And in a world of Open Telemetry, are you going to keep supporting using remote write or the metrics endpoint in the long term? Or are you going to start telling people to use like OTLP to get data into your back ends, your Prometheus back end? You're federated. Can you repeat that? So up there you're recommending use like remote write to do your exports and potentially like opening a slash metrics endpoint on your hotel collectors. In the hotel world are you planning on deprecating that in favor of just doing OTLP direct as the first class citizen for Prometheus? So the question is, would you in the future deprecate remote write and ask people to use OTLP? If the endpoint supports OTLP, yes. But I've already suggested adding OTLP into Prometheus itself. And it was not accepted. I will try again. Maybe in the future if it's accepted and tested at scale. Yeah, why not? That would be awesome. Great. Any other question? Okay, we have two. We have a lot of time. So if you have questions, prepare them. When you deploy the open telemetry collector, do you do it as an agent or as a gateway or both? That is a good question. And I mean, you can do it as a demon set or you can do it as a stateful set like we saw in the previous talk. In this case, I think the current best way to do it is you can do either actually. So you can do it like as mentioned in the previous talk, you can configure the collector to only scrape the endpoints on that node and use it as a demon set or you can configure the collector as a stateful set and use hash mode. And if eBay open sources, there's fancy scheduling, you can use that scheduling. Yeah. So one thing you mentioned was that resources were added with label as sort of a separate metric. And there's inspired to enable resources as regular labels in all metrics. And the thoughts around that sort of exploding the number of labels you have, even though they would be sort of the same continuous alright, so they would probably be rather compacted. But any potential issues with Prometheus, if you do that? So essentially the resource labels could be a lot of different labels and adding to adding them to every single metric doesn't make sense. However, you can use what if you have a resource label that you really want to be part of the metric, you can use the processor to add that resource label to that metric or to all the metrics. But typically because of the wide varied nature of resource labels, I think a target info label is it makes sense as a trade-off, but you can always use the processor to add it to the actual metrics. Okay. Any other question? Going once, twice. Okay, thank you very much, Gautam.