 Okay. Hello everyone. My name is Daniel and I'm with my friend, Rhys, and I'm excited to bring this talk, Oh, Tell Me How to Build a Data Pipeline for Observability. Before we get started, I'm going to tell you a little bit about me. I'm a developer relations engineer at New Relic. I'm really passionate about getting developers better visibility into their systems. And yeah, who are you, Rhys? My name is Rhys. I am also a developer relations engineer. I also work in the open telemetry community on the end user working group. So I hope to be talking to most of not all of you at some point about your adoption and implementation experiences. Yeah. So let's just scooter right into our presentation. So when you're starting to develop your applications, you're probably not worried about the amount of observability data that you're generating, right? You start with a couple thousand metrics, a couple hundred traces, a couple thousand logs. It's not a huge deal. But as your application scales, it becomes to become a large problem. And that's because doing anything with data costs a lot of money, whether you're processing it, storing it, moving it. And that could go to a particular vendor to do that for you. Or you could be paying a team of engineers to maintain your Prometheus Grafana staff. Whatever you decide is going to cost you money. So how do we make sure that you get the most bang out of for your buck? So here's a couple of things that we at New Relic think that you can do to make sure that you can increase the ROI of your observability setup. The first thing you should do is have a centralized data pipeline for all of your observability data. And this is because as you have more and more complex cloud environments, you might have multiple sources of observability data. You might have multiple silos that are processing only logs or only traces or only metrics for various parts of the application. And when you have scenarios like that, it's really hard to optimize because you don't have a holistic view of your entire environment and how you're generating observability data and how your engineers and your teams are leveraging that data. So having a single source of truth for all of your data is really important in your journey to optimize and kind of clean up your observability environment. The second thing you do is increase the usefulness of the data. How can we make sure that the data that we collect has all the context as well as the information we need to make sure it's as useful as possible? The third thing you can do is control the amount of data that you're exporting to your back ends because a lot of the data that is generated by our systems, whether it's through logs or metrics, are kind of never used. So how do we make sure that we're really cognizant and mindful of what we export to our back ends? So Open Telemetry has an awesome solution for this and I think our previous lighting talked a little bit about the Open Telemetry Collector. So the Open Telemetry Collector is an open source data pipeline for your observability data. So it's made out of three different parts. The first part is the receiver. So this part of the system kind of collects all of your data, whether it's through Open Telemetry SDKs or frameworks or it's through like other observability sources, whether it's like Prometheus or Fulumbit, and receivers kind of connect all of that data into the pipeline. Then with the processor you're able to take all that data that you collect and decorate it. You're able to batch it. You're able to do a bunch of things to make it more valuable as well as to reduce the overall amount of data that you're trying to export. And finally, with the exporter, you're able to take all the data that you collect and process and send it to whatever back end you want, whether it's a managed back end or a open source back end. So with the Open Telemetry Collector, how can we have a centralized data pipeline? So let's jump right into it. So because Open Telemetry is a community driven project, there are so many open source receivers that you can leverage out of the box. And so if you're already running Prometheus or Fulumbit in your Kubernetes environment or your Cloud Native environment, you don't have to just kind of turn those down and then leverage all Open Telemetry. You can still connect those to your Open Telemetry Collector data pipeline. So an example of doing this is collecting infrastructure data. So if you already are running Prometheus and in your Kubernetes environment, you can directly connect Prometheus, using the Prometheus receiver to the Open Telemetry Collector. So you can continue using Prometheus data that your Kubernetes clusters are generating in their endpoints, but just make sure that you stream the data through the Open Telemetry Collector. And if you leverage platforms like AWS or Azure, they have also written receivers specifically for their various services that connect directly to the Open Telemetry Collector. So recently, I actually had to instrument Kafka environments for a particular customer to get metrics. And there were three different ways that we could pull data specifically metrics from Kafka clusters. And here's a couple of ways that you can do it using the Open Telemetry Collector. So the first way is through the Kafka Metrics receiver. So the Kafka Metrics receiver leverages the admin API to pull data directly from the Kafka server to get very specific like partition topic specific metrics for your Kafka clusters. That's like one way to do it. But what if you don't want to give your Open Telemetry Collector access to your admin APIs, which is very understandable? Then what you can do is use the JMX receiver. So the JMX receiver basically uses targeted, basically targets specific ambient servers to directly pull metrics from Kafka, JVM, Cassandra, whatever you want. And we were able to collect health and performance metrics from our Kafka servers using the JMX receiver and putting that all through the Open Telemetry Collector. But let's say you don't want to do that and you want to just use Prometheus. That's fine too. You can deploy the JMX Prometheus exporter directly in your Kafka cluster and leverage the Prometheus receiver to constantly scrape the publicly available endpoints in the Kafka server. So this is a very long winded way of saying that Open Telemetry is very powerful because it puts the decision in your hands. There are so many different ways to get data out of your environment. And if you don't see an open source way that's already in the contributions repo to get data the way you want, you can write your own. So that's the real power of Open Telemetry. It's very flexible. And there are so many ways to do one thing. So yeah, once we've gotten all of that data, whether it's from Kafka or other technologies into the pipeline, how do we increase the usefulness of the data? So a really cool processor that I use at New Relic is the Kubernetes Attributes Processor. And it's really cool because it really dramatically increases the value of the logs, traces and metrics being generated from my application. And that's because the Open Telemetry Collector with this Attributes Processor injects metadata about my pods, about my deployment. So in the back end, I'm able to correlate application level metrics with infrastructure level metrics. And the reason that's so cool is because, let's say I have like an outage or a lot of spikes in error rate or latency, I'm able to correlate that if it's an infrastructure issue immediately with a CPU usage or other infrastructure metrics, so I can ping the right person if there's an outage. And this is really important because as we collect more and more data, we need to make sure that it is as valuable and as contextual as possible. And doing things like metadata injection to be able to correlate different pieces of data together is a really easy and powerful way to make your data more valuable. So now that we talked a little bit about first collecting data and then making it more valuable using things like metadata injection reads, can you tell us a little bit about how we can control the amount of data exporting that we're exporting from applications? Yes, this part is going to be very lightning. Okay. There are multiple ways to control the amount of data that you're generating as well as exporting. I'm going to specifically talk about how you can control the amount of data that you're exporting using the following two processors. The first one is the filtered processor, which can be used to you can configure it to include or exclude specific metrics, logs and traces. For example, you can use it to filter logs based on severity level or even the text itself. The second one is going to be the tail sampling processor, which is specifically for trace data. Let's say these are the traces that your system is producing. That sounds so loud. If you're using head based sampling, you are either one going to get a lot of noise because you are keeping all the traces that your system is producing or two, you're getting a randomized sampling based on the probabilistic rate that you've configured. So for simplicity's sake, let's say you've configured it to be 10%. As you can imagine, that means you're getting one out of roughly every 10 traces your system is producing, which could mean that if there's errors going on, it could be a little bit longer before you see them because it might take longer to surface them depending on how often these issues are occurring. By setting up the tail sampling processor, you can implement one or more policies to keep traces that are important for your system. So you can keep traces with errors, specific span attributes or that have cross-certain latency threshold. This helps to ensure that you're exporting only the data that you've determined is important for understanding your systems. Finally, as Daniel mentioned, you're going to want to export all this data somewhere to either a managed or open source backend so you can analyze it. The component that you're going to use to do this is called the exporter. You can use the default OTLP exporter since most backends now support OTLP data natively. And you can also use different exporters to send multiple data formats as well as to multiple backends at the same time. You could even export your traces and metrics to one backend and your logs to another if that's what makes sense for your observability strategy. This is a quick recap of all the components that we mentioned and where they fit into the OTLP collector architecture. Once you've got your receivers, processors and exporters set up, you now have the ability to build your data pipelines using the OTLP collector. All the configuration is handled within a YAML file so updating is fairly trivial depending on, of course, the complexity of what you have. And you now also have a centralized location for your observability configuration. These are just some challenges that we want to leave you with before we go to take into consideration with using the collector. The first one is it can be complex of scale. You will have to consider deployment patterns as well as load balancing the more data your system is producing. Monitoring the collector is only available with the collector at this time, which is not an ideal situation. And the current stability status is mixed since the core components are also at varying levels of stability. And that's it. That is all we have for you. Before we go, we hope to see you tomorrow at a vendor neutral open telemetry community event that is going to happen at the Colony Club Detroit. Go to that tiny URL to register and find more info. And thank you all so much.