 Hi, my name is Steve Flanders and I'm a Director of Engineering at Splunk, responsible for observability getting data in. My team and I work extensively on the OpenTillimetry project and I'm excited to join you today to talk about the OpenTillimetry Collector. For those of you not familiar with the Collector, I'll start with a quick introduction before showing how you can use it to really control all your data. So what is the OpenTillimetry Collector? The Collector is a vendor agnostic implementation that really allows you to receive, process and export any telemetry data in a seamless way. It provides a single binary that can be deployed in a variety of different ways, including as an agent on your hosts or as a standalone gateway. And it's the default destination for the OpenTillimetry client libraries. Let me walk you through how it looks from a reference architecture perspective. Let's assume you have one or more hosts, each host containing one or more applications and you want to send your telemetry data to one or more different backends. Typically how you get started is you would deploy the OpenTillimetry Collector as an agent on your hosts. For example, in Kubernetes this could be a daemon set. This would allow the collector to go ahead and gather metrics from the host itself and it provides a destination where if you deploy the OpenTillimetry client library, it will be able to send metrics and traces from your application to the collector itself. Note that the collector also supports popular open source client libraries, including Prometheus, Zipkin and Yeager. In addition, you may also want to deploy the collector as a standalone gateway service. The OpenTillimetry Collector is made up of a variety of components. First we have receivers which are used to get data into the collector. These can be either push or pull based. On the other end we have exporters, how you get data out of the system. Again, these can be push or pull based. In between you have the notion of processors, which are used to massage the data as it moves through the collector. Now for each of these different components you can build what are called pipelines. For example, maybe I have a pipeline where I have an OTLP receiver, OTLP is the default protocol used by OpenTillimetry, it passes through a batch processor and also a Kubernetes tagger processor before being exported out through a Yeager destination. In this case you can see that the converter is translating from one format to another. You may also have a second pipeline where again you are leveraging the OTLP receiver, this time with a separate batch processor and Kubernetes tagger and this time you're sending it to two different destinations, OTLP and Prometheus. As you can see, pipelines give you flexibility and choice. When it comes to telemetry data, at the end of the day it's your data, so you need to make sure that the components that are available meet your specific requirements. The collector provides flexibility and choice through configuration and there are a variety of different ways that you can leverage it to control your data. First let's talk about PII redaction. There are a few different types of scenarios where this may be applicable. Most commonly we have things like database queries, which can contain sensitive information, but this could also apply to application-specific metadata. In the example on the left we're doing two things. First, we're saying that if we notice that a span has an attribute with a key of PII and a value of false, then we don't actually want to go any further with this attributes processor. Otherwise we want to look for the key of database.statement and if we find it we want to hash that value. On the right hand side we have a different example. Here we're looking for spans where the service is called sellonamazon and we're looking for an attribute of job underscore args. If we find it we will delete the job underscore args attribute. Next we have sampling, which is typically used to control costs. In this example we're doing two things. First, we're looking for all spans where the service is data-platform and we're adding an attribute of sampling priority, setting it to 100 to ensure that these spans are always sampled. Secondarily we're enabling the probabilistic sampler and setting the sampling percentage to 15%. This applies to all data processing through the system, but note that the sampling priority takes precedence. Data enrichment is used to add metadata or to add resource information. For example, we can add an attribute called environment with a value of production for all spans that pass through this collector. In addition we've enabled the Kubernetes tagger, which allows us to add information such as the pod, deployment, namespace information directly from Kubernetes. This is possible when running the collector as an agent on hosts and it's something that an agent is in the best position to go ahead and add given that the application may or may not be aware of this metadata. Next, let's talk about reducing cardinality. You may come across services that are instrumented in such a way that the span named values return a large number of unique values, resulting in high cardinality. This can be overcome using the span processor in the collector. For example, we can include certain service names and we can write rules to actually normalize the span names, thus removing the cardinality. Here you can see an example with core DNS where the service instance and request name are being normalized. Finally, let's talk about fixing instrumentation flaws. It's much harder to actually change and modify code instrumentation than it is to update collector configuration. So the span processor also allows us to override the span name given other attributes on the span itself. For example, here you can see that we're looking for the Rails.controller and Rails.action attributes. We're separating them by a period and we're using them to rename the spans, thus providing value where value may not already exist. As I mentioned, for each of the different processors, receivers, and exporters that you have, you want to define one or more different pipelines. Pipelines are actually defined per data source. Today, traces and metrics are fully supported and initial log support does exist. The order in which you define the processors actually matters. The great thing about the collector is that it handles translation for you. This means you can receive in one format while exporting in a different format, thus providing a vendor agnostic solution. Now that you know different ways in which you can control your data with the collector, I want to actually show you live with a demo. For this demo, I'm going to start with a base YAML file for the collector. This will allow me to receive span data in either OTP or Zipcon format and export that data via a logging exporter so we can see it locally on the screen here. Let's go ahead and run this Docker container. Then I've actually constructed a JSON payload that consists of a single span trace, which I can then send via curl to the collector, and we can go ahead and check the output. Now above here, you can see that we did in fact receive a span. A span with a service name of API. It looks like it is a server span kind. And it has a variety of different metadata. For example, you'll notice the HTTP underscore response code, which is 201 in this case. It also has some interesting additional metadata. This looks like it might be a social security number, which would be PII, and definitely not something I'd want to attach to my spans. I can also see an email address here, and well, that might be sensitive and maybe I'd not that comfortable sending that data either. So what can I do about this? Let's go ahead and go back to the configuration here. You'll notice that I'm using an attributes processor, and it's currently configured to go ahead and add some metadata. In fact, it adds an environment tag where the value is set to Penic Clinic. Well, I can actually use this attributes processor to do a variety of different operations, like for example, that social security number really needs to be removed. So let's go ahead and use the delete action. And I saw the email address too, and that's a little bit sensitive. So maybe I'll go ahead and hash that value. So we'll just make those two changes. We will fire the collector back up and go ahead and send the exact same data again. And let's see what we get this time. So I still see, I get an environment pet clinic that's been added. So that's great. I can see that the email address is now hashed. So I don't really know what the value is, but I have a unique identifier that I could then query in the back end that I send this data to. And I no longer see a social security number. So that's great. As you can see, the collector makes it really easy to control your data. We hope you'll come take a look at the project, join the conversation on Gitter, or PRs are definitely welcome. So take a look at the good first issue and help wanted labels. Thanks so much for joining, and I hope you enjoy this presentation.