 Hello, my name is Steve Flanders and I'm a Director of Engineering at Splunk, responsible for observability, getting data in. My team and I work extensively in the Open Telemetry project and today I'd like to talk to you more about it. Let me start with a few slides just to introduce you to the project and then we'll jump into a demo so you can see how you can get started. Open Telemetry is a CNCF project and is currently in Sandbox. The Open Telemetry project deals with cloud native telemetry. If you've heard of observability, you've probably heard of the three pillars of observability, which includes traces, metrics, and logs. These are just different data sources, but for each of them there's a variety of different telemetry layers that need to be considered. For example, you have the instrumentation APIs and canonical implementations. The net result is a client library that you add to your application in order to generate and emit the telemetry data from your environment. These are all language specific according to the data sources that you're trying to emit the telemetry data from. You also have the data infrastructure, which include things like agents or standalone services or gateways. These are deployed in the environment typically to collect, aggregate, process, and send this data to the backend or backends of your choice. And then you have a variety of different interrupt formats, which includes the protocols for sending the data over the wire, as well as things like context propagation in the case of distributed tracing. All of these aspects are what Open Telemetry is providing a solution for. The way you should think about it is, it's everything that you deploy in your environment in order to generate and emit telemetry data as well as collect it and send it to the backend or backends of your choice. Open Telemetry does not actually provide a backend, instead it offers a vendor agnostic solution, which supports all the popular open source solutions available today. In the case of distributed tracing, this would include Yeager and Zipkin. In the case of Metrics, this would include Prometheus. In addition, there's a wide variety of vendor support to send it to the backends of your choice. The Open Telemetry project is made up of three primary components. You have the specification, which is foundation to the entire project, and is subdivided into the API, SDK, and data aspects. These basically define how you go ahead and construct and generate the telemetry data from your environment. Next, you have the collector, which is used to receive, process, and export telemetry data. It is a single binary that can be deployed in a variety of different form factors, including as an agent on your host, or as a standalone service or gateway. And it's actually the default location where client libraries will emit their telemetry data, thus providing a seamless end-to-end implementation. Finally, you have the client libraries. These are language-specific and are used to actually instrument your application. There's broad support for all major languages today, initial support for traces and metrics, and log support recently kicked off and is part of Open Telemetry's charter. In the case of client libraries, you have two options. You can either manually instrument your app, which requires code changes, or you can automatically do it through runtime parameter and dependency changes. The Open Telemetry project is currently in beta, and has been since March. But it's actually a rather mature project. It's the joining of two other projects, being Open Census and Open Tracing. And a release candidate was recently announced for the tracing aspect, with metrics soon to follow. And Open Telemetry is actually a very active project. In fact, it's the second most active project in CNCF today, behind only Kubernetes. And this is according to CNCF DevStats. Open Telemetry sees a wide variety of contributions from the industry and brought adoption as well. All three major cloud providers are actively involved with the project. And Azure and GCP currently instrument their backend services with Open Telemetry and emit that data to their customers. Every major vendor is involved with the project. And from an end user perspective, you'll see companies including MailChimp, Postmates, and Shopify, who are currently maintainers of different client libraries in Open Telemetry, and have actually instrumented their production applications with Open Telemetry today. Beyond this, you'll see a variety of other contributions throughout the community. For example, Yeager, which is another CNCF project, recently announced that they are moving from the Yeager Collector to the Open Telemetry Collector, and they actively contribute to Open Telemetry today. You also see a wide variety of roadmap items, including support for Envoy, Open Metrics, and Spring. With that, I'd like to jump into a demo and show you how you can get started with the project. For this demo, I'll be using the Spring Pet Clinic Microservices demo application that's available on GitHub. If we actually take a look at the readmate, you'll see that it actually provides a architecture diagram depicting what the environment actually looks like from a microservices perspective. There's a variety of different Java services that communicate with one another, and it provides a Pet Clinic web application that you can interact with. That application looks like the following. You have the ability of viewing a variety of different owners in the system, as well as veterinarians, and you can even add your own users to the system. Now, what I'm going to do is I'm actually going to modify this repository to add distributed tracing because it doesn't have it today. I've actually put together a test PR here showing off how easy it is to get started with Open Telemetry in order to omit that telemetry data. As I mentioned, the Open Telemetry Collector is the default destination for sending telemetry data today, so I've gone ahead and added the collector to the microservices demo application. The demo application is actually composed of multiple Docker components that are in a Docker compose file. I've gone ahead and provided a collector.yaml file to configure the Open Telemetry Collector. I'm ensuring that it can actually receive data, can process that data, and it can send it to a locally running zip-con server. This is a one-time configuration, and it's very easy to modify this configuration later should you have additional requirements in the future. Beyond this, I have to go ahead and modify the individual applications themselves that are written in Java. Now, thanks to Open Telemetry, this is actually super easy. We can use auto instrumentation and just change runtime parameters in order to actually get instrumentation injected into the environment. To do that, you'll see that we're running the Java command, and what I've done is I've gone ahead and added this Java agent flag, which points to the Open Telemetry Java agent jar file. With this in place, and by starting the application, it will go ahead and auto instrument for me. The only piece of additional information I need to provide is actually the service name, and I can do that through the use of an environmental variable as you see here. As you can see, this is the discovery service. There's also customers, visits, and a variety of others. For each of the Docker containers, I do the exact same thing. I add the Java agent command, and I define the service name. This way, I can see each service independently in the back end to which I send that data to. Finally, I've gone ahead and added the collector to the Docker compose file, so we can fire it up with the configuration we showed at the top, and I've ensured that the Docker file itself goes ahead and pulls in the Java agent jar file, so when we're building, it is available for use by all the Docker containers. That's it. Now, I can start up the application, and I can see the Telemetry data that it's emitting. I have these configuration changes running locally. After making the changes that I just showed, I went ahead and built the Docker compose file, and then started it up. You can see here that it's actually emitting Telemetry data, and that's being reported by the collector. This is because I enabled the logging exporter, just so I can show for demo purposes that Telemetry data is being emitted. But now, let me actually show you in the UI how the Zipkin server is actually receiving this data. In Zipkin, I can go ahead and start searching for service names. You'll see a variety of different service names are available to me, and these service names will actually match the service names you'll see in the PR that I put up. This is why we went ahead and named the individual Java services, so we can see them each independently. Let's go ahead and look at one. For example, we'll call up the customers one. We'll run a query. Now, we can actually see distributed traces in the system. I can see a variety of different calls, for example, the admin service calling the customer service, or the customer service calling the discovery service. We can drill into these services, and we can also expand them to see their metadata that they each have. For example, here we have the customer's service with an HTTP put command. That was a successful command. We have the HTTP status underscore code. We can see the URL that was called, peer net name. We can even tell what open-symmetry client library was used. So this is using the Java auto-instrumentation. It's actually a JAXRS client in version 1.1. So a rich amount of telemetry data will be provided for me out of the box. You may recall earlier that I said there's specifications in open-symmetry. One of them is actually around data, or more specifically the semantic conventions. These are a generic way, an open standards way, in which you can actually name something. For example, in the case of RESTful services, this would include things like the HTTP.method, or the HTTP.bet status underscore code. By having an open standards way of actually defining something, you can actually have a vendor agnostic solution. This way we all know what a RESTful service is, because it has the same metadata consistently across it. In addition, by having the information about the telemetry data itself, you can tell what versions of things that you're using. This can, of course, also be complemented with infrastructure information. Open-symmetry calls that resources, and also has semantic conventions around resources as well. This way you could, for example, know what container your application is running in, or which cloud provider, which data center, which region. All of this can be enriched and added as additional metadata onto the spans that are being generated. Next, I want to go back to the collector. As I mentioned, the collector can be used to receive, process, and export telemetry data. But what are some of the use cases for it? To demonstrate that, I'm going to pull up a similar configuration to what we used to earlier. This configuration supports receiving Zipkin data and sending it out locally to a logging exporter, so we can see the output on standard out. Let's go ahead and run this Docker container. Then I've actually constructed a JSON payload that consists of a single span trace, which I can then send via curl to the collector, and we can go ahead and check the output. Now above here, you can see that we did, in fact, receive a span, a span with a service name of API. It looks like it is a server span kind, and it has a variety of different metadata. For example, you'll notice the HTTP underscore response code, which is 201 in this case. It also has some interesting additional metadata. This looks like it might be a social security number, which would be PII, and definitely not something I'd want to attach to my spans. I also see an email address here, and, well, that might be sensitive, and maybe I'm not that comfortable sending that data either. What can I do about this? Let's go ahead and go back to the configuration here. You'll notice that I'm using an attributes processor, and it's currently configured to go ahead and add some metadata. In fact, it adds an environment tag where the value is set to Penic Clinic. I can actually use this attributes processor to do a variety of different operations, like, for example, that social security number really needs to be removed. Let's go ahead and use the delete action. I saw the email address too, and that's a little bit sensitive, so maybe I'll go ahead and hash that value. We'll just make those two changes. We will fire the collector back up, and go ahead and send the exact same data again, and let's see what we get this time. I get an environment pet clinic that's been added, so that's great. I can see that the email address is now hashed, so I don't really know what the value is, but I have a unique identifier that I could then query in the back end that I send this data to, and I no longer see a social security number, so that's great. One particular use case of the collector is to go ahead and add, remove, or modify the metadata, but the collector can do other things too. For example, not only can it support traces, it can support metrics. Given I'm running everything locally here on Docker, let's go ahead and enable the Docker stats receiver. We'll set the collection interval here to one second, just for demo purposes, and let's quickly go ahead and build out a metrics pipeline. We have a Docker stats receiver. We can configure processors. In this case, I'll add a batch processor, and let's use the same logging exporter here, and then try running the collector again, and let's see if we actually get any stat information. Look at that. We now have metrics being collected from Docker, and I can see, for example, the container network I owe usage transmit packets, and all the associated data with that. Pretty cool. As you can see, very simple and easy here. It's all configuration driven, and it provides a lot of capabilities. I only showed a couple, but there are several more supported, and it's a very extensible platform, so very easy to add additional receivers, processors, or exporters to the system. I hope you enjoyed that demo, showing how easy it is to get started with OpenTelemetry. We would love for you to come join the project and get involved as well. You can join the conversation, which happens all the time on Gitter. You can come join one of our many special interest groups. Every single language client library has its own special interest groups, as well as the collector and all the specifications. And of course, PRs are welcome. We try to label all of the GitHub issues with either good first issue or help wanted, which is a great way to get involved with the project. Thanks so much for joining me.