 Hi folks, thanks to be here for this talk about an introduction to open telemetry tracing. You can think about observability at the next level or as to provide insight into not only one single component, but about distributed system. And there actually are three main pillars of observability. The first is metric, the second is the logging, and the third is tracing. So basically tracing is a set of techniques and tools that help follow the business requests across the network through multiple components. This is really, really important. Again, in distributed system, you've got multiple components, your transaction, your business transaction will require all of those components to work together to achieve the business goal. If one of them fails, you're in trouble, and if it's repeatedly the same component that says an issue, you need to be aware of it. So tracing the request across all those components is important. I believe you probably are aware about some of the tracing pioneers existing already. So there is Zipkin, Yeager, Open Tracing. Those are the three most widespread and famous. Every one of them has proprietary interfaces and implementation. But as you know, we want something to be standardized so that the whole software that you write, we can write libraries that have one standard. And for that, there is something called the double-width 3D trace context configuration. It's quite easy to read, so you can already do that at home. And basically, if you don't have time, let me refresh it for you. Basically, you've got two important concepts. The first concept is the idea of the trace, and the trace is actually like the abstraction of the business transaction. So the trace will go from entry points to like the most deeply nested component and back again. That's the trace. And then you've got the span. The span is actually the execution of one part of the trace in a single component. And you can have, of course, for each component. You can have multiple spans, if you're interested, where inside this component, where the flow of the request goes. And each of those spans are bounded together in a parent-child relationship. So the first one is the important one. The first component is the important one. It gives the first span ID. And then the next component will check the span ID of the parent and will say, hey, I am this span ID. So it also generates a new span ID and say, hey, I'm bound with this one. And through all those parent-child relationships, you can actually make sense of the flow and check the flow overview in one set. Open telemetry is a set of tools that implements observability and that relies actually on double-free C trace context. So it's an implementation and more. So it's compatible. And it also go beyond because W3C is good for web stuff, but perhaps you want to trace the request through Kafka, somewhere where you can store. And in this case, the specification does that handle it. Open telemetry allows you to trace these business transactions across many different components, not only web ones. If it's a merge of the open tracing or an open-sensors project, so it's one of those few merges that were successful where people decided to join their efforts to create something better. It has become a CNCF project, which is good, meaning that it has support. It will be supported for a long time. It's licensed under the Apache license, so it's also good if you want to use it right away. You don't need to acquire a license or whatever. And it's usually popular, especially on GitHub. The architecture in itself is quite easy. Basically, you've got components, whatever they are. And then you've got what they call an open telemetry connector. And this open telemetry connector accepts data in a specific format. And this specific format allows to have these like parent-child relationship between spans. So the ID is on the client side. You've got stuff that dumps data into the hotel collector. And then you've got something that is able to search and display data from the hotel collector. The open telemetry collector itself doesn't provide anything. It's just storage stuff in a certain format. So we need something afterwards. Open telemetry provides a dedicated collector. But actually, Yeager and Zimkin, which are also tracing providers, they are able to provide the same collector, or let's put it that way, they also provide a collector which accepts open telemetry data. So basically what they did, they kept the storage engine, they added a new interface where you can set your data in open telemetry format. If you already have your architecture, you're tracing an architecture, you can easily move to open telemetry because, well, the collectors of Zimkin and Yeager, they have this additional interface. You just need to change the formats and the ports because I think every one of them has different ports. On the client side, it's, well, I wouldn't say easy, but the first step is straightforward. First step is auto instrumentation. This is only available when you've got a runtime. For example, on the Java side, you have the team that is on runtime. On the Python side, well, Python is a runtime. Node.js is a runtime. In this case, you will delegate to the runtime to do auto instrumentation. I told you about, like, automatically logging, entering, and exiting a method. It's exactly the same. Here you will do that automatically. It gives you already a lot of insight. Now if you want to go further, you can actually get the library, depending on your text stack. Again, you can check the open telemetry website. You will notice there are lots and lots of stacks that are supported out of the box. And for Java, there is one, for Python, there is one, for Rust, there is one. You will find there and then you can either call an API or use some annotation. I mentioned it's practical introduction, so here let's try to do some practical stuff finally that we have dealt into the theory. So here is my use case. My use case is, well, simple for a real application, but a bit more involved for a demo. So at the beginning of an API gateway, I'm using the Apache API 6 gateway. It forwards the request to my main application, which is a string bootcoding application. Actually, it gives, like, products API. And then it has the detail for the product itself. But it relies on two other components. The one for the pricing. The pricing is implemented in Python through a Flask framework application and for the stocks. So how many items do I have in which warehouse? I have created a Rust application using the XM framework. So the entry point is actually the reverse proxy API gateway. Most information systems have such an entry point. You probably never expose your application directly over the Internet. You have something in between because, well, you want to protect your information system from an illegal access. So that's the most important point. As I mentioned, I'm using Apache API 6. Perhaps you don't know about Apache API 6. It's an Apache project. So basically, again, good for maintenance. Everything will be there with a license that will never change. It's based on the very successful engine experience proxy. Then you've got a Lua Jits additional open resting layer on top, which allows you to do scripting in Lua over the NGINX. And then you've got out-of-the-box plugins. So to configure it, there is this general configuration. As I mentioned, they're like Apache API 6 as a plugin architecture. So here I say, hey, I will be using open telemetry. It's an out-of-the-box plugin. You don't need to write any code. And then you can tell, hey, this is the name by which I want to be known in the data of open telemetry. And this is where I will send the data. So I will be using Docker compose. And so I have the dedicated Yeager component. Then for each route, so here I have a single one, but you can have different configuration, depending on which route you will say, okay, how much sample do I want? Normally, and depending on your volume, you probably don't 100% because it will overflow your stuff. You want a sample. Here, this is a demo. I will say I want to sample everything. Again, probably not what you want to do. And then you can log additional attributes. So here, for example, I decided for no reason thus for a demo purpose to have the route ID, the request method, and an additional hair. So if I can pass through the clients some hairs and they will be traced along the span. The next step is the GVM level. As I mentioned, GVM is a runtime, so I can easily use auto instrumentation. And on the GVM auto instrumentation is for Java agent. So this is quite easy, actually. I just need to pass the Java agent when I start the application and I don't need to write anything. So your developers, they are completely isolated from this tracing concern. They can write their code and everything will work as expected. This is regardless of the language and the framework because it's the Java. So here, this is how it works. Here is my Docker file to build my Docker container. This is a multi-stage Docker file. First, I will compile everything for a GDK. And afterwards, I will run it for a GRE because I don't need a GKE and it's actually bigger and less secure. So the first thing I do is like normal standard builds. And then afterwards, I get the job that I just built and I add the Java agent through GitHub. And when I run it, actually I run it through the Java agent. And this is as simple as it gets. You cannot be simple. Afterwards, you can do more precise, more fine-grained calls through manual instrumentation. It leads an explicit dependency in the application. This time, your developers need to be aware of it. And then there are two ways to do that, either through a regular explicit API call or for annotation. I'm benefiting from Spring Boot, so basically I will use annotation. I will issue the code just afterwards. Okay, now it's time to delve into the codes. I will just focus on the like cutting Spring Boot ports. Everything is on GitHub, so in case you need to check, you can check the Python part. You can check the Rust ports. Here I will focus on the Java stuff. I've created my application on Spring Boot Starter, start.spring.io. So here I'm using the latest version of Spring Boots. I'm using the like latest LTS version of Java, which is required by the latest version of Spring Boot. I'm using also the latest version of Scotland, and then it's a reactive application. So I will be fetching data using R2DVC. Otherwise, I'm using Web Flux. So again, to be reactive. And the rest is just like standard cutting stuff. I didn't want to bother myself with a regular database. So I'm using H2 using the R2DVC H2 reactive driver. On the code side itself, here I'm using Kotlin, so I want to use core routines because well, that's how you can do easily reactive code stuff. So I'm using the core routine credit repository. This is my R2DVC repository. Here I have the handler. And you can see that I have suspend function. Suspend function are like for core routines in Kotlin. Then I have one endpoint and the other endpoint. This one endpoint is for all products. This one endpoint is for single products. Let's see the first one. Okay. And then the rest will be exactly the same. So I will fetch all product. I will find all of them into the repository, which in turn will look into the H2 database. And for every one of them, which probably what you shouldn't do in real life, I will fetch the product details. So whether, not whether, but their price and their availability in the stock. So here I can see how it works. Again, I will have two different calls, like protected under a nascent block, which means that here, because I'm using this picture.io, they will make the calls in parallel. And we can check that it worked like this in the traces. So I will get the price. I will get the stocks. Then I will merge everything. And here is how I merge everything. So I transform data into the expected data. And at the end, I create a product with details, including the products from the database in the catalog, plus the price plus the stocks that they have, like change of it. For example, I don't want to return to the client anywhere else where the quantity is like zero or less, because well, not zero less, just zero, doesn't make sense. So I just filter them out. And at the end, I'm using the pins DSL from Kotlin and the router DSL, or here the co-rotor DSL to assemble everything. And I start my application with these pins now. So even if you are not familiar with Kotlin, if you are a Java developer, I think it should talk to you. And here you can see my two endpoints slash products for products and slash products slash id for my products. Now I assemble everything through Docker compose. So this is the Docker compose file. I'm using Yeager. Yeager is available in multiple triggers. You can have like different containers. For example, here I'm using the old one. So I'm using the like batteries included package, Docker image in this case. So I can already have the open telemetry collector provided by Yeager. So it's not the open telemetry collector. It's the Yeager collector that allows an open telemetry interface. So here I don't need to think about the architecture of Yeager. I'm just using the Docker image that does everything. Then I'm using API six, because I want to protect my services. Then I have the catalog, which is the Spring Boot Kotlin application that I've just shown. And here I need to tell several like configuration parameters. The first one is where does the Java agent need to send the data? Well, to Yeager on this board. How does it, how will it flag this component? Here we call orders, which is long. It should be catalog. Then here does I want to export metrics? Here I said no, I don't want. Of course, depending what you want to do, you can also export metrics and logs, and logs the same. And now pricing. I do the same for the Python application, but again, this is not relevant for this talk. So here pricing, same stuff for the Python application, not relevant for this talk. Stock, same things for the Rust application, not relevant for this talk. Let's start this architecture. So it might take a bit of time, especially with the JVM to start. I will just like speed the time and let's go very fast. Okay. The logs tell us that it has started. We can check with Docker PS. Docker PS. So here it's in that everything has started. We've got catalog, the pricing, the stock and Yeager. Now we can issue our first like curls or curl. I will be using the header that I have configured Apache API 64. So if I remember its hotel key, then I can say, let's say in the world, because I have no imagination. And I'm on local hosts, 1980, which is Apache API 6 default ports and we'd say products. So it takes a bit of time because it will go through all our systems. So here you can already see that the catalog has taken then none of the stocks, the pricing and Apache API 6 as well. So we've got the response, which is not very interesting as it is, but it still gives you the data you ask. And now the idea is to check the traces. So I'm now on like the Yeager Web UI and I can check and I will go exactly here and here we can see all the services. So there are some traces here I need to refresh because here I need to find the traces. And here we have a single request because we sample everything here we have it. And we can see already with only auto instrumentation a lot of interesting data. So we can see that we have our API 6, which is the entry points. And here we have the orders, which I misnamed should be called catalog, but here it's orders. And here we have the product. Here we have the first auto instrumentation inside of product because we are using spring boots. We have lots of proxies inside, you know, how spring boot works. And here decided, hey, here I will make a call for a proxy. I will trace it. Here we have the final, why? Because it's an interface provided by spring boot. We didn't provide the implementation. So basically again, it's a proxy. So it's automatically traced. We can see what here that there is a call to the other components called a stock. So it's traced as well, which is good. And here we've got the second one. So here there is one for stock and one for pricing. And here we see that in one case, I went directly to the component and the other one went through the API gateway. Both are completely possible. This is how I configured it inside, sorry, my architecture. Basically, in one case, you can say, I want to protect everything and always need to get back to the API gateway to do some authentication, authorization, whatever you want. And the other side to say, oh, I'm pretty secure, I can directly go through it. But it gives you insight into your architecture as well. In case you misconfigure something, you can check it through the traces. Something interesting as well. We can see that the get calls to the stock and the pricing, they are made in parallel because we use call routine. So this is also a good way to check that you actually coded your stuff correctly. If you see one going after the other, then probably your code was not right. Though tracing is not made to do that, you can also validate some of the codes. And then as I mentioned, it's not good to do that, but here for each of them, I go to the stock and the pricing, stock and pricing, stock and pricing. And we can check, for example, that on Apache API 6, I actually add the additional stuff that I sent. So basically the routes and the get. And here I'm missing the hotel stuff, so probably I didn't choose the right one, but believe me, it should work. Now that also that already give us some information about our flow, but we might want to bear. We might want, for example, on the get to say, like, which internal method did we code? Which parameter? So let's do it. So now I want manual instrumentation. It means I need to explicitly couple myself to the library. So here, because I'm using Spring, as I said, I want annotations. I don't want to have API calls. Actually, if you check the documentation of the open telemetry in Java, to get an exporter is not that fun, requires a lot of API calls. And well, I have annotations. Spring Boot is compatible with open telemetry, so let's use it. So I've added this, like, additional dependency in my code, and now we can check the application itself, code itself. And we can go here. And here we can see that I've added, like, here, this with span. So this with span means that it will be instrumented, and you will find it in the trace. So I should have these product handler products. If I'm calling one single product, I will have this one. But it's also possible to use additional details. So for example, here I will have this product handle fetch, but I also say, hey, here, not only capture this code, capture this ID, so which product ID will I fetch? Which means that here it's interesting because normally I shouldn't need it. Here you see that the ID parameter is not used because I already have the product. But because I want to capture the ID, I need to separate this parameter. So I wouldn't be able to capture the span attribute product because then I would have not the ID, but the whole memory reference, unless I create a two-string whatever, which is not a great ID. So here I change my method signature a bit to explicitly pass the ID, and then, well, I don't use it. But then it means that this will be captured by the tracing, by spring boots, and I will find it, which might be super useful, especially if it fails. So normally now everything should have started, and we can try again with this configuration. So here it's the same request. I have to change the header because I missed the previous header. It was not hotel, it was OT. So let's run this. Again, we can check that everything works on the logging side. So here I'm in the catalog, then I'm in the stock, the pricing, whatever. I've got the response, and we can check back on the Yeager UI how it looks like. We expect more details. So first we can already see that we have more spans than before. Just to check on the Apache API6 site, we can see that now my OT key, this header has been logged, which is good. And then we can see that I have the products here. I have the fetch here. So basically we added additional data. So inside the components, we added a couple more spans to understand how the flow of the code went inside the components, not only throughout across components. So thanks for your attention. I hope you learned something. I showed you how you could use open telemetry, how you could use auto instrumentation, how you could use manual instrumentation, and I believe now you can start your journey.