 This is going to be a session on Yeager and deep dive into what it is, how you can use it, and how you can get value out of it. I'm kind of curious, though, for a show of hands, how many people have used Yeager? No one. Two, a few hands. All right, there we go. Awesome, great. So I want to talk about the project, its current state, and where it's going from a roadmap perspective, as well as some of the new features. I want to actually show it live, so you can actually see how you can actually leverage it as well, and I'll try to leave some room at the end for some Q and A. First, a quick introduction on myself. My name is Steve Flanders. I am head of product and experience at a stealth startup in the Bay Area of California in the United States called Omniscient. We are in the observability space, and we actually contribute to open source software, including Yeager, open sensors, and open symmetry. If you're interested in learning more about observability in general, I'll have a few links available. All right, so what is tracing and why is it important? Well, if you look at very large companies, the Netflix, the Facebook, the Amazons of the World, what you'll see is that they have very complex architecture. They have moved to a microservices-based architecture where they have a lot of different service-to-service communication, and this can lead to complexity. Uber, who actually invented Yeager, they have a very complex architecture as well, and so calls actually go through multiple of these different services in order to call and Uber to get transportation, and so every request within your system that's known as a distributed trace, and when you try to troubleshoot a distributed tracing system's problems, it can be very, very hard. So when you have tens of services, maybe it's not that complex. When you have hundreds of services, it gets very complex very quickly. One of the biggest problems is you move from a monolith to a microservices-based architecture is problem isolation. Where is the problem? Where is the root cause? And a monolith is in the monolith. With a distributed architecture like this, one microservice can actually cause errors upstream and downstream, a rippling effect. So while this red directionality of arrows shows a single call, there are actually billions of them within Uber's infrastructure every day. The same applies to Facebook, Microsoft, Google, Netflix, any large company that has a lot of microservices has a lot of calls in a given day. Now, what you wanna know is what's going on and who's to blame for issues when a problem arises. If you're a developer and you're on call, if you're an SRE and you're on call, sorry, I don't know what happened, sorry for that. If you're an SRE or a DevOps engineer and you're on call, you want to be able to isolate the problem and remediate it as quickly as possible. So how can you do that? Traditionally, back before microservices, you used metrics and logs to do that primarily. Metrics were used to measure like CPU utilization, memory utilization, logs were used more as a root cause. Who made what change, why, what was the error as a result of that? These tools are powerful even today, even in cloud native or microservices based architectures, but the problem is they're missing contact and correlation. So I like to refer to them as symptoms based tools. They told you why something occurred or that something occurred, but not why it occurred. Now, in a microservices architecture, this can cause a lot of problems like alert forms. If one microservice starts generating high CPU utilization or a large number of errors, those errors or that resource consumption can actually propagate upstream and downstream from a given service. If different services are owned by different service owners, well then you end up waking up a bunch of people at the same time in order to troubleshoot an issue that actually started in one service. So how do you get context and correlation? Distributed tracing is the way. Prior to the cloud native error, this is known as application performance monitoring. The focus is really on performance, but distributed tracing can actually do more than performance. It can do availability as well. It can help you with on-call, root cause, any troubleshooting within your environment. What makes distributed tracing powerful is that it can actually follow a request throughout your entire system. That means it knows who called who. It knows who originated an error and who propagated an error. This graph here on the screen, this represents a visualization of a distributed trace. This was constructed from distributed tracing data. You can actually see different services. Red indicates errors that occur in those services and you can also see via the arrows which service called which service. Completely constructed from distributed tracing data. On top of that, we can actually enhance metrics and logs with distributed tracing data so that metrics and logs have context and correlation as well. So with that background, let me introduce Yeager. So Yeager is a distributed tracing background. It doesn't focus on metrics and logs, it focuses on distributed tracing. It was actually invented at Uber. If you've heard the name Yuri before, he's one of the main architects and contributors to Yeager. It was open sourced and made available in 2017. It is a CNCF project right now in the incubating phase but it is up for graduation from CNCF and hopefully that will be occurring soon which speaks to the maturity of the solution. Now it's a very active community. It's completely open source and I definitely encourage you to get involved if you're interested. You can see some of the stats here. It's primarily driven by maintainers from Uber and Red Hat but there's a lot of open source contributions as well. There are actually issues that are tagged with like good first issue or help wanted. Those would be a great way to get involved with this community if you are interested. Now what is Yeager? I said it's a distributed tracing platform. It's actually made up of four different components. You basically have the, in the bottom right, instrumentation libraries. This is how you extract distributed tracing data from your application so that you can consume it. Then you have the data mining platform. You can think of this as collection such as an agent and a collector as well as the ability to run analytics on top of that data so that you can query over it. You have the tracing collection backend where you actually store that trace data as well as the analytics that you have constructed and then of course the visualization front end how you can query the data out and visually see it so that you can troubleshoot your environment. And I'll demonstrate each of these in a moment. Okay, so there are two different architectures for Yeager. This is V1, V1 and V2 are very similar. This is actually in the Yeager tracing.io documentation. So very easy to reference. Most of the blue items or light blue items represent components of Yeager. So basically you will have an application as shown in yellow here. That's the software that you have written. Yeager provides a client or client library as it's known as which can be used to extract distributed tracing data out of your application. It then offers an agent that can run with your application or the host or container that the application is running on. The agent then sends that data to a collector. The collector can then send that to a database of your choice. There's a few selected. I'll talk about them in a moment. There's this notion of Spark jobs in V1. This is what actually performs analytics. It's used to build dependency graphs so we can actually visualize the distributed trace like I showed earlier as well as to compute SLI or service level indicators metrics from the distributed tracing data. And then you have the query interface in UI so that you can get the data out of the database and actually be able to see it. Now the only difference between V1 and V2 is the analytics engine. So in V1 it's done via Spark jobs after the fact. In V2 it's done via a Kafka message bus and Flink streaming for analytics. So it's done more real time. Now it's not like V1 is deprecated and V2 is what you should use. There are valid use cases for both V1 and V2 and both are fully supported depending on your business requirements. The advantages are V2. This is more of a production, enterprise ready, highly scalable architecture. The Kafka message bus makes it so that you can handle bursts of traffic, bursts of traces in your environments. The Flink streaming jobs allow you to get analytics data as the data is being ingested instead of after the fact. So it's a little bit faster and a little bit more scalable. Both architectures are encouraged. Now while the blue boxes represent Yeager components, you can mix and mingle different components as you see fit. For example, let's say that instead of Yeager you want to use a Zipkin client library. That is supported as well. In fact, Yeager fully support Zipkin. So if you're using Zipkin or coming from a Zipkin environment, you can actually leverage Yeager as well. In addition, if you wanted to leverage, let's say the open census service which is made up of an agent and a collector instead of the Yeager agent and collector, you can do that as well. This architecture is still fully supported. You could still get the data into a Yeager backend and you could still use the Yeager UI to query the data out. So you can mix and mingle however you see fit. The advantages of this are clearly flexibility, but from a collection standpoint, the agent and the collector, I could now send to Yeager and Zipkin at the same time. I could send to Yeager and Omniscient, let's say for example, at the same time. So there's flexibility if you leverage different components, but that's not a requirement. You could also skip the analytics engine if you really wanted to. If you didn't need the service level indicators or the dependency graph, if you just wanted raw traces and the ability to query over that, that's possible as well. So again, the architecture provides flexibility. All of the components are available to use. It's a modular architecture, so every box basically represents a separate Docker container that you can deploy and so you can mix and mingle to fit your needs. Now I should point out that Yeager loves open standards. I'm actually doing a talk immediately after this one in this room on open telemetry, which is another CNCF project, the merging of open census and open tracing. The Yeager client libraries that I mentioned, they are actually open tracing compatible today. OpenCensus fully supports Yeager today and these two projects are merging to form open telemetry. In fact, Yuri wrote a post on this. I have a link at the bottom, it's a medium post and Yuri talks about that in the future, Yeager is considering removing its client libraries and relying on open telemetry's client libraries and also removing its agent and collector and moving to the open telemetry agent and collector. So Yeager is open source software, Yeager loves open standards, open telemetry is providing some of those standards, but it's definitely something to take a look at in the future. All right, going back to Yeager, the technology stack. So Yeager is written in Go, it's completely Go language today. It has a very pluggable storage architecture. This is actually critical, it's one of the newer features that came out in the 1.9, 1.10 timeframe and 1.12 is the latest version. So basically the way Yeager handles this is they support three core storage back ends in the Yeager repository itself. Those back ends are Cassandra, Elastic and Badger and then it has a pluggable architecture so the community can contribute other back ends if they want those as well. And there's a separate repository for that. It's maintained out of Yeager core. Today that includes Influx DB and Couchface. There's also a GitHub issue open for other types of back ends such as DynamoDB. So if you're interested in contributing to that, PRs are definitely welcome. The front end is a modern React JavaScript based front end. It's very responsive, you'll see it here in a minute. I already talked about the open tracing instrumentation libraries and how that uses an open API standards and there's already integrations you saw in the V2 architecture, Kafka and Flink and of course this could be extended to support other analytics back ends if you want it as well. Okay, so integrations. I just wanna cover this real quick. Depending on where you are in your cloud native journey there's a bunch of ways to get going with Yeager. So for example, if you're on Kubernetes there's a native operator that you can deploy today. If you're on Istio, there's integrations for that. If you leverage Envoy as a sidecar for your data plane or as part of a service mesh that's already instrumented and fully supports Yeager. So there's a bunch of ways to get started with little work on your end to actually consume Yeager. I definitely encourage you to take a look at these resources. I don't have time to cover all of them. All right, new features. So the latest version is 1.12. I wanna talk about some of the changes in the 1.10 to 1.12 timeframe. I think this is important to speak to the maturity of the product. As I mentioned Yeager is up for promotion in CNCF to become a graduated project. So a lot of the focus that you see today is around hardening the existing feature set to make sure that it is production and enterprise ready. So for example, Elasticsearch being one of the primary back ends. There's been a lot of improvements to the security side of the house to ensure that it's hardened to best practices to ensure that your data is secure in that database. GRPC support is natively being added to it. They see that as the future. In fact, the front end and query interface was recently migrated over to GRPC in order to get data out of the back end. Better Zipkin compatibility. As I mentioned, Yeager is fully Zipkin compatible. So as changes are made to Zipkin, Yeager is also enhanced so that it can support that new functionality. You can mix and mingle Zipkin and Yeager as you see fit. It's not uncommon. I didn't talk too much about instrumentation, but there's something known as context propagation. That's how you actually pass, trace, and span data between your applications. There's different header formats for that. Yeager has one, Zipkin has one. Zipkin's format is probably the most common today. It's called B3. So if you're using B3 headers, you can still leverage the Yeager back end and Yeager collection mechanism. It's worth calling out. I talked about open standards. There is a new context propagation format standard that's coming out from W3C. It's called W3C Trace Context. All of the instrumentation libraries will be supporting it going forward. OpenCensus is natively adding it right now and OpenTelemetry will have it as part of its first version. That is the context propagation format that you should probably be considering going forward as that will become the standard. And then from a UI perspective, there are some enhanced capabilities. I'll actually demonstrate some of these. And I provided links on how do you can actually get the change log for the changes that are coming in these releases? You can actually follow along. They do tag the milestones on GitHub, so it's very easy to see the new features that are coming out. And then definitely take a look at the website. I mentioned YeagerTracing.io. You can actually contribute to this as well as part of the GitHub repository. A lot of rich resources here, very easy to get started. With that, I wanna switch to a quick demo and then I can show you a few more slides. So what I've done here is, I mentioned everything is kind of a Docker container. You don't need to memorize these. I'll show you the slides. It's actually uploaded to this session. So if you open up the schedule builder, I have these links directly in there. But with one command, I can actually go ahead and fire up an all-in-one Yeager instance. It has the collector, it has the database, it has the query interface and the UI all built in. And then in addition to that, there's also an example app. It's called the Hot Rod app. This is on the YeagerTracing site and I will show you the links for that in just a minute. So with that in place, I can actually locally run this and let's make sure it's easy to see. So here is the Hot Rod app. It basically rides on demand. Think like Uber or a DD here. And you can go ahead and request cars. I can click it a few times and it will dispatch them. And this is basically calling a microservices back end that has been instrumented and that data is going to a Yeager instance. And I can choose different cars. I can click it multiple times and it makes a whole bunch of different requests. Now on the flip side, this is the Yeager UI. So the first time you log in, this is what Yeager would look like. It basically provides a query interface over your distributed traces. So you can see here, I have a bunch of different services. This is from my Hot Rod app. I can go ahead and select one. In this case, I can just run a query over that data and I can very quickly see the distributed tracing data in my environment. What this will actually highlight for me are important things like the number of errors. So if there's errors in the trace, I might want to go investigate that or the latency of the request. So for example, this call took 1.6 seconds. Each of these boxes represent a trace. You can see them summarized here, but you can click into them to get rich insights from that data as well. So for example, now I have clicked into a single trace. This is one request in the Hot Rod application. And you can see the call stack as well as latency information for each of those calls and if there were errors. So here you can see the front end took that 1.6 seconds totally to compute. The front end itself took about 1.21 seconds to get the data out of my SQL and to return it back. You can see calls into Redis. And Red here tells you that an error has occurred. In this case, it looks like there were some Redis timeouts in this particular request to the Hot Rod application. And there's rich metadata associated with this as well. So when you instrument your application, you can actually tag it with important pieces of metadata. Things that you might include could be let's say local things to the app I'm running on like the host name. It could be things specific to the environment I'm deployed in like the data center or the region that this pod or application is running in. Or it could be application specific things such as calls that it makes, garbage collection time or anything that a software developer may find useful to troubleshoot availability as well as performance problem. Now, from a distributed trace, I mentioned that you can actually build a topology. So for example, here is what the Hot Rod app looks like visually and each of these represent a different service node and hovering over it tells you the call paths of them. In addition, it provides a DAG of that same exact infrastructure. So again, you can see here you have a front end which calls the customer driver and route microservices. The customer one is backed by a MySQL database. The driver one has a Redis cache in front of this. So again, very quickly I can visualize my infrastructure and as new calls are made, this service graph can be updated. This is typically handled by the spark or flink jobs that you're running from an analytics perspective to build these graphs. And then what's really cool is that I can do comparisons of different things. So for example, maybe I want to look at the longest running trace. I'll take this one and I want to compare it to the shortest running trace. Let's say this one and I can say compare traces. And so what this just did is it took two traces, it overlaid the data over it and it's representing what is similar and what is different between those two traces. So gray means they're the same. That means both traces contain that call path. Red means that one contained it while the other one did not. So in this case you can see the call that took longer actually had the Redis call. And that's the call that timed out. So in this trace I can see that Redis was causing the problem that resulted in this 1.8 seconds, whereas a typical call path is typically under a second, about 700 milliseconds. So very easy to compare what a trace looks like against another trace looking for differences. This makes it easy to do problem isolation and determine where you should be focusing your efforts going forward. So that's a very quick demo of the Yeager UI. I do want to show you some future things as well that are planned. And as I mentioned, I do have these commands in the slide deck so it'll be easy for you to get after the session. So graph visualizations, what I showed you with the Hot Rod app is really a basic app. It only had five or six microservices. And if you have a small microservices-based architecture, that will work just fine for you. But if you have a lot of microservices, hundreds or thousands, or if you have a lot of calls, fans inside of your traces, then visually representing them in a UI like what Yeager provides isn't always ideal. It can be visually complex to see that data. So if you have small traces, we can do a comparison. I showed you two different Hot Rod calls that was pretty easy to visually compare. But if you have hundreds or thousands, that may be a lot harder to see. In addition, while comparison of how the trace looks is valuable, what could be even more valuable is differences in the latency, the time it took between those calls. So let's actually see what that looks like. Here's a much more complex example. This is something more like an Uber, for example, where I'm comparing two traces to an EATS gateway. Again, the colors represent what's similar and different. Gray is the same, red is different in one. So we can look at this. If we compare the two traces and we want to know why one took longer than the other, we can see that they share the same structure. Visually they look almost identical. With that said, one of the traces is missing a whole bunch of spans, those represented in red here, while the other one has it. And then we can also see that more or less the calls are happening within a single node at the top of the charts. So the divergence here at the bottom probably accounts for why one span or one trace, one call, one request, took longer than the other. Now, what we might want to do instead is what if instead of comparing the structure, we compare the time of the spans and see if there are differences there? Just because it's red doesn't mean it's faster or slower in the previous example. Here I want to show a future roadmap item for Jaeger. The idea is, again, I'm comparing two spans where one took 2.74 seconds and the other one took 50% longer. And I want to answer the question of why? Why did it take longer? So I can see that there are some new spans. I might be curious are those to blame? Is that why it took 50% longer? I can also ask the question, is it just because there's lag, there's increased throughput for this request? Well, that's why it took longer. And so the idea here is instead of comparing a span visually from a structural perspective, I compare them from a duration perspective. And here if I look at duration, I can see very quickly that the durations are very similar even between the different nodes. I can see that nodes that are not shared, they don't really have a difference here. But I do see a few black and reds. It's a little hard to see in the chart here. But those are different from a duration perspective. And the idea here is I'll be able to hover over any of these spans, any of these calls, and quickly see, hey, this call took two seconds, or 1.7 seconds. Why did it take that long? This is problem isolation. I can now go to the service owner for this particular service and say, hey, why was your service slow? How can we speed it up? Because it's impacting the rest of the calls downstream. It's impacting customer experience. And there may be multiple services that are slow. And so you can see that very visually. Now, this is not available today. You can actually compare two span structures to one another, but you cannot compare two trace durations to one another. If you're interested in contributing to that, there are PRs and GitHub issues open to get that added to Yeager. Of course, if you're interested in learning more, the website and the blog are really great resources for that. In addition, there's a bunch of different ways to get involved. The GitHub repo is listed here. All communication happens in Gitter. So if you're not familiar with that, you can take a look at it. There is a mailing list where you can ask questions as well. Twitter is also very active, and there are bi-weekly community meetings that anyone is welcome to join. You can ask questions or get involved to become a contributor to the project as well. With that, I would open it up to some Q&A, and I've covered a few of the most common questions that I typically hear while giving this talk, so I figured I'll skim through them very quickly since I have a couple minutes. One is, where can I get more resources? Yuri just recently published a book on mastering distributed tracing. I definitely encourage you to take a look at that. It should be available to download or purchase. Overhead of tracing, one of the most common questions that gets asked is, is this going to slow down my application? The answer is typically no. The client instrumentation is very efficient. With that said, if you have a very latency-sensitive application, you should of course test this beforehand and make sure it's efficient for your environment. Gabby actually recently did an internship basically with the Linux Foundation, and she posted a Medium post, comparing the client instrumentation and the overhead that it introduces. She has some really cool graphs that actually show how much resource utilization was consumed. It's a great resource if you're looking for some benchmarking information. Sampling is a common question as well. I didn't touch on this because I only have 30 minutes to talk about Yeager, but in distributed tracing, it's very common to sample the data. Instead of consuming 100% of it, you only take a smaller percentage of that data. This is not as common in logs and metrics, but for better or for worse, it is in distributed tracing. Now, Yeager, the agent and the collector, has a cool feature called adaptive sampling. That means it can say, hey, the backend is overloaded. I want to dynamically tell an application to slow down and send me less traces. It's a really cool feature that is in Yeager today. The goal is to add that to open telemetry going forward, but it's not available in open tracing or open census today. What open census and open telemetry do offer though that Yeager does not is tail-based sampling. So there's two different types of sampleings in Yeager, head-based, which means it happens at the application level when the trace is to be initiated, and tail-based, which means after the entire trace has been collected. The advantage of tail-based is I can make a sampling decision with all the data. So for example, I can say only sample traces that contain errors. I can't do that with head-based sampling because unless only the first span would be known, if that one contained an error, I could collect it. But if a downstream span had an error with head-based sampling, I don't know that. I've already made a sampling decision, so I lose that data. And then finally, message bus, async, long traces. There are always corner cases when it comes to observability. We distributed traces. These are three of the biggest ones. The answer is it varies between them. Open tracing has this notion of follows from. So from a message bus or from an async perspective, I can use the follows from construct in order to basically link traces together. Open census and open telemetry have a similar notion of linking this data. But the visualization is still hard. To be fair, Yeager's primary focus has been for more real-time RPC calls, not for some of the longer running hours or day distributed traces. That would be possible to add if there was enough interest, of course, PRs are welcome. With that, we have five minutes. I would like to open it up to some questions. Yes? Exactly to the collector. Is it something that is acceptable or would it have any kind of implications on the architecture? The question was around, I don't want to send the client library distributed traces to a local agent. I want to send it directly to the collector or directly to the Yeager backend. Is that possible? So the answer is yes, that is possible, actually. You can send the client library directly to the collector itself. The collector can run with a database node. I wouldn't recommend that in production, but you can do it. For example, Yeager publishes an all-in-one. So that is totally possible. With that said, the best practice is to run an agent. Let me explain why. It's kind of important. There's a couple reasons. One is client instrumentation requires you to edit your code, which means you have to commit it to your repository and push it through your CICD pipeline. That's overhead for software developers. They don't like to touch it multiple times. If I run an agent, I can tell the client instrumentation to point to local host. Local host never changes, always local host. So no matter where I deploy it, anywhere in my environment, the client instrumentation just points to local host and the agent will work. If I point to a collector and it's in a different data center and I have latency problems or I wanna rename it or re-IP it, I now have to go touch my code. And maybe I'm in a polyglot architecture and I have five different languages. I now have to go touch it in at least five different code bases. That's overhead. So in general, the recommendation would be use the agent because having the ability to send a local host is very powerful. The other reason is because, let's say I'm running in Kubernetes, I can actually run it as a daemon set, a local agent to the host. Well now I can do aggregation to the host. The agent provides buffering and retry capabilities. If I don't have an agent, then my client instrumentation has to have buffer and retry capabilities or I might lose that data. Well that's more complexity in the client library. I mentioned that like latency sensitive apps don't want overhead, that's more overhead in my application that I don't really wanna manage. I'd have to do that for every language, Go, Java, C++, Node, Python, whatever. So ideally you want the client library to be as simple as possible and not have to have that logic and to be able to point locally to itself so it doesn't have to be reconfigured. That's some of the advantages of the agent why you should probably use it but it is not a requirement. Sure, so just a follow up question on that. So how do you plan or is there a capability where the agent can be plugged in in a Cloud Foundry based environment? In Cloud Foundry, I mean the agent can be deployed as a binary, sidecar, daemon set. It's a flexible deployment model. It's just a Linux binary. So you could deploy it anywhere. Right, so you mean to say that we can deploy this agent binary as a sidecar in Cloud Foundry deployments? I don't know specifically for Cloud Foundry. I used to work on it a long time ago and a lot has changed. I worked at it when it was at VMware, it's not pivotal. But I know you can deploy containers and I'm assuming you can deploy sidecars. So we did try that but apparently this as a sidecar, the binary as a sidecar does not really work with PCF or the pivotal Cloud Foundry, right? And there's an open issue on, yeah, there. Okay, let's think after this. I'd love to hear more about that and let's see if we can figure that out because it should be supported. Our goal is to make it flexible for any platform whether it's Kubernetes or Cloud Foundry or something else, it should work. Absolutely. Thank you. Thank you. Other questions? I have one more minute. Yes. We send out a request and then we get a response with a spend. But after we're introducing the service mesh, for example, these two, we think most of the things are dedicated by the service mesh. They record the request, they record the response. So we find that most of the things could handle the bias too now. So do you think it's a good idea we dedicate all things to these two? We're removing our record or you think some things should be handled by these two and some extra things is better to handle by our application. Gotcha. So the question's around service mesh and Istio is one example of that. Linkerd is another one. What's nice about service mesh is that it basically puts a sidecar into all of my application containers and that sidecar is already instrumented. So in the case of Istio, it's typically Envoy. Envoy is instrumented with Zipkin and Yeager compatibility so I can get distributed tracing data for free, if you will, directly from Istio. I highly recommend you leverage that. Now the question is, well then, do I need to really instrument my app? The answer is it depends, but probably. Here's why, two reasons. One is while I can get service to service communication because I now know that this sidecar talk to the other one, it's very hard to get inter-process spans. So all my function level calls, I wouldn't have spans for that. So I wanna instrument my app if I'm trying to track performance or availability issues in the function calls themselves. The second is even if you don't instrument your app, you still need context propagation because while Istio generates the span and trace information, it has to propagate that information through your app. So in the case of like REST-based RPCs, you still need context propagation in your application at a minimum so you can pass baggage and other trace information. Okay, thank you. And another question is about the message bus. We also have some communication with the message bus. Then the things must be transferred from the message between the message bus. I'm not sure you mentioned you integrated with the Kafka. They have maybe the transparent about the span, but if we're using another message bus, then which thing, how many extra things should us do for the tracing get a completely change? Yes, so from a message bus perspective, Kafka works, but other ones you'd have to add. The primary format of doing this through open standards, when the screen comes back, open tracing is one of them. Open telemetry and open census also have this notion of follows from or it sometimes is called links. You would have to make sure to pass that as part of your trace data and that way it could literally drop it off when the message is received into the message bus and then you have to propagate that through your message bus. So when it's picked back up, it tags that follows from or linked information when it sends it back out and that way you can make the entire trace. Okay, thank you. Thank you. I'm available for questions after we are at time, but if you wanna meet me over here, I'd definitely be willing to answer them for you. Thank you. Thank you.