 Okay, so yeah. Hi, I'm Josh. Hi, I'm Interpolar Manager for Distro-Tracing at Einstein IBM. Yeah, and today we're going to talk to you a little bit about open telemetry and observability and using the two of them together for healthier software. So today we're going to cover a few topics. First we'll talk about sort of observability broadly and how it differs from traditional monitoring and APM. We'll talk about the components of distributed tracing and how it's a different challenge than what we maybe used to coming from monoliths. We'll introduce the open telemetry project and its benefits and talk about why you might want to use it together with something like Instana. And we'll talk about the different ways of instrumenting your services for tracing specifically. So we'll talk about some automatic and manual instrumentation methods. And then finally, we're going to show you what you can do with all of that data in an observability backend like Instana and how you can actually use the data to gain insights into how your software is operating or failing and remediate it. So first, we want to just briefly talk about what is observability. I'm sure this is the topic that many of you have opinions about. I just want to get us all on the same page for this talk. So to us at Instana, this quote is a paraphrase of the original 1960s paper by Coleman on observability. So it's observability is certainly to understand the interstate of a system by measuring all of its outputs. It's very relevant to microservices because even though we can technically see into the communication between the various services in practice, the complexity of the architectures makes that very difficult. And so with traditional monitoring, you know, you're setting up monitors on known services and how they interact and communicate with your other services. The landscape is shifting too quickly for you to keep up with your manual configuration and how do you, how do you deal with that. How do you gain the insights that you need. And so with observability and especially at Instana we believe that to achieve this complete visibility of your application, you need automation. And so that's why we try to automate as much of the process of gathering telemetry from all of your various services. So it's a complete picture that adapts and changes as quickly as your application does. And that then in turn allows you to answer questions that you didn't know you were going to have when you set up your observability tool, as opposed to just being able to answer questions that you have thought about in advance, which is not enough. So I talked about a little bit about telemetry data. So Yinshi, why don't you tell us a little bit about the signals that we're dealing with. Yeah, so basically there are mainly three types of signals, one is metrics, basically measurement of your services captured in runtime. So you have logs like timestamps data, string data, unstructured on nodes, and then you have traces. Traces is really about give you full picture of how your systems works, and we will focus on traces in this webinar. Yeah. And I think, you know, like I said, like we said earlier, coming from a traditional application where you maybe only have one process running on, you know, one machine. Your logs are in effect your traces, you can go to your logs and you can follow, you can correlate the events happening in the software in the logs with the requests that triggered those events. So distributed tracing is really the art and science of correlating and user requests with processes inside your software, no matter how many services those calls without. Yeah, definitely so it's definitely we people start to do distributed tracing because we move from monolith application to more distributed like complex services, then you does not know like how it attacks in your system right that's why we need distributed trace. So this quote kind of kind of lays out what we're talking about. Specifically, the traces are made up of calls and spans I'm sure many of you are familiar with this terminology we're just going to review briefly. Yeah, so basically a trade trace is a series of calls and calls are like this page said communication between two services, it could be a TV request or database queries, or a message in your cute message queue, and a cause could be combined by spans like access spans, which is basically a request so recorded by your server, because it's enter the your service or exit span which is more like a client request it goes out from your service that's why it's called. access bands and if we combine those two span you will get a cause and also sometimes you will have like intermediate kind of spend like internally in your system, you know, in that service, you have made some, I don't know function costs or something except. Because it kind of an illustration. Yeah, yeah, so basically this is kind of a trace. So from like black box prawn view, maybe if you're a e-commerce like company, your clients make some requests, right, maybe, I don't know, add say item to their shopping cart. And when they miss that request it goes through your API's and the API's internally will cause some awesome also relation authentication to know if that's a user and right if they want to pay you will have to go to payment. You know, some database course and also call external third party external API's to know okay if that's payment is successful or not. So it's definitely a lot of like service or cause going on here. And that's why we need like this free tracing to give you a full picture, say if if customer cannot add their item to their shopping cart you will definitely want to find the root cause very fast and distributed tracing enable you to do this. Yeah, excellent. And so there, there are some components that we need to enable distributed tracing. Yeah, so they are mainly we categorize into three components. So the first one is like instrumentation of your application rights. You need to be able to generate those signals are in concept of distributed tracing like spans, a cause in your application rights to do that, we can use inside as auto tracer which will do it automatically for most cases. Also, also you can use some other SDKs like SDKs provided by Instana open telemetry. And then you need a collector to collect those trace data. It can be pool based or push based. And for that you have, you can use Instana agents or you can also use open telemetry collector. Yeah, by the way, Instana agent is also compatible with open telemetry collector so it can ingest open telemetry data. And then you need to back end right because open telemetry really focus on data collection part of observability to be able to process your data and analyze and maybe making some actions based on the data you need to have back ends. That's what Instana can also provide. Instana provide your whole ecosystem of your observability solution. Yeah, and I think this this highlights a really important portion of open telemetry in general, which we'll talk about in a little bit, which is the portability of the data so if your services are instrumented using the open telemetry SDKs or using open telemetry protocols, then you can just, you know, switch out the Instana agent for the open telemetry collector or vice versa or even run them both if you want to use the collector to kind of direct the data where you will. So that's that's really a huge benefit is, you know, you just you take care of instrumenting your services, and then you know that the rest of the process will be handled one way or another, whether you're choosing a vendor solution or an open source solution. Yeah. So speaking about service instrumentation there are a couple different ways that we can instrument our services with, I would say varying degrees of convenience for the developer. Yeah, so based on here with based on here we categorize them into three markets based on how automatic the instrumentation is. So first you have automatic instrumentation. It basically means you as like owner of the service you don't need to do anything basically like in some environments like Java, because it provides such a functionality of GVM to a dynamic attached to the GVM like from outside through agents. And we just discover it and attach to it and then change some bytecodes and yeah. Traces are generated. That's the first way. And the second way is what we call like semi automatic, like in some environments you still need to do some configuration, right, but not like heavy lifting kind of instrumentation. The rest is already covered by like in Stana. Some cases, maybe Python writes you need to basically specify as this environment variable auto wrap and it will do the rest. By the way, this approach, maybe not needed in Kubernetes environment. Because Kubernetes have this auto trace kind of webhook, right, you can have this mutation webhook already done those, those things need a great detail configuration thing for you. But in like normal host you still need to do some minimal work. And then, like, unluckily in some cases you still need to manual instruments, your applications or SDKs, typically cases would be like C++ which is a compiled kind of language. So you cannot really do that instrumentation in runtime. You need to add some codes there before you compile your codes to get your traces generated. Yeah, yeah, definitely. So, so I think for me the key differentiator with automatic instrumentation, and specifically with the Stana auto trace is that you don't, you don't even need to restart your running services. If you have a, you know, if you have a Kubernetes environment like you were mentioning in G and you install the Instana agent. It just, you know, as a sidecar container in the environment, it will detect all of your containers and will detect all of the run times such as JVM that that it's capable of attaching itself to, and you'll start seeing data. So one of our engineers did a test and installed Instana on a completely new host and started seeing the completely populated infrastructure map within about a minute and 20 seconds. So, so that's, that's very fast for Instana to kind of spread out throughout your environment and discover all of the things that they can instrument. With semi automatic instrumentation, it's still very easy. And it's usually just a question of changing, you know, a few lines of config, or adding a single library to your service, and that's, we'll give an example of that towards the end of this talk. Manual instrumentation open telemetry has come a long way and making this a lot easier. You're not worried about trying to emit data in, you know, OTP formats because for pretty much any language that I've had to work with. There's an SDK available from the open telemetry community. And, and usually instrumenting your, your code manually. You know, it can't find the code paths like it can with the semi automatic method, but really all you need to do usually is get in there and you know maybe add some direct excuse me decorators to your request handlers. You know, maybe some function calls to create new spans when you're starting an asynchronous request, but it's really a lot better than it used to be. So we're talking about open telemetry a lot. So let's talk about some more. What is open telemetry and and why do we care. Yeah, so a little bit about history. So in the telemetry is a merge of two opens a project one is open tracing and another one is open census. Open telemetry right now is a since they have incubating projects, and it aims to provide a vendor not your toolkit for harvesting your traces metrics and logs. Yeah, that's that's basically the hottest topic in open observability industry, I would say. Yeah, definitely. And so, so why would you want to use it. Yeah, so as we already talked before it's provide your like unified formats based on the open telemetry specifications. So you have like data probability among different vendors from and it can avoid vendor looking right if you you can just instrument your application use open telemetry and send it to whatever back ends, it would be right you can use open source base one or you can use commercial one like in style now. And it's, yeah, it's also like a common conventions across different technology and language, because right now people use like different languages mean a single system right by using open telemetry you will have like unified approach. So that's what open telemetry really shines to provide a standard way and to avoid vendor looking basically. Yeah, and I think I think it's really amazing, you know that the open telemetry project is able to achieve this. That there's such a unified vision throughout the entire community I think that's that's really kind of the promise of all open source right is that you could have this, this one way of doing things that is open and portable and then you have your choice of vendors to back that up. So it's really the best of all worlds. And I think another perspective probably my personal perspective, it's to have such standards is also great for like for the earth like different vendors already have like a similar approach but basically they are repetitively done similar things. But if you can have a standard ways and you can save some like energy even in, don't know, build your codes in different ways that's also quite eco friendly, I will say. Yeah, it's more sustainable right and it's, it's easier than trying to wrangle, you know, 15 different teams using 15 different exciting new technologies for their particular stack. It's just way better. Maybe you can save some energy effort to this so we can go to Mars. One day. Yeah. So the, the open telemetry is CNCF incubating, as you mentioned. It's fairly stable, you can see here, the only portion that isn't completely stable is the logs API and SDK. But, but we're using them, right. Yeah, yeah, right. So metrics are traces. We are instant agent can already ingest them and logs it will be supported. But since it's still in a unstable kind of phases right now, and the community is really awesome they really aim to deliver it this year or next year. But yeah, eventually, when it's stable, we will support it. Allow me to introduce my friend Stan. So Stan, Stan is our automation expert and intelligent assistant at at it's not not really you don't actually have to talk to a robot. I know that can be an annoying experience for some of us, especially, but but yeah, it's really just about it's about the automation and the intelligence that we can add to your data. So that you can get to the root cause of your issues quicker, and, and also improve the overall performance of your services and applications. Yeah. So auto trace is definitely like one of most important superpowers than half right in standard have. And what's great about so our goal for auto trace is really to minimize your efforts to let you gain the maximum observability in your services. So it can automatically discover your service right if if it's in Kubernetes if it's a normal host if it's in container and so which processes that it's that Java application or NodeJS Python, etc. And it will collect those informations and when we auto instruments your application, those information will be correlated as later in the demo you will show you will see that's like a full context we provide. And in Santa auto also provide you a automatically created dashboard right so you'll see okay how things goes, and you don't need to create your own dashboard but still you can customize it. And we also based on like I said since Dana is also backends for observability which open telemetry does not provide or not aim to provide. So we can utilize the data we get and do do data processing and analysis based on the data like trace data or metrics data. We can set some alerts for you automatically like okay this this traces it's it's gone for for some sometimes probably you have some host changes. And because in Santa also correlate is this information it can also also say okay there is a one host that's maybe full of storage or something like that. So you can get to your like root cause more quickly. Yeah, that's, that's what's in Santa auto trace aim to provide. Yeah, absolutely so kind of an example of the smart alerts and the anonymity detection that she's talking about in a previous example. We had an application with a messaging queue in between two services, one of them is processing user requests and the service on the other side of the messaging queue is handling rights to the database. So, you know, a little chaos engineering we take down the messaging queue, and instead is going to going to create several alerts for that it's going to notice that the requests on the front end service are failing or coming back with with the error codes, you know, however that front and service handles the message to being down, and it's going to notice that the back end service is no longer receiving requests it's going to notice that the database is no longer receiving the same volume of traffic that it was. And I kind of sort all of those issues together and and create an incident for me so that I can zero in on you know rather than having to deal with any one of those things separately. I can see the complete picture of all of these issues and kind of from those data points. That's enough for for me as a developer and also for for instance I kind of go, you know the problem is probably with this messaging queue that sits in the middle of all these things. And so I can get to that, you know, very very quickly and we'll show that in a little bit. So if auto tracing is so awesome, then why do we even need open telemetry, you know, well besides all of the reasons that we mentioned before, with with data portability and sustainability and and you know saving effort. There are a few reasons that you would want to use both of these things together. So the first one you mentioned open telemetry does not provide an observability back end. So you could self host something like Jaeger. You know, there are other tools out there, but none of them are really going to come with the intelligence and the expert knowledge that is built into Instana. And as far as I've seen none of them come close to the convenience of Instana as far as setup. Instana supports most technologies but as we mentioned at the beginning of the talk observability really requires that we have a complete picture of everything happening within our application. And so we want to make sure that no services are left uninstrumented and and open telemetry can be a huge help in that regard. We can use it to to instrument the things that that Instana hasn't gotten to yet that are maybe a little older or a little more esoteric. And then we can also use it as a migration path for legacy systems, as we're trying to bring them into our observability framework. And also like some companies they have self invent some framework right it's can we cannot know you have your sets so in that case you're probably also needs SDK to instrument your application. And so to kind of give you a picture of what that math looks like. On the left we have some of the technology supported by the Instana sensors. And on the right we have the open telemetry libraries. Yeah right so basically we for for those that could be technically there are no limitations will aim to provide your auto trace rights. Like I said we really want to minimize your efforts to get your observability rights in. In some cases like we talked about before like in C++ or us where you can have a kind of compiled language, you probably needs to do some manually instrumentation, and because you don't want to be in window locking, you can use open telemetry SDK. And also another example would be caucus right so caucus is really a hot topic in JVM areas, it provided it provides two ways to deploy application one ways, like normally a kind of bytecode and drawing JVM things like that. And another way is that it can use GraphVM to combine to binary. In the later case it will gain you some like super fast, super fast like loading time and warm up time will be low, but since it does not provide this attachment kind of mechanism normal JVM provides. We cannot auto trace that through our JVM auto tracing, then you'll also maybe you want to consider use open telemetry to provide the traces. Yeah, definitely. So, so right so using, you know, using the JVM you're firmly in, at least with Instana, you're in the fully automatic category with the JVM because we can attach to the bytecode and inject what we need to and follow those code paths to detect your endpoints. With the native binary, which in my experience is just so much easier to work with. You can build an open container image which is, you know, so easy to deploy just anywhere compared to, you know, Java jar. And it also like as you mentioned the runtime performance just in some quick examples that I ran, it was two orders of magnitude faster to process requests using the binary versus the JVM for the same code base. So that's great, but we're giving up that dynamic environment where we can just attach and discover what we need to. And so yeah, we would need to include some library but thanks to the open telemetry project we're still in that semi automatic category, we're not having to go in and you know decorate all of our request because we just installed the open telemetry library, the open telemetry library is, you know, specific extension for the corpus framework. And so it knows exactly what what methods need to be instrumented and what it needs to do. So setting this up as just an example. And, and one of the things that you can find on our website we should actually probably link to this I'll make sure that goes out. Our open telemetry robot shop so we have an example application called robot shop which is a simple, you know, make or services e commerce store using a variety of languages. And then we have a variant on robot shop robot shops little brother hotel shop. And so hotel shop is fully instrumented with open telemetry protocols. We're still using the Insta agent as the collector portion. So you can also run the open telemetry collector if you want to. But getting back to to corpus so so to instrument corpus, we install this extension. We install the Insta agent which I'll show how that process in a little bit but it's a one liner. And we modify one config key for our Insta agent, and it's not it does support updating config the get. So this is very easy to do. If you have a you know if you have a lot of hosts that you want to enable this on setup kit ops and take care of that. And then we just need to, you know, unfortunately we are going to have to add some configuration to our corpus application and in this case we will have to rebuild it and redeploy it. We can't just, we can't just do this in place on production. But this is, this is a very minimal change to have to make to an application to get full full tracing of all of your endpoints. It's that easy. Okay, so let's let's show it time for a live demo. Yay. Okay. Um, so I was still looking at the correct screen everyone can see hotel shop. Yeah, I can see it. Okay, excellent just making sure because. So, here is hotel shop running in one of our QA environments, you can kind of already tell just from looking at the golden signals here that, and this is actually a good time to point out. So it's Donna, you know, we, we really like to highlight these Google S3 golden signals and you're going to see these and a lot of use these three, these three signals, right, which is throughput error rate and average latency. So, this is a demo environment, it's probably under provisioned. And you can kind of see that just, just right from this dashboard. One thing I do want to mention about this dashboard is, I didn't build this, no one built this, we installed the install agent on our hosts, and based on the data that was coming in, we get this dashboard. We build custom dashboards if we want. But for the most part we don't need to, and it's not just about this one dashboard that we're looking at here which is in an application perspective, which we'll talk about. It's all of the dashboards throughout and starting you'll see me clicking around from, from, you know, services and entities. They're all automatically generated, and they're, they're designed to give whatever your team is whatever your particular slice of the service is that you're interested in. Whether it's, you know, SLIs and SLOs for a specific service, whether it's infrastructure and overall health of the hosts. There's going to be a view for you that gives you the information that you need. I promised that I would show the installation so let's get into here. And, oh no I'm in a demo environment so I can't show that, but it's a one liner. It's, it, if you go here in an environment where adding new agents is allowed, you'll see the list of supported technologies and types of hosts that we can install on so you can see these Docker right your containerized environments open shift VM where Tanzu, and we'll have specific instructions for each one of those but it's, it's just one command that you're going to copy in all of these cases. And then we can also support, you know, installing the agent directly on your hosts, whether it's Mac or Linux, or Windows hosts. In the case of Mac hosts, like maybe you want to run the install agent and your local development environment, and that's something that we'll be talking about in the future as part of shift left, but really there's there's no limit to where you can install it. Installing it in a containerized environment is incredibly easy. We have Helm chart or operator or whatever your chosen, you know tool is, we can provide that. And, and if you do it through the dashboard that one liner is going to be pre populated with your, your keys that are needed to connect to your tenant. So getting back to the actual data. So this right now what we're looking at we call this an application perspective. And what that is is so go to the list of all of our applications. And it's really just a collection of filters on the services right so if we look at this tab this is these are all the services reporting to this environment. And so, I'm obviously not interested in all of these there might be, you know, services here that are completely unrelated to my application, or that just aren't the responsibility of my team and so I need a view that excludes those and includes the ones that I care about. The application perspective comes in, and we can define this with just a couple of filters. So in this case, all we're doing is filtering on the Kubernetes cluster and anything running in that cluster is going to come into this application perspective. So, so getting back to the application perspective. You can see we have our golden signals we have our top services. We also have our time perspectives over here. So this is a really important part of your debugging and, and troubleshooting journey in it's Donna. So, you know, from experience that a lot of the times when you're trying to debug something, you might be correlating, you know, logs from one service metrics from another platform, like from atheists, and, and maybe you have traces somewhere else. And so you're kind of trying to zero in on okay well this is the time window, when this problem occurred. You know, it's just that we're having the problem. This is the resource utilization on those posts and you're just kind of stuck in in tab hell. You know, trying to trying to keep all of this data in sync across these different services and zero in on. Well, what is the issue in this in this giant haystack with inside application perspectives. You can choose any timeframe. You can also use releases if, if your CI tool is reporting releases. And this is going to be sticky as you navigate throughout the Insta dashboard. So you know if I choose last 10 minutes, that's going to stay as I navigate to other pages through to other applications, which is fine for just this, but where it really comes in handy is say I want to zoom in on. Well, there's an error spike right there. Let's zoom in on that and see why that happened. Then I can do that. This is going to bring us to our unbound analytics feature. So this is sort of your place to, to filter through and, and you know you can see we have all of these manual filters on the side that I can use to find the specific calls or traces that I'm interested in. And you can use this manually, but you can also get to it the way that I just did by by being on a screen choosing some data, zooming in on it and getting the filters pre populated for you so you can see exactly what you care about. Now this environment is not that interesting because we actually I think our low gen is broken so I'm going to switch over to our regular robot shop, which has a little bit more, more stuff going on. We'll go to the last hour, the time deal. So you can see there are a lot of JS errors happening here. But nothing that's really causing oh well that's interesting there's a there's a huge spike in the load time right there. So let's zoom in on that and see if we can figure out what was going on there. Looks like the issue got resolved by now but we'll pretend that it hasn't yet. What's happening. Oh yeah. Okay, so let's let's specifically look at traces during this time frame. And let's see the error. Okay. Erroneous traces 100% that's smelly. So let's see what's going on here. So we can see all of these requests are timing out after about two seconds it looks like. And I'm going to zoom into one of them. So, so here's a view of a particular trace. This is a problematic trace in this case. We can see the logs associated with this trace. We can see the calls we can see the end user information from, you know, the end user monitoring which is a JavaScript tool that you can stall on your front ends. And there's also a tool for mobile applications so you can get this this information from native apps. And, and this is going to be correlated with any of the back end traces that can be associated with this end users requests. So this is really a huge help in in figuring out exactly what's going on and who's affected. And then scrolling down we can see this Gantt chart, kind of similar to the one that we were looking at earlier. So this is is our complete trace and we can see that it's terminating in this connection failure. Let's open that up and see what's been on here. Oh, okay. So it's failing to connect to the database. And yeah, that's, that's probably at this point we kind of know what we what we need to know. The database has a 100% error rate. Either there's a catastrophic failure or there's something malicious going on. But either way we know what we need to do next which is get this database service back online. So the traffic can continue to flow. Now that was a troubleshooting journey where we kind of saw something, you know just through through one of our workflows we saw something anomalous on a graph and we zoomed in on it to try and find the issue. But you'll notice Instana actually already knew about this issue before we saw it, and was ready to alert us if we had any kind of alert and channel setup. So, like I said earlier, you know you kind of got Instana it's noticing all of the changes that happening in your environment. So this could be things like a configuration change, or it none happens during this time window but a configuration change a service coming online or going offline. Anything that's going to affect the Instana agent, maybe a new container gets deployed. Anything like that. Then, you know, as those as things go outside of the normal band of healthy, it's not going to raise issues. Now these aren't necessarily actionable. But we want to collect all of them and make sure that we have all of them, because there might be a bigger problem that these are going to be part of the picture and figuring out what that bigger problem is. So we want to make sure that we have all of the abnormalities logged and tracked for you. And then we can kind of sort those and see okay actually we do have a big problem. We have an incident occurring a sudden increase in the number of erroneous calls. Okay, if we scroll down to the related events we can see, Oh, well the root cause of this was a signal on the my SQL service. So again, you know, this the second path getting through to to this point took me two clicks. I could do that in 30 seconds, I can see okay. My SQL services the problem signal that that's, that's bad that's probably malicious actor so we need to do something about that. And that is sort of the power of the both the navigation and the intelligence built into Instana and what we can do with your traces. Another important thing to mention is that Instana does not sample traces. So if you're using an open telemetry collector, you can, a lot of the open telemetry libraries will have a configuration variable for your sampling coefficient. It's going to be up to you if you're using open telemetry, but I would leave it set to one. And that's how the Instana sensors operate is just no sampling whatsoever everything gets reported, which means any communication between any service internal or external is going to be detected and and will become part of the map. What those zero, what that zero sampling enables us to do is create the real time dynamic dependency graph. So I'm going to go back to back to regular timeframe that incident that we were looking at has been resolved. And this environment is set up to sort of deploy that and then, you know, roll back the deployment because of the incident being automatically And that's another thing that you can do with Instana is you can combine our intelligence with automated tools so that some of these issues can get resolved with minimal or no human interaction or intervention. But getting back to our application perspective. So here we have the application dashboard that we started at. And if I click here to dependencies, we're going to see something generated by that dynamic graph that I was just talking about. So this is all of the communication happening inside the robot shop application within the last five minutes, we can see all of the calls we can see all of the services we can see all of the databases. None of this was configured, nowhere did I have to go in and edit some YAML file and say oh this service talks to the service and the service is called this. This is all just automatically detected and and built for us and it's updated in real time. So as services come offline or online. This is going to get updated. And this always up to date graph is what enables us to do things like if you if you zoom in on a specific service, like say the cart. We can see the complete stack of for this service so we can see the application, the endpoints that it provide the application that is a part of those are the filters that we talked about at the beginning. We can see the four endpoints that it provides. We can see that it's running in a Kubernetes cluster we can see all of our Kubernetes metadata that we could ever want here. We can zoom into any of these entities and see the specific configurations for them. And then we can get down to the infrastructure level and see the specific hosts that these pods are running on. So that's, that's the power of the dynamic graph is that we can sort of correlate all of the activity within your application with the related services and infrastructure from the front end user all the way down to bear metal. And speaking of bear metal, the last thing I want to show you running at a time there's there's a lot to show. So this is our infrastructure view. So kind of coming zooming out from the perspective of an application developer to, you know, ops or an SRE. So this is kind of an overall view of all of the infrastructure reporting into this host. And right now it's sort of it's separated by zone. There's custom defined zones, you can see we have our, our hotel demo, we have our robot shop demo. But I can also change this and I can, you know, change the perspective to be by container instead of by host so you're all of our containers running in those zones. And then I could, for example, say, Oh, well what Kubernetes namespace is, is the container running. And so I can just kind of sort and filter this view. Going back to the host of you. I can zoom in. And you can see that each one of these towers is is a specific host. So these are three Linux hosts running this Kubernetes cluster. And I can kind of mouse over them and see all of the services is as pancakes they're kind of difficult to see but you can see all of the processes and all of the containers that those processes are running in. And I can open up the dashboard for any of these. I don't want to make sure that we have time for questions. Oh, I do see that there are questions. Okay. So switching gears, and I'll leave the demo open as we answer some questions in case there's anything else we want to show. I guess that enabling open telemetry for son agent will consume more memory and CPU at the agent. How much impacts will we have at the agent enabling this option in she can you answer that. So we definitely aim to minimize the impacts when we design the agent so for open telemetry to enable this it's just expose a endpoint right so it should it would be minimal. So still have concern you can like you like, I think you mentioned it's in Kubernetes right you can limit the resource for agents like home, many units of CPU it should limit and how many unit of memory. You can limit sets in Kubernetes. Yeah, so should not be a problem. Our agent is very lightweight and you only need one instance of the agent per node. So depending on the size of your nodes it's it's it can become an almost unresolvable impact on. Yeah, I think that also answers him a new air question right it's installed as a demon sucks. Right. Yes. Okay. So Zia asks, does open telemetry or install cover overall asynchronous workflows, e.g. an end to end order fulfillment. How an order performed in various stages. Yeah, so I think it's really depends on which framework it is right so it's it's really case by case because there are tons of frameworks out there. And if we say okay we support this frameworks and if you have asynchronous handling there we will still support the correlations there. But yeah, it's really depends which framework you are using. Yeah, and I would say, you know, for for some of those asynchronous tools that you might be using in there like Kafka or MQ we handle that I think out of the box correct engine. Yeah, and open telemetry is just going to require some some minor instrumentation. Yeah, you define the beginning and ends of your spans in your asynchronous processors, and you may need to put some trace context messages in order to get that to your downstream services. And since there is another one, Raina, Kamala, Instana, trace contains Instana and open telemetry spans. Yeah, that's a good question. We support it. You can limit and mix it and it will get correlated but there are some limitations because open telemetry internally it use W3C trace context and it's only defined for for HTTP calls, right. And if that's message queue there are no defined thing, but since then we support correlations through some message queues. For more detail just check out documentation. But we definitely support the mix of Instana and open telemetry spans. So Zia also asked does open telemetry and Instana handle any time synchronization or clock queue issues in low latency. I'm not sure that I understand that question. Yeah, I guess that's like some low level thing. I think it really depends on which runtime it is. And I mean if you have further questions like how specific, if you can specific question later or get in touch with us. We can get our engineers for SME to answer those questions. Yeah, definitely definitely reach out to either one of us and we can put you in touch with people who can answer that question. That does remind me for anyone who thought that my demo was too short. We do a live demo every Tuesday and I believe that the link will be in the chat so I definitely encourage all of you to sign up for that if you if you're hungry. Okay, so the next question is, which SDK we need to use for NodeJS REST API to have open telemetry traces and metrics deployment. So if you want to use the open telemetry SDK just go to the open telemetry project. They have a list of supported languages so you just click on NodeJS and they would show you the preferred packages to use. But Instana Auto Trace also works great with NodeJS so you could skip that and just use Instana. Yeah, and another one is Instana Auto Trace for open telemetry. I mean if that's, I guess that's similar to the previous run right so you can mix Instana Auto Trace and open telemetry. If both Instana and OTEL is enabled on a service, will Instana agent avoid instrumenting something that's instrumented by OTEL and just use the OTEL data. You know the answer to that. I believe the answer is yes that Instana will leave it alone because we do detect agents from some other observability tools and and halts our operations if those agents are protected. Now that's a that's a configurable thing though so if you're trying to do something really specific, you can probably override whatever the default is. Yeah so agent provides configuration like in a YAML file or you can use it's GitOps so just disable some Instana sensors. Okay, how is open telemetry different than Grafana? So Grafana would be one of the tools and organizations contributing to open telemetry and that you would use open telemetry with. I know Grafana provides a dashboard tool that I've used quite a lot. I think Instana dashboards are a little bit more convenient than you know editing YAML files to create a dashboard, but maybe that's a matter of preference. But yeah you would use Grafana sort of as the back end similarly to how you would use Instana and you would be still using open telemetry if you wanted to. Yeah so open telemetry is really scoped to data collection. So it's just let you create those data but how to illustrate it and how to analyze it. You need to another tool. Okay and finally we have a question about Instana self-hosted in a cloud platform. So I assume this means the Instana back end is being self-hosted which is something that we do support, we do support on-premise installations in the cloud or even in air-gapped environments. We also are very excited to have open shift support for Kubernetes for on-premise coming up in the next quarter so. Yeah. Do you know specifically where the logs end up? It starts internally, you don't need to manage separate things to do logs. Yes. It is, I believe it would be elastic search correct but it would be, you wouldn't have to manage it, it would just come with the Helm install for the back end. But you don't need to manage any databases or data storage. It all comes with the back end installation. Yeah so those three signals are native supported by Instana, metrics, traces and logs. Do the SDKs have an impact on our application forms or service latency? Not that I've ever noticed. No. We're talking about if there's any difference, it's immeasurable in the end of seconds. By the way, your English is not that great. Yeah, agreed. Okay, so when registering an agent, the console finds out or do I need to register on Instana too? No, so the one liner that I couldn't show you, it will include your agent key and then Instana back end will use that key to associate the agent installation that you've just done with your back end. So you don't need to do anything to tell it about new hosts other than install the agent on the host. Okay, we have about five minutes left. Are there any more questions? Okay, well thank you so much for coming everyone. I hope that, oh here we have one more question. Okay, nope, never mind. Yeah, I hope that you'll check out Instana. You can take it for a spin. We have a 14 day free trial or you can just click on the play with button on our website and get into an environment very similar to the demo environment that I showed here. We also have our live demos on Tuesday. And if you're going to any conferences this fall, we wait see you there. So thank you so much. Thank you. Thank you so much and she and Josh for your time today and thank you everyone for joining us. As a reminder this recording will be on the Linux foundations YouTube page later today. We hope you join us for future webinars. Have a wonderful day.