 Hello everyone. Welcome to the open source summit. This is Aditya Kanekar. I'll be talking about implementing observability in microservices running on Kubernetes. About me, I'm a cloud architect at CTSS and I'm part of the cloud practice team there. I'm also a certified Kubernetes application developer. I've also done certifications for Azure IoT developer. In my 15 years of experience rate, I have worked on different domains which include healthcare, digital marketing and IoT. I have done a lot of work on Kubernetes and distributed messaging platforms like Kafka and Flink. I'm an automobile enthusiast and my favorite show is Brand 2. So that's a short introduction about me. So without any further delay, let's start with today's talk. So this is today's talk agenda. We'll look at what is observability. So we'll do a overview on observability, the role of observability in microservices, tools for implementing observability, libraries, distributed testing systems, and we'll be specifically focusing on new feed on the cloud, that is open telemetry. And we'll have a followed by a demo for using open telemetry with Azure. So without any further delay, let's jump in. So everybody knows, most of you might be aware of this term observability. We'll just touch base on some of the aspects of observability, what is observability, and then we'll move forward. So observability is the ability to measure the system's current state on the data it generates such as logs, metrics, and traces. It relies on the telemetry derived from instrumentation that comes from end-to-end points and services in your computing environment. So when you host your application, it consists of different components. So the components will be your hardware, your hosting on a cloud infrastructure, it will be your cloud infrastructure, then there will be services, different cloud services, or any other third-party services, or maybe and your application. So every component generates a core of each activity that happens. So this data is the goal of the observability is to understand what's happening across all these environments and among the technologies. So you can detect and resolve issues to keep your systems efficient and reliable. So that's a high-level overview of observability and let's move forward. So I'm going to use an analogy here. So we'll see what is the need of observability, why do we need observability? So the analogy here and using it, we all must have seen the R-instrument cluster. The R-instrument cluster is the monitoring dashboard for your car. It gives you all sorts of information, maybe your oil level indication for your parking brake, your fuel level in your car, speed, engine RPM, etc. So this helps you in understanding your car's health as well as the speed you are going when you are driving your car. Now let's say your car's instrument looks something like this. Now you sit in the car and you see and you don't have any visibility into the car in the car system, whether your car is running fine, what are the stats of your engine or different components? You don't have any visibility, even you don't know how much fuel is left in your car. Now you start driving the car and you don't know what speed you are doing. So let's map it to your application. Now if you apply this analogy to your application, at any point of time, you should be able to understand what's happening with your application. And when we have micro-services, the entire thing becomes more complicated. You have service dependencies, then you have database dependencies or maybe your cloud service dependencies. So then tracking the issues, it's something, tracking the issues becomes very difficult. And you also need to know what is the health of your application at any given point of time, whether it is meeting the given SLA, whether it is meeting the KPIs which are defined by your client. If you don't have that visibility, it becomes very difficult to maintain such application and it becomes painful for the operations team. And normally we have different environments. You have their environment. You might have a QA or a QA environment, maybe a sandbox environment or a pre-prod environment or an end product environment. So maintaining all this environment becomes very difficult. And the other aspect of this is you don't know how much force you are incurring because if you don't know how much, if you don't have the observability today. So let's look at the role of observability in micro-services. So micro-services are dependent on database, cloud resources and other micro-services. So micro-services, when you develop micro-services, unlike, you know, singleton applications, singleton ET attitude, which we developed, micro-services are developed by different teams. And in a complex distributed system, it becomes very difficult to identify the source of an issue. Why so? Micro-services, one micro-service can depend on several different micro-services. Now, I will tell you a real scenario. So we were working on a platform. And in that platform, we had different micro-services. So and every micro-service was and our micro-service was dependent on three or four other micro-services. So ours was a cloud, ours was an IoT platform. And we were part of the data model, the data pipeline model, which we used to, which was the entry point of our application. So it's a typical IoT scenario. In yesterday, and then you, then you read the data for monitoring your IoT devices. Now, what used to happen is we, if something, since our micro-service was entry point, let's say, reading the telemetry data, which was received from this, from the IoT devices for the dashboarding purpose, so that you can understand the health of your IoT applications. The response time for that was very, very short because it was, it was meant to be monitoring dashboard. And response time was less than, response time was less than around 10 seconds. That is, that is also higher, higher from standard, but that was the max, which was, which was allowed. But we had a case where, where, where our API gateway used to time out. So here you, so the API gateway normally has a time out of around 30 seconds. And so that means our service request used to take more than 30 seconds. Now, when we started debugging the issue, we're not sure whether the issue is in our service or the dependent services. Our service was dependent on several other services to query the data. Now, in this case, we're not sure whether our service is at fault or other services. So the latency was unknown. So when we started, so then we, then we started debugging the issue. It was not very evident which, you know, API, which, which API or database call, what, what is taking time. It spent like hours and days to understand the issue. And then we relied on logs to find out which service was taking time. But this activity itself took too much of time. It was, if, if we have implemented observability, it would have matter of, you know, max, max, max 15 to 30 minutes, 30 minutes to understand the issue. So observability gives you the visibility in your system, so that you can detect the, detect and identify the source of the issue. Now, this is, this term is called MTTD. So there is a term called MTTD and MTTI. What is MTTD? So MTTD is the mean time or average time to detect an issue and MTTI is mean time to identify an issue. Now, this is a very time expensive operation. As I told you, one or two days when you are in production is very, is too much. And if it hampers your spend deliverables also, because your team is spending time, spending time debugging an issue, it used to, it used to take a lot of time. And at the end of it, we used to understand that the dependent, dependent services were taking time or there, and there were cases where there were also used to be cases like there can be cases and most of you might have said there might be a configuration issue, which has gone wrong. And you did all, all, all the debugging and at the end of it, you found out that there was an issue with the, with the, with the configuration. So this thing happened. So I did define, so this is, but this is a time expensive operation. So observability helps you in reducing this MPTD and MPTAC, mean time to detect and mean time to identify. So this, this leads to effective investigation and it minimizes obviously the downtime and other disruption to the end users. The other aspect of observability is in, in microservices, you have different applications running, running, running, running as a service and then part, right? And now when you have like 10, 20 microservices, how do you know what is what should be a Nord cluster site? This, this becomes very difficult to identify. So what, what, what normally we should do is, right? This is the practice we follow. We, we, for, for the single instance of your application, that is the port, one port, right? That is the single, that is the smallest unit of execution in Kubernetes. A single, a single instance of port, how many requests per second it can deliver under and when, and what is the threshold where it can deliver in the right, and your KPIs and ATLAs are free, right? For example, I want my API response to be, to be under, under one second, right? And I, my, my application should serve around 1000 requests per second and the memory consumption for the product, let's say to, to be the point to CPU or 4.2 of CPU time. Now, how do we arrive at this number, right? So for that, we need to do a performance range marking for that, for a single instance of that application. Then we'll know. So then, then we can know the threshold of the threshold of, of your application, right? How much memory your application use, how much, how much CPU time it uses on how much, you know, how many requests per second it can handle when, when it can deliver the right accepted response times. So all these numbers, observability can help you get all these numbers, right? And even a small change, right, can save you a lot of money. So for example, now my, I want my, my application serves, let's say 1000 requests per second with 266 can be a pram and 0.2 CPU time. Now, that is a secret. Now, let's say I want my, my client say we want to scale the users to 10,000 per, per second, that many requests your application should be able to, the application should be able to handle. Now it is very, now since I benchmarked my application, I know that my application can sell 1000 requests per second. So what I can do is I can, I can spin up around 10 or 12 instances, right? I can over-provision a bit, but for over-provisioning also you need some numbers, right? So I can spin around 10,000, if I spin around 10 rupees as of my application, I can easily manage 10,000 requests per second, but I can obviously over-provision. And then accordingly, I can size my cluster, right? So I hope this, this gives you enough insight of what is the, the role of availability in microservice. Now, let's look at a sample microservice deployment. So this is a very simple microservice deployment. If you add service mission and all, then, then it will become more complicated. However, this is, this is normally how, how, how it looks like, right? So you might have a API gateway where you host your endpoint, then you will, then your application, then in, in your Kubernetes cluster, you will have an increased controller, which will, which will talk to your API gateway. And then you will have services, the services will, in terms will, will talk to your port, right? The port, the port is, is, is your application, right? And you can have multiple duplicates as your port. And now the service can, so the different services will have different database. And here in this example, the service A is using cloud SQL and service B is using cloud big tables, right? There can be dependency between service A and service B. Now if you, and now if you multiply this number, right, you have more, we'll have, you will have more and more microservices and there can be dependencies between the microservice. So this is, this is a sample microservice deployment to give you a fair idea of how the complexity can increase, right? So this is a very simple deployment diagram, but it can be more complicated than that. Let's look at the three pillars of functionality. So first pillar I'm going to talk about is tracing. Tracing is basically a span that represents the execution of port. It provides you end-to-end flow of an execution path for a So what do I mean by that? So let's say I have a, I, let's say I have a API called a API called product service and then I have a authentication service. Now authentication service is shared by all different, different services to authenticate, to validate or authenticate and authorize the user. Now when I receive a, let's say my product service has a API called add to shopping cart, right? And my API, my, my, my, my particular service, product service uses a database, right? So which stores this item into the shopping cart. Now the first thing which, which first thing that the API would do is it will authenticate the, so let's say now as a user somebody's adding item to a shopping cart on, on, on, in the product service. Now let's say while adding the shop, item to shopping cart, he, he gets an error. Let's say find it, right? Or maybe a bad request. Now this is happening in production now. Everybody's clueless while now this issue comes to, comes to the operation team. They, they forward the issue. They are not able to debug the issue today. They forward it to the development team. Now you're not sure why this issue is coming because there is, there is no observation, right? Now your service is dependent on authentication service and it also depends on your database and there can be bugs in your code. Now how do I identify the source of the issue? So this is where tracing comes in. Tracing helps you in identifying the source of your issue, right? So if, if I implemented tracing, right, I can identify for, let's say the active shopping cart, when I, when I called authentication service, maybe the, and I get a error code from authentication service that it fits. So then I can understand, okay, the problem is in authentication service and in, in not my service so that I can forward the request to the appropriate team. So this is, this is what tracing is all about. Now let's look at the second pillar. The second pillar is, is logging. Now every, I don't have to tell you what is logging and more and everybody here must be aware of what is logging. So, but we'll still touch base on this. So logging, why do we need logging space? So let's say in production environment where you don't have a developer level of access, you need logging to identify the cause of the issue. So let's say something failed, in the last example, something failed. Now for with tracing, you can identify that something failed in the authentication service. Now, how do I know what is failed in the authentication service? So authentication service might be using its own database, it might be calling some other services, there can be different dependencies like that. Now how do, how do I identify that? So obviously tracing can also help you that, but if you want to go into more details of what's been, the logging can help you. Obviously the logs should be meaningful. So I normally tell, so I normally have, if I talk about it, right, I think this is a responsibility, this is a shared responsibility of the developer's team and the operations team. So developer's team, the responsibility of the developer's team is to log meaningful information and tag the information, tag the log correctly, that is the log level, should be tagged correctly. So this helps you, once the developer does this, the responsibility of the operations team is to only ingest the logs which have critical in nature, at least in products. You don't need all the logs, let's say you have 10, 20 microservices and then logs from each of the systems will be, will be like, will, will can run into one or two GB per day, right, depending on how much log your application generates. Now that's the huge cost to, to iCommerce. So if you, if most of you might, might be aware, might be aware or might not be aware that the cost for logging is not in storage, the cost is in the ingation, right. The amount of data if your application generates, right, if your application is generating around one or two GB of logs per day, then something like cloud, cloud watch can charge you around 0.5 GB per ingestion, per GB, right. So that's a huge cost if you, if you keep on multiplying and you'll have a big bill at the end of the month, right. So this is, this is a dual advantage when you, so there is a dual advantage when you filter out the logs. One is the developers have less log to play with. Obviously you can apply filters and all, but it is better to have, you know, meaningful information log, which can help you debug, you can use that budget to save from logging somewhere else, right. So, so this helps you, first thing you will have less log to play with, and then it can also benefit you from, from, you know, for first perspective. Now let's look at the third pillar. So the third pillar I'm going to talk about is metrics. Metrics is basically a numerical representation of data measured over time. So some of the metrics might, most of us might be aware of like CPU consumption from last five minutes. From API perspective, how average number of requests my application received in last hour, HTTP shaders, code received in, in last five minutes, right. So this can be, so these are some of the metrics, right, which can help you in, which can give you more visibility in your application by implementing observability. Now let's look at different tools which are available for implementing observability. So we have, so most of these tools are opponents libraries. So we have different instrumentation library, open senses, screen slew, open tracing. Now open senses and open testing are merged into one, one library and that is called open telemetry. So this is the, this is the latest library and most of, and it is getting a lot of, and it is getting a lot of, you know, it's a birth word right now. And most, most of the open source community, whether it is open source community or cloud platforms or, you know, even the third party providers like Sumo logic, then New Relate, Dynatry, they all support open telemetry, right. So open telemetry is, so open telemetry, so we'll, we'll talk briefly about open telemetry and we will cover that in today's demo also and we'll also cover the implementation part of how do you implement open telemetry for your application. Now let's look at the distributed tracing system. So what is the difference between tracing library and tracing system is you have, you use tracing libraries to export the telemetries which is required to implement observability and distributed tracing system helps you in, you know, storage, in, in, in storing the spaces and metrics and then there is a part where you can visualize and analyze the analysis. So some of the, so some of the open source, you know, tools available for this is the Agar and Triton. From cloud platform, it is Azure application inside that you have AW63 and Google Stackdriver. And from third party perspective, you have Superlogic, DataDoc, Honeycomb, Dinatrace, New Relic and many more, right. So these are the different distributed tracing systems providers. They also have, they are much more than that, right. Superlogic and DataDoc, DataDoc, the third party providers, they provide much more than this but this is, but this is one part, this is, this is, this is also a part of their office. Now, let's look at the open, now the, the focus for today's discussion is open telemetry, right. So I'll be talking briefly about it. So let's look at some of the features of open telemetry. Basically a set of APIs and SDK, which is used for, for, by implementing this, you can implement the traces, metrics and logs. It basically has this vendor agnostic, vendor, so it follows open standard semantic convention to ensure vendor, vendor agnostic data collection. So that's why different vendors are able to, you know, they will do whoever, whoever, you know, is, is complying with open standard and can, can use open telemetry as one of the, as one of the collector, right, for, for, for collecting the traces, logs and metrics. It also supports auto instrumentation with options for manual instrumentation. So the, the auto instrumentation, right. So once, so there are two ways of implementing open testing. One is you implement, you implement using, using the auto instrumentation. How, how does, so, how is that? So what you have to do in auto instrumentation and we will cover that in the, in the next page. You, you need to add, reference your application and then reference to open telemetry to your application and your application will be able to send the required traces, metrics, et cetera. And you can also, if there are any, you know, cases where you want to manually have instrumentation, if you want to manually, you know, phrase specific scenario, right, you can also do it through manual instrumentation. So auto instrumentation, you don't need much of a code change. It might be, you, you, in some languages, there can be code change, a little bit of code change by adding references, et cetera. In something like Java, it is very simple. You don't need to modify your code. You just add a Java agent over, you download the open telemetry jar in your container, and then just run it as a Java agent that will, that will, that will send, that will, you know, collect the required, the required telemetry, which is required for implementing open telemetry. Open telemetry also supports multiple destinations in parallel through contributing. So let's say, for example, I want to send data to Yager, and I also want to send data to Data, right? That is also possible, right? I can, I can send data to both these destinations in parallel. And it's basically a single collector binary that can be deployed in variety of rates, including as an agent or a gateway. So we'll, we'll come to this once we go to the implementation part. Now let's look at the language support for open telemetry. So open telemetry supports dot net Java, majority of the languages, right, which, which are popular languages. So Java dot net for Ruby. It also supports PHP, C plus plus, and, uh, then, uh, lang, Python, go, and you also, it's also supports it. And the other part is it also supports, uh, uh, in the client side application. So it supports JavaScript. So if your application is using JavaScript, uh, you can, you can, you can also, you know, implement observability by in, in your JavaScript application as well. But if you add open telemetry, it reference to your, uh, application. Now let's look at the, let's look at the implementation site for open telemetry. So there are two patterns which I'm going to cover here. One is the sidecar pattern and another is the service or gateway pattern. So open telemetry, so, uh, here is the example of the sidecar pattern. So what you do is in your app, in your application, like you, since we are talking about Kubernetes, your application is running inside a container. Now this container, uh, now, now in this container, right, uh, is, is, uh, is, uh, in this container, you need to add the open telemetry reference. Now, if you, if you talk about dot net for, or dot net for, or if you're talking about let's say Java, you just need to download the jar, attach it as a, and, and yourself. So the implementation differ for, for different languages. However, the, uh, the concept remains the same. So you basically add reference to the, to the container, uh, to, to your application and then you export the traces, right? Now, uh, since application, right, the, the difference between the container and port is you can have only one image running inside a container. However, in Kubernetes port, you can run more than one container, right? So this takes the, so here we'll take advantage of that. And this is called tied carpacker, right? So your, your application is running, your main application is running. And then you have a side by side. You have a different applications running in the same port, right? So here you can see we are running, uh, we, we have hotel collector, which is, which is basically collecting the traces from the application. So your application will, uh, hotel collector will host an endpoint where your application, uh, where the open telemetry library will send the traces or metrics to the open telemetry collector. So open telemetry collector, you can have, it's completely configuration driven and you can send the traces to different, different destination. So here we are, we have shown that we are sending to, sending it to the, uh, sending it to Yahoo. Now let's look at the other pattern. So this is the service pattern or a gateway pattern. So what do we do in this, right? So we have service A and service B here. And if you see, there is no side cut pattern implemented. The port remains the same. You don't need to modify any deployment type. Instead of that, what we can do is we post the open telemetry collector as a service, which have OTLP receiver and obviously it can have different receivers as well. There are different receivers available and then it, it has a Yager exporter, which is exporting it to the, uh, which is exporting to Yager. Now the problem with this approach is first is you need to maintain the collector service. That is one thing. So it becomes an overhead. The other part is you will not be able to get the specific pod metrics because the open telemetry collector doesn't have access to your pod because you have moved it out and running it as a different service. So specific, so when it runs, when the open telemetry collector runs in a sidecar pattern, what happens is they also collect some of the pod specific metrics, right? So this metrics won't be available when you are implementing service pattern. However, when you are running, right, in a sidecar pattern, you will need to understand that there will be a little bit spike in the memory usage for your application. So let's say your application is using 100 MB of memory, then you might, then you might need 100 MB more just to, uh, so do you need to account for that? But it is, but it will give you a lot more information than, than the service pattern. So that's why the sidecar pattern is recommended for Kubernetes. Now let's see the implementation part of it. Uh, so here I'm going to take example of a ASP.NET for, uh, for Subway So, uh, in Java, it is very simple. You just need to download the jar and, you know, set the Java to section to option to set to run, to basically export it. Now, ASP.NET for, uh, there are, there are some changes you need to do in your code. There are not a lot of changes. Those are very, very small changes. So, uh, the way ASP.NET 4 handles this is, uh, you have, uh, we have, you know, open telepathy library for HV.NET 4 for, uh, for, for, uh, so it covers, you know, uh, instrumentation, then different instrumentation, like if you want to see the process for HTTP calls happening from your system, then, uh, if there is an SQL client call, which is happening from the system, you also have a console exporter, you have actor exporter, et cetera, et cetera. So what you can do is here, if you see the definition, right, this is a startup, once you get a startup file where you normally have your, where you normally inject the dependencies in your service. So if you look at this, uh, there is, uh, uh, there is, uh, this, uh, resource builder section, right? And let me just highlight it. So I will just highlight this part. So this is the part I am talking about. So you have, uh, you basically add open telemetry system. Uh, then in, in, in that you, uh, you basically add the service. So this is the service name you should use, right? So this is where you would specify the service name, what type of instrumentation you want to do. If you want to add, so you want to have some 8.NET 4 instrumentation, you want to also have a SQL client instrumentation and let's say HTTP client instrumentation. So all that you can keep on adding to, uh, to this configuration. And where do you want to see instead, send the data? So in our case, we'll be sending it to the open telemetry collector, which is running in a side car. So that's why we have given local host, the 14174. So this is the port where the, uh, open source telemetry collector is running, right? And these are the references, right? You can find more open telemetry services in you get packages for HP.NET 4. Now let's look at the side car. So this is, so from the side for the side car pattern, this is, uh, this is, this is the deployment file for deployment file on the left, uh, deployment file on the left for the side car pattern. And then we have on the right is the open telemetry connection configuration. So before going into deployment, yeah, let's look at the open telemetry configuration. So what do we have here? We have, uh, so this is the configuration for the collector. And we have three sections can be retrievers, exporters and service, right? In retrievers, you can specify different retrievers. So you have OPLP retrievers, which follows the open telemetry protocol. Then we have exporters. So we have, we are using the other exporter here. Then we have service. So we have pipeline. Then, then we have traces and, uh, then we have traces. So let me just, so, so it's the highlighting control. So far, so forgive me for that. Let me again, you know, highlight it properly. So if you look at this thing, we have, we're going to go through this. So we have, we have exporter here and we are using Jagger for, uh, Jagger as one of the exporter. Then the endpoint for the exporter. So this is the endpoint for the exporter. And then we also are using data doc, right? So as I told you previously, right? You can export to multiple destinations. So here I'm exporting to Jagger and data doc. And, uh, for data doc, you just need a TIT and that's it, right? You don't need anything else. Uh, and maybe the reason is you want to, uh, export it to a different other than yours. Now, let's look at the service section here. I've got, I think I have got a hang of it, uh, of highlighting part. So, uh, on the service section, right? If you see, you can define different pipelines. So you have traces and metrics, right? So, uh, in the traces, the trace, the receivers we have defined here as OTLP. Then we have exporter. Here we have, uh, Jagger and data doc, uh, which we have defined here in the exporter section. Then we have metrics, right? You can also have, you know, processors, uh, here, uh, which I have not added. But the processors, if you want to limit the memory of your thoughts, that is also possible if you, if you add the processor section, and then you can have a similar section here in the processor where you can limit the memory of the, uh, memory of your application. So that is also configurable. So that helps you in keeping, keeping check, right? On, on, uh, on, on your memory limitation side, right? Uh, memory limit set. Now let's look at the deployment.yaml site. Let me again, you know, uh, remove this. So what do we have in, uh, um, so what do we have in the deployment.yaml? So, uh, as I told you, we are using a, uh, we are using hotel collector as a, as a sidecar character, right? So hotel collector here is, uh, is the name of the, so I've just given me any name you can do. I mean, it's a, it's a, it's a container name so you can give any name here. Then the image and what I'm doing here is I'm mounting this configuration as a volume, right? So you will have a volume section where you mount this, uh, mount this particular file in the volume and there will be a volume mount section which will mount the particular file in the, in the, for, for open telemetry collector. So here is the mount path for, for this thing. And then you have a collector configuration. So if you see in the argument, he has specified the configuration file here. So advantage of this is you don't need to modify your applications anywhere. You just need to modify your deployment configuration file. You modify your application once and the deployment file once. And if you want to do any changes to your configuration, you can just change the, come here and change the configuration here and maybe restart your application. That should, that should, that should take the latest, that will take the latest configuration. So let's, let's move forward. And now we'll be coming to the interesting part. That is the thing, right? So we'll be seeing. So I have two applications before moving forward. I have two applications. One is the patient service and another is the allergy, allergy service. So patient API is a patient API has, has a method for getting the patient, getting the, getting patient and adding patient and the allergy API has, is, is basically used for adding allergies for that particular patient. So let's, let's, let's, so this is, this is Yager. We'll come to this for, let's, let's just first, you know, go to the, let me share my screen. So this is the, this is the Yager UI. And let's, let's, you know, go to the post. So let me, let me share that. So I have postman, where, what do you have, where I have gate and post. Now, allergy service, I'm not exposed because I kept it as an internal service, but it is used only by the patient service. So what I'm doing here is I'm getting the list of patients so you can see the list of patients here. So, so that you know that this is happening, this is working in real. So everything is running in my Kubernetes cluster. I have Yager running on my Kubernetes cluster as well as the application site. All this are running inside, inside the Kubernetes cluster. Now, let's, let's look at the patient service. So the patient service here is, so this patient service, we, this is a post, a post service. This is, this is basically where I create a patient. And I also add allergies here. So if you can, you can add multiple allergies by separating, by, you know, you can have pharmaceutical allergies here. I'm not, so this is a sample application. You can have multiple allergies and this can be represented as an addict, but I'll just get it simple. Okay. So before going there, right, let's, let's look at, let me just bring up the terminal where I wanted to. So just give me a moment. So I'm bringing up the terminal window here. My, my Kubernetes cluster is running on my MacBook Air. So I'll just do a get for, get pause. Okay. So these are the two applications which are running. And I have the services for the same. And I've also set up an ingress controller. So this is, this is what my application looks like. So this is similar to the sample application deployment which I've showed you previously. Now there is a difference with dependency between patient API service and allergy API service here. So this is, let's, let's now look at what, what's, look at the, so I will, what I will do is I will add one more patient. So I will add the, let's say James, James Bond, add some allergies to the daily product. Now I'll just change the code to something else. Okay. So you can see that I got 200. Okay. And now let's do a great patient. So Mr. James Bond has been added as a patient. Now let's look at the traces for the thing. So this is my Jagger which is running on my machine. This is also working. This is also running in Kubernetes. So let me just find out the traces. So let's look at the traces for patient API. So these are, these are the, so let's look at the So this is the trace which I've received. You can see that my API is calling allergy API. So this is the trace I've received. You, you can see that my API is calling allergy API here. And it's called C time. Why is that? Because it is calling for three different patients. So that's why you see three different calls here. So you can also see all the calls. So I have made a great API call to you for this patient ID. So you can see here also on the allergy API, if I open this, I can also see the status code for that particular call and also see the open telemetry version, et cetera. And you can see that what is the method, et cetera, and the call section call. Now let's look at the post call. What I can also do is here, I can also filter by get method. So this is it. Let's do it by force. And I can also see the, right? So this is, this gives you a complete, so if something failed here, right, I would, I would see a difference. So that helps me identify the issue. The other beauty of this is I can also see a dependency graph. So my patient API is dependent on allergy API, right? So this gives me a fair idea of my service dependencies also because I'm able to see the dependency graph. Now we'll also, there is one part which I want to cover. We have a Kubernetes dashboard which I developed internally. And so this covers the tracing part and monitoring part, the tracing part. Let's look at the, this is a bonus for the people who stayed till the end. So this is, this is where I have, this is where we have a QPost dashboard. So I can also, so this is, this is the, so this is that what I welcome to QPost. And also it tells you different ports which are running inside the systems and the cost you are getting for that. So this is, this is one of the useful dashboard. Similar to this, you can explore different metrics. And, and, you know, you can also see the EBS volumes which is being utilized. You can also see the cost for the same. Similarly, you can have a dashboard for monitoring the, monitoring the Kubernetes application by exploring the metrics. Since we have limited number, limited time, I won't be able to cover the whole dashboard with you. But this gives you a fair idea. And one more dashboard I would like to cover here is the POE dashboard which we have. So it gives you, so this dashboard uses, uses, you know, QPeng to get the details for, get the details if you are applying, if your COCO Kubernetes cluster is complying with the security norms. So you can, you, so this is, so these are the, so this is, so these are the status rates of different, different checks, right? So it has paid, you can see they say it has passed this However, some of the state data space, right? So this kind of dashboard you can build by implementing observability in your application. And this is, so this is very useful when you are setting up application for product production. So that's it folks, I'm open for Q&A. And the source for the code which I've shown you is available in my GitHub repository. And the observability application I have taken from Dynatrace, they had a very good definition for observability so I have taken it from there. And so thank you, so give us your feedback and thank you so much for attending this talk and happy coding, thank you, thank you so much.