 Let's start our session about troubleshooting.net application. Welcome, everybody. My name is Sergey Mityukhevich. I'm a cloud engineer from Alturas. I'm sure you have probably already seen this slide. But for those who didn't, please review it, because it contains important fire announcement. But while you're reading, let me very quickly introduce myself. So I started as a .NET developer 70 years ago, but then I moved to DevOps stuff completely. And I can claim that I have seen the platform from both points of view as a developer and as a DevOps. And my current topic is closely related to both of those experiences. So today, we are going to talk about efficient ways to troubleshoot .NET application deployed to Cloud Foundry. I can split the topic of troubleshooting .NET application into three broad areas. So first of all, we're going to talk about how to utilize logs. We will see how we can use structured log into efficiently log messages and help platform to aggregate those logs and present them for us. We will talk a little bit about monitoring as well, how we can use monitoring to notify us if something goes wrong with the platform. And finally, we will talk about debugging and remote debugging of .NET application deployed to Cloud Foundry platform. A few important things that you need to know about logging in Cloud Foundry and this is related to most Cloud platforms. So first of all, your application is not supposed to log anything into log files. So instead, you should print output of your logs in the standard R standard out of your process. And also, it is important to know that a platform is responsible for aggregating and handing logs for you. So later, you can consume those logs and utilize them and use some log aggregation frameworks that I'm going to show you a little bit later in my presentation. Another important concept that is related to logging in Cloud application is called structured log. So what does it mean is that usually you don't just print some string message as your log message. You attach some important information to each of your message. And this allows later to very efficiently sort your messages, filter your messages, and drill down into an error if you had one. And later, I will show you how we can utilize facilities provided for us by .NET Framework to implement such kind of logging. Here is the code that does this for a sample ISP.NET core application. So the top code snippet is responsible for configuring logger. So here, I'm using SeriLog library to implement structured logging. But that's not as important because almost all log libraries, like nLogs, for example, or log for net, implement similar kind of features. So here, we configure default log level for standard libraries because we don't want to be overflown with messages from Microsoft namespace and system namespace. But the most important line here is the one where we configure that we are going to write our message to console. So that's essential for cloud application because, as I said previously, we don't want to log into files. We just output everything to console. And also, we are using JSON formatter. So if you use JSON formatter, this allows you to wrap your messages in JSON object and attach fields or text to each message that later can be used to search for logs. And second code snippet is configuring the application itself. And the most important line here is the one that says use SeriLog. This is extension method that actually applies previously configured logger to current application and allows a speed.net core app to use this logger. Next, after we have our logger configured, we can try to utilize our logger. But what we really want to achieve is to transparently add those kind of texts that I was telling you previously to all our messages. We don't want to attach request ID and user ID each time we log in something. And that's where middleware comes into play. This is a very useful method to implement some functionality before and after your request is processed. So here you can see that we are writing some string to log before the request, after the request. And most importantly, we use beginScope method of isp.net core logger. And this method allows us to create custom scope. What does it mean is that it creates a new field that is going to be attached to each subsequent message in this scope? By default, isp.net core creates scope for each request and it attach such fields as request ID. We will talk how to very efficiently use request ID later. But nobody prevents you from creating custom scopes. And it's not necessarily should be the case that you should wrap only request into custom scopes. You can wrap any other functionality into custom scopes as well. So if we are talking about custom scopes, a few important things that I want to highlight. So first of all, scopes allows you to make your logging transparent for the code. So you don't attach user ID and scope ID each time you log in something. Instead, you use such facilities as custom middleware or custom filters to do this job for you. And secondly, it ensures consistent logging format so you know that request ID is going to be attached to each message. And then you can utilize this when filtering for your messages. And as I said previously, you can wrap any other kind of functionality, for example, request to database transaction. Database transactions can be wrapped in a custom logger scope if a lot of work is going to be done during the processing of this transaction. Let me also show you a few fields that I usually consider to be important when talking about log. So first of all, if we are talking about custom scopes, so usually we attach ADs that later can be used to filter our messages. The most important one is request ID, but also it's important to attach user ID, maybe thread ID if you are doing some multi-threaded processing during your action method execution. But also, those fields are used to filter your messages. But also it is important to trace everything that goes inside your application and goes outside it. So what do I mean by that? So it's a good practice to log the content of your request because later you can reproduce this request. And if you rely on custom headers, for example, you can log completely the whole request. Also it is important to log everything that you send to external system, and you can consider your database as external system. So you can log your SQL queries, and later you can analyze performance and find bottlenecks when analyzing those queries. And if you talk to some external system, it's also useful to log this. And the main idea why I pay so much attention to this is because this login can be implemented in a consistent way using such techniques as I showed you previously with custom middleware. For example, you can use your ORM to attach some login information before each request and after each request. OK, at this point of time, we have our login configured for our simple application, and it brings something to standard output and call finder is responsible for collecting those messages. Now we need to talk a little bit how we can utilize this and how we can filter our messages and search them. So in my presentation, I'm showing you screenshots from applications that is called Kibana. And here I'm using a very popular stack that is used very commonly with Call of Foundry that is called ELK, Elastic Search Log Stash Kibana. So this stack is responsible for taking messages from Call of Foundry, storing them. And later, Kibana, this is the screenshot from this application, provides you a way to see your log messages. And here you can see that we have three messages, something that two messages from our custom middleware. And the important thing here is that our custom field is attached to all of the messages. And we also see that request ID is attached here. Very briefly, I want to explain how ELK stack works. So Call of Foundry by itself doesn't provide you any possibility to efficiently search for logs. Instead, Call of Foundry just provides you a way to grab all logs from the system. And it should be a responsibility of some other system to collect those logs, store them, and provide search and filter possibilities. So the application that collects logs from Call of Foundry is called Firehose Nozzle. Then logs are sent to intermediate buffer. Usually we use radius for this. Then there is a component that is called Elastic Log Stash. So Log Stash takes JSON strings from an input. The ones that we have been logged previously and converts them to an object. And those objects are stored in Elastic Search. And later, Kibana can talk directly to Elastic Search and take log messages out of there. This is very briefly how it works. So now I want to show you how we can use these to search for logs in case if something happens. So for example, let's consider a very common scenario. We have an error in our application. So first of all, we find our error message with stacked rays in Kibana. And then we can use request ID field attached to this message. We can use this request ID to filter all messages that belongs to this particular request. Then we can analyze what parameters have been sent to us. We know which user was in context of which user our request have been executed. So we have basically all necessary data to fully reproduce the request if we need. And usually this should be enough to troubleshoot any kind of error. And from my point of view, a good way to say that, OK, I'm doing login right for my platform is that if you can troubleshoot any issue just by using logs, then you are really logging enough information. And you are really logging information that is necessary for you so I'm not overflowing with some unnecessary information and that doesn't help you during debugging your application. Next, I'm going to talk a little bit about metrics and how we can use them in Cloud Foundry and utilize metrics from .NET applications. So if logs are mostly used to troubleshoot some issue, like in the examples that I discussed previously, we have an error and then we use logs to troubleshoot the issue. Metrics usually are used to send a notification that something is wrong with your system. Like disk utilization is too high or memory, we are going out of memory or we're having too many requests and our application cannot keep up with them. By default, Cloud Foundry provides you a possibility to access basic metrics such as, for example, disk utilization, memory utilization, number of requests, average time of processing the request. And those metrics are very useful. And if we are using PCF, there are recent applications that is called PCF metrics. And those metrics are available by default. And you can just open the dashboard and see how disk utilization changes over time. But that's actually very easy to use. But what I really want to focus on is how you can create your custom metrics. Creating custom metrics can be very useful because obviously, Cloud Foundry doesn't know anything about your application business logic. It doesn't know when your customer logs in intersystem or when some important event happened, when an error happened, and so on. So it's usually useful when designing your application to came up with a few important metrics. And here I'm showing you how you can very simply send value for some metric. And I'm using Graphite for this. So Graphite is a time-serious database that can store values for some metrics. And later, we can use other software to visualize those metrics. So let's see how it works, for example. If we, this is a diagram taken from one of products that is called Harbit that is doing exactly this. It hosts time-serious database and connects to Cloud Foundry and grabs all default metrics out of the system. By default metrics, I mean metrics that came from virtual machines where our Cloud Foundry is hosted. I also mean metrics came from the platform itself and from all components of the platform. But also, there is a way for your application, like I showed you previously, to send metrics directly to this database. This is very useful if you want to compare two metrics. For example, you want to see how disk utilization changes when a number of login attempts grows over time, for example. Or you want to correlate two metrics, one of which is application-specific and other is platform-specific. And such kind of solution provides you a way to do this. So all of them are stored in time-serious database. And we use an application that is called Grafana to visualize those metrics. So actually, on my previous slides, there's a sample of graph that can be built in Grafana to visualize some metric. And the last functionality that is very important is alerting. So if you have some metric, usually you want to be notified when the value of this metric goes beyond some threshold. For example, if CPU utilization goes beyond 80%. And usually, monitoring tools allows you to use logic to say, OK, when this event happened, I want to email to be sent to system administrators. And this interface shows you an example how we can create such queries and how we can use value of particular metric to notify user. So same as for logs, I want to talk a little bit about what kind of metrics I consider to what kind of metrics are important to be monitored. So first of all, default metrics that we talked about previously definitely are very important, container metrics, CPU memory, disk number of requests, request latency. If we are talking about application-specific metrics, from my previous experience, I can say that the most important one is database metrics. Because I have seen a lot of times situation when application works well in development, but as soon as we push it into production and put it under a load, the processing time goes exponentially. And that's where monitoring can help you because you can see, OK, request time goes up and we see the number of database requests. We can relate those two metrics. And if we see that a number of database requests also grows exponentially, this might indicate an error in your application. For example, one very popular error is N plus 1 request. When your application doesn't use SQL to get all information it needed in one round trip to database, but instead talk to database iterate over all items in some database. This is very common and it's very easy to see when you compare a number of database requests with a number of items in a particular table. And finally, the last topic for today is how we can use remote debugger in .NET application. So I want to tell about this on using ISP.NET core application as a sample deployed to a Linux container. And I will talk about Windows in the end of my presentation as well. So this diagram illustrates how this process works. So inside Cloud Foundry, we have a .NET process running. And in order for .NET debugger to be able to attach to your application, you need to use special software that can help you. So in case of Linux containers, we use VSDBG application that should be installed in the application container. And VSDBG is responsible for monitoring your process. And then VSDBG talks to debugger and actually is responsible for executing all debugger commands. In order for this to work, we need a special file that is called LungeJSON. And this file can be used to configure debug in VS code. It is possible to use LungeJSON in Visual Studio. But in Visual Studio, there is no native support for LungeJSON. So you need to type a few commands from command window if you want to do this. But if somebody is interested, you can connect to me after presentation, I can show you how it works. But from VS code, you just create this file. There is a native support for this. And what's important here? Important thing here is how we connect to remote debugger. Do you see that we use CFCLI? There is a command. So it's called pipe program. It's called CF. We use CF and then pipe arguments. So we use CF SSH to go inside container. Then we specify name of our application. And dash C means commands that we want to execute inside container. And VS code attach actual command that launch VSDBG and attach to running .NET process. In order for this to work, obviously, we need VSDBG to be installed alongside our .NET app. And here is a sample of a Docker file that can do this. So what is done here is that we are using two containers. The first one is used to build our app. We execute publish command here. And the second container uses output of the first one. And it runs our application. This is done in the last line. Here you see that we dynamically create URL to run our application. And we also install VSDBG alongside with some other packages that are required for VSDBG to run inside container. So after you've done something like this, this is rather simple Docker file, what you can do is to just use normal CF push to push your .NET application into a Linux container. And here you specify name of your container. You should put it in Docker Hub or your private Docker registry. And after you've done this, VS2Diacoat will show you a dialog similar to this. Here you can see all processes running on inside remote container. So you can just pick the .NET process, the ones that you are interested in. And basically you are done. You connect it to the application that is running inside container. So on Windows, when doing application dodges previously, most of the times what we did is to use some hacks to get inside VMs that runs on Windows. And usually we deploy a separate VM, debugger VM, in the same network. And then I install Visual Studio on this VM and use RDP to connect to this VM and just use native Visual Studio debugger to debug our application. So Cloud Foundry was not involved. And actually, this is not a cool solution because you need to access your operation team to help you with this. Just as a developer, you can just deploy VM in the same network to the same network where your Diego cells are host. But I advise you to attend tomorrow's keynote because tomorrow Piotl is going to announce support for native debugger on Windows. And that's a very cool feature. And you can use Visual Studio to attach to any remote process that runs inside Cloud Foundry. So I'm very excited about this announcement. I definitely want to check out how it works. So I hope they'll show us a live demo on this. And a lot of things that I want to tell you is that actually, sometimes it's forced to use debugger because you don't have any other way to troubleshoot an issue. But if you have to rely heavily on debug, especially in production, this might indicate that you have problems with your architecture and problems with your login infrastructure. Probably you are not logging enough information and you can troubleshoot it. The problem with debug is that it's a spent state. So actually, you cannot debug application, like live applications that works under loads that because as soon as you attach to a process, the process is stopped. And another problem is that we cannot use debugger for post issues. If somebody complains about your application and you cannot reproduce the issue, you basically don't have a lot of opportunities to troubleshoot it using debugger. So in such situation, logs are much more helpful for troubleshooting. And if you want to see how all this work in practice, you can check out my repository that implements the sample application that I have just shown. It's very simple default.net core application that has all those features that I have just shown you. So you can clone it and play around with it if you want. So basically that's it that I want to share with you. And if you have any questions, you are most welcome to ask them. I will try to answer all of them. You mean debug during development? So not log as much as possible, but instead I recommend to when logging something, just have clear picture in mind why you are logging this, in what case you want to log this. So I recommend logging such scenes that can't be logged transparently for your application, such as I mentioned previously, all request data, all database data. It ensures consistent logging format. But I cannot say recommend log as much as possible because I see a lot of times code like this, like one line of code and okay, we are calculating parameter x, then we are talking to database. And this makes your code less readable. Instead it's much more useful to have a wrapper that will log everything that you log to database and get rid of this login code from your business logic. So that's how I see a good debugger implement, good login implementation in the application. Dynamically turn what? Yeah, that's a perfect question. Yeah, so most of the loggers support doing this and one common way of doing this is provide you a logger configuration that externalize from your application. Instead of doing the things that I have been doing like hard code, the logger level, you can use JSON file to configure your logger levels for different namespaces. And also some logger libraries support configuration based on environment variables. So you can set environment variable, restart your application. If you have a lot of instances, then they will be restarted one by one. So there will be no downtime and you can dynamically adjust login level. But important thing here is that, okay, if we are logging a lot, how we can avoid impacting the performance of our application. So first of all, in your log aggregator, in such tools like ELK, you can set up a special process that will clean up all logs and you can set up policy. For example, all logs, older than one month should be deleted. Or for example, we keep error logs forever and we delete debug logs as soon as one week, for example. And this can be very useful. But in terms of slowing down application itself, well, I don't think it has very like extremely large impact on application because a lot of logger libraries introduce intermediate buffers. So instead of just writing each time to log file or to standard art, they log everything into file and then flash it periodically. So I never seen problems with application performance because we are logging too much. All right, anything else? Questions there, yeah? All right, thanks everybody.