 Alright, so I'm Scott Smirchak and I'm here to talk to you about bulletproof logging. I work for a company called SMART here in Kansas City and we write logs to help us with our applications. So most people start out looking at logs, something like this. They're blurry, they're scrolling really fast down the screen, and they're probably just text and you have to do things to help figure them out. So what do you do? You use grep. And you search for errors and you try to get clever. You search for the dates. You try to get the query time over four seconds. And this can work, but it's troublesome. So the next step, you use logstash and you can get these plain text logs into a JSON format. You can extract the fields and this can help a lot. And then you can start messing with them. But even better would be to go into your application and write out JSON logs directly. So we can get more detail like the request and its status and its URL and that can all be contained within the request. And we can see other things that are relevant like the hostname. So onto some practical tips. Don't do this. If you're logging in error, the log level is important. So use an error log level and put infos in the info. But don't forget about debug. Because debug is also very important. You may not have it turned on in prod, but you can use debug logs to provide much more context in your critical sections of code. And you can leave this on in your staging and dev environments to help you get more information. So time stamps. Time stamps are very important. Received time is not the same as when you logged it. So make sure you're pulling out the right time stamp. It can be very confusing if you have a whole bunch of logs all received at the same time. And add a bunch of context to your logs. When you're logging JSON, you can add a bunch of context like the inputs and the outputs of a function. This can really help debug later when you're not sure what's going on. And keep a consistent naming convention. If you log errors three different ways, you're going to have a hard time finding all the details of those errors. Same thing goes for ID, request ID, entity ID. Those are all very important to keep consistent. Has anyone done this before? Logging up error and it actually throws an error when you're logging it. Don't do that. Probably helped use a type system. So tracing. So you won't get this type of output from your logs. This is Zipkin. But using correlation to make sure that you know how your logs are all related is going to help you troubleshoot things in production. So if I know I have a request and it's coming down all the way through the stack and hitting the database and then going all the way back out, I need to be able to associate all those together. You can use the ID or correlation ID or request ID, but use something so that you understand how these are related. And then review your logs. So if you're in development and you're or staging, look at your logs, look at how they look, how they feel. If they make sense, if you look at this and you can understand what's going on, great. But maybe things like database result aren't really useful to you. So the message is important. If you can read this and understand what your software is doing, then you're going to be, it's going to be there for you in production when you're trying to troubleshoot an issue. And then you can start to build dashboards. So this is a gray log dashboard and you can start to see your requests and your exceptions and updates in real time. And you see pretty charts and graphs. And you can start getting these metrics that really help you understand what your application is doing. These dashboards with elk and gray log and Splunk can be a really great stand in for understanding your system without having a formal metrics application server. And then you want to move on to alerting so that you don't have to keep track of that dashboard all the time. So set up some alerts, set up when something happens, maybe that fires immediately, maybe it's when you get 10 of these messages in an hour. But identify what you need to track and alert on it. And then take care of your log aggregator. So if you take care of it, it can take care of you. Don't let it get messed up and start losing your logs because that would not be good. And then finally, we also need to make sure we clean up and keep it in good condition. So that's bulletproof logging and take care of your logs and they'll take care of you. Awesome, thank you Scott.