 Okay, so next up we have Irina. So she has worked with Python for a decade in multiple roles ranging from manual testing to QA team lead and Python development. She currently works at Cloud Linux. Hi, Irina. And she loves mentoring beginners and speaking about automation technology. So Irina, what else do you like in addition to well-covered tests and well-tested automations? So monitoring, of course. And I will talk about it. So different type of monitoring metrics and in this particular case, Sentry as a real-time error monitoring tool. So hello, everyone. And thanks, Europe Python, for having me today. And thank you all who I cannot see for joining this talk. It's very weird to talk to myself and not seeing anyone, but I will try. So the topic that I want to cover, as I already mentioned, is Sentry. And it is tracking software for web services and applications. A couple of years ago, Sentry's motto that they displayed on their main page was software errors are inevitable, and the cause is not. And I think it is a very good illustration of what the purpose of Sentry is. So it gives you a confidence that your service works the way you want it to. And if it's not, it shows precisely what is going wrong. And I know that it sounds like a silver bullet, but in many cases, it really is. And I hope that at the end of this speech, you will agree with this bold statement. So when I talk about Sentry offline, I always ask these questions and look at raised hands. So now I know I don't have such opportunities. So I will just tell you that it is always less than a half of the audience rising their hands to any of these questions. And with each question, the number of raised hands decreases. So the reason I want to talk about Sentry so much each time is that very few devs know about it. And if they know about it, they don't want to use it because they don't think that it is for free and they don't know that we can self-host it. And so I want to do something about it. So three of the last companies I worked for didn't have any real-time monitoring. The only way to know what's up with their services and if it's working correctly, it was to check logs. And so in all three places, I suggested to integrate and start using Sentry instead. And we eventually did. But at start, nobody even heard about it. And when I said that it's a real-time monitoring, the answer was, but why we have logs? And so don't get me wrong because this approach really works. But I also have to say that looking for thousands and hundreds and thousands of lines of logs is far from efficient. And during my speech, I will try to prove that Sentry is better than logs because of its convenience and describe situations when logs won't help you and Sentry will. So let's move to my toy project. Here it is. It is very simple. It is made with the Flask framework, SQLite and Uwiski server. And as you can see, it is really very simple. So there is one field for a user ID, one field for a value we want to insert to our database and associate with this user ID. These button add values and a post request with this data user ID and value. One button to send the get request and find, so get the sum for all values associated with the current user ID and two dummy buttons that do nothing just push some strings to our console log. We will use them later to show off Sentry. So let's start with the population database with some values and let's add a couple of it, couple of it. And we can see that there are two post requests, everything is okay. And let's check our database for this user and see that everything is okay. Everything was written to the database and let's return to the web. And now let's try to get the sum for this user. And if we send this get request, we will get the eternal server error. So 500 error code and let's go to our logs and look at them. And here we will see that we have this type error, so unsupported operant type for int and mysterious row. And in general, this seems an obvious bug. However, the problem is that this message doesn't give us complete information about what was happening in the servers at the moment. So what is the rule, what we tried to sum up. And the only information that we have here is the location of where things went wrong. And let's compare it with Sentry, what we have there. Let's go here. So this is the main timeline of Sentry and its project. We'll talk about it a little bit later. Now we just refresh the page and see what is the freshest error here. And here it is, so 68 seconds ago, this is our exception, sorry for big zoom. And you cannot see the message here, it's just cut, but let's dive in and we will see that we have lots of information inside. So for example, there is a type exception, message exception, yeah, message exception, then some information about URL that invoke the exception, then the version of browser and lots of other stuff. And then if we go further, we will see the same stack trace that we have in our logs. But here is the place where the cool stuff starts. So we can expand each frame and see local variables and its values that they have on each level. And here we can see that our variable values have not a list of integers as we expected, but a list of lists. And this is the problem that, the reason why we have the exception in the first place. So we sentry helped us to understand the reason of our exception and the reason behind the exact failure we've had without the need to reproduce the error. And so when I was in UB and even for a while when I wasn't, so exceptions horrified me and it was hard for me to understand them and fix them and it was eating up a lot of my time. And let's use this as a starting point and as an example and think about what I would do back then without sentry. So probably I would start with checking values in the database and we have seen with checked already that all our numbers. Then I tried to find out something on the internet and we'll find some stack overflow page that says so that the data coming from SQLite might be in some wrong format. Then I will try to run service locally and try to print values coming from SQLite. But what if I can't run it locally and I would, in this case I would deploy the debug version to a test machine and then check logs there and all this is painfully slow. So these could be go harder and worse in case if an error was more complex and if it involves user input or if it's some unlucky corner case which is hard to reproduce and happens really. So of course, in some cases you can extract URL parameters from your access log in case if it's query string but what if it is JSON parameters so they would long gone. And in this case, who will help us, it is sentry. So the only thing we need to do is extract outside data to a separate variable and you will see it in sentry right after the exception occurs. And here is the bad news for one line function lovers because it is really important to extract data in a separate variable. So without these variables you won't be able to see this data in sentry. So it's just good for you. If you use sentry, please use variables because it will make your reports more detailed and will help you better. So we've looked up the sentry event and realized that SQLite returns not a list but a list of tuples and let's fix it. I will just uncomment a good one and comment a bad one and here is a good one and you can see that I just unpack this list. And let's check that everything works fine in my project. Show user count and yes, everything is good now. And now let's change a user to another one and try to show its user count. And we have a problem again. So we can have internal server error again 500 and let's check our logs one more time. And we have the same exception tied and slightly different message but now this message is absolutely clear. So integer and string and it's obvious that there are some strings in our list extracted from database. If we check the database we will see that yes, here it is we have the string here as a value for this particular user and everything seems clear, obvious and very simple and why do I even bother you with this example? So there are several important moments. So if we check database scheme, we'll see an important thing that those value must be integer but SQLite allows us to insert values with string type. Okay, so did you know this? When I encountered problems caused by this behavior, I didn't and it was just like, I have no idea that such thing could happen. So I have a scheme, come on, what can go wrong? And here we come to the next slide. So there was an unexpected SQLite behavior and we always don't know something about tools and we always can have some problems that we don't expect it. And the next thing always, users always find a way to break a service. So there are, for example, there will always be a user who insert a string to a quantity input and in this particular case, developers can easily prevent this exact case and defend their applications but there will always be a way to break your application and someone can always can find it. And I separated these two users intentionally. So this is the illustration of different user behavior. So in my toy project, there are two users. In some test projects, there will be 10 and in production, there will be thousands, hundreds of thousands and more of real users and all of them will provide different inputs. All of them will use a service at a different time and with different frequency and some of them will make mistakes, some not. Some of them read tips and insert that accurately and some will insert that immediately and in a way they think is right. And in the end, an error could be rare and reproduced randomly and be associated with a small number of issues. It won't be visible in logs but users will be disappointed. So they, for example, they won't sign up because there is a bug in the sign up window or they won't buy goods because there is a bug in a cart. And for all these situations, H cases are critical because service loses real money in these cases. So even 5% of users who failed during their way to the pay button, they become important and so these rare bugs are important too. And the last thing, what's the chance that these rare bugs will be lost in logs? And here we come to the next slide, the disadvantages of logs. And the two main things are, so the first, there could be several servers with logs with different environment and no aggregation tool for them. So yes, I know that there are lots of different solution to aggregate logs, but there are lots of different companies that have no aggregation tools for their logs. So for example, in the previous company where it worked, I had 70 machines and no aggregation tool. But even if you have five servers or even three machines and there is again, no aggregation tool to make it convenient, will you check each one regularly with each deploy, for example? So I can say for myself, no, check one, maybe two, everything is clear and leave it. So however, the error could occur only on one machine, let's say 10 minutes. And again, we lose this error. And the second thing, logs could be noisy. So you always have these good errors. So expected ones, for example, network hiccups or third parties, service and accessibility, such things shouldn't be turned off. If there is no trustworthy monitoring, disabling these errors will make a developer blind. And you don't know if a service is alive or not from the backend side, I mean, without touching the user interface, for example. So these are the reasons why lots of people don't like to steer in logs. And these are the reasons why rare exceptions are lost. So it is cool if there is a QA who checks user stories, it is cool if a user cares and writes a report, but often we have no QA and the user prefers just to use a different service without the problem and leave us. And who will help us? Sentry. So let's go to the main Sentry page. So I mean the main Sentry project page. And here is the timeline with events that I mentioned before. So Sentry stacks the same events and places new ones on top. So here are two different groups. Again, sorry for Zoom, but if the Zoom won't be so large, you will see that these messages differ. So here we have for row and here we have int. And this is the reason why Sentry distinguish these exceptions to different groups. And this means that similar exceptions similar errors, they will be in one place and won't clog the feed. So you can see here that I have six events for this type of error from my previous rehearsals. And they are all here in one event group. So if you have 50 groups, for example, and you still have lots of events that clog your timeline, you can mark them as ignored. Here is the button and they will disappear from your timeline and won't disturb you here, but all the information you need will still be in logs just in case, but Sentry won't contain this noise. So you can make your timeline clear and clean with only that errors that you need and care about. And let's go to the presentation. So here you can ask me about notifications. So if Sentry will send me notifications, if Sentry will bombard me with notifications because there could be lots and lots of errors. So no Sentry won't bombard you with notifications. Sentry send notifications only about new errors. And I want to emphasize this one more time that on the new errors, I mean, if you already have a group of the same errors, the occurrence of a new member of this group won't trigger an email. So Sentry won't bombard you with letters. And it is possible to configure time intervals to aggregate error messages and events which will then be sent in one email. For example, over 10 minutes or 30 minutes or an hour. And also if you fix an issue, it can be resolved in Sentry. So just push this button here. So this will first of all remove an issue from Sentry timeline with a refresh on the page because by default, Sentry shows only unresolved issues here. And so you won't see it after resolving it. And secondly, if the same issue comes up again, Sentry will send you an email that there is a regression. So there are only two cases when Sentry send you emails. So first one, we have just new error, new event and the group just created. And the second thing, the second moment, we have resolved this issue and it comes up again and Sentry will tell you about regression. And for super speedy notifications, Sentry can be integrated with your favorite team message service like Slack. And messages there appear instantly and are seen by the whole team. The same rules with notifications as with emails, Sentry only notifies you about new groups or aggressions and they can be aggregated and all this stuff. So we can do all the same, but just for instant notifications. And let's move further with presentation. So of course, it is easier to turn Sentry into yet another dead log. And so if you don't pay attention to it. So while there is a narrow subset of normal bugs, which I mentioned before, it is very important to fix all the bugs which you catch with Sentry. But you can say that what if there is a deadline coming up and if there is a critical issue with the user registration and we probably don't care about some parsing error once in every two hours. And yes, this is true and this is life. And there is a way to not turn Sentry into a bug cemetery when such situation happens. So if error groups have already been created, there are no email notifications. There's no visibility for these errors. You should come exactly into Sentry and look at the timeline and see that there are some fresh events. Maybe I should work with them, but we know how it works. So good practice against that is to auto resolve all errors in Sentry in several days after the first occurrence. So there is an option in project settings. You had a deadline and there was simply no time, resolve and you get an email in some time or maybe it will be in the moment when you have time for it. Then you got 100 of emails, notifications while on vacation, same story. And also you can use this resolve feature for situations, for example, imagine Sentry caught some strange, hard to reproduce issue and wish you don't think it's important right now and you don't want to investigate it, but you still want to know if it happens again and when and how frequent it will be, so resolve it and Sentry will let you know if it happens again. And let's move to the next slide. This is some conclusion here. It's not the last slide of my presentation. I have more to say, but I just want to sum up some things here. So Sentry allows you to be notified about errors instantly as soon as they occur. You don't need to crawl through the logs with table plus grab and you don't need to search necessary logs on different servers if you have those aggregation tool and you won't miss an important new error or aggression. So, and one more time, please clean the bugs, fix them and use outer resolve for the events in Sentry. In this case, Sentry will be a good variable too and not deadlock instance no one cares about. So I hope at the current stage, Sentry already looks interesting to you. So let me tell you a few general things about it and some facts about customization, deployment, self-hosting and so on. So this is the most important slide and the most important part of Sentry. Sentry is free to be specific. You can use it for free or by subscription. So I just, so it's not very obvious from the site. So it's common for someone who heard about Sentry but never used it to justify it by thinking, I need to pay for it, I won't use it because it is paid. So that's all, I won't use it. But you don't and there are different ways and paid subscription has some advantages and what is the paid subscription? So with the paid subscription, you get the cloud service and you are provided with the DSN to Sentry which works in the cloud instead of your service. And there is, for me, there is a problem with it. Events are coming to Sentry cloud with a significant delay sometimes. So up to two to six minutes from my personal experience. So I was trying to use the cloud deploy of Sentry when preparing for this talk for the first time. And it was often a waiting game until the event hit Sentry. So I've deployed it locally and same story in production. So you've deployed a new version with a critical issue that which of course you didn't know about. And if your service is immediately torn apart by hundreds of errors, these two to six minutes are very important. If the issue is small, this time might be enough to fix the issue and redeploy a new version or just to roll back the version and use the old previous one. So while with the Sentry cloud, you will only be notified after that time. So I prefer a self-hosted Sentry for my project. So it's really easy and won't take a lot of effort to do. There is a page in documentation and let me show it to you about self-hosted Sentry. So there are very simple instructions. And in previous years, in previous versions, I've made my own docker compose file and shared it via my GitHub. But during this year, Sentry made it. They made their own docker compose and now it's really three simple steps. So just check out repository and then run one comment, run another comment and profit you have for your own self-hosted Sentry. About load, if you had a lot of events and don't want to think about distributed Sentry, so use Sentry cloud in this case with basic subscription. So it is the case when you need it and it is good to give money to Sentry team because we all want to Sentry to lever and be with us. But if you're, again, don't want you, it is possible to configure rate limits in Sentry, then some events will be ignored, but it will guarantee that there won't be a load issues in any case at all. So your choice for those who just want to try Sentry out on some local pet project, but don't want to jump into deploying it locally, there is an option to use cloud version for free and now there is a button on the main page, try for free. Just register on the Sentry site and connect your project to use it. You can use it for free if the project has only one user and the number of events is less than 10,000 events per month and it is actually more than enough for many small projects. But just remember about delays I was talking about. So let's pretend that we deployed our self-hosted Sentry and registered to Sentry in the cloud and now let's look how easy it is to create a new project. So I need to go to just project page and here is the button create project and here is a bunch of different languages that frameworks that Sentry supports and they separated in different groups and loss and loss of them. I think that everyone can find its own tool and instrument but here we will use Flask and several settings that you don't need at the moment and I will set a project name and create project and Sentry is very nice and after creating a project with exact language or exact framework, Sentry will show us this part of the code that you need to add to your project to integrate Sentry and let's look at my project here and this is the whole configuration that I have to integrate Sentry. The only thing that I need is this variable with the descent to my project and I can find it in the settings of this project. So we go into project, then we go to settings and in the bottom of the page we will get this client keys DSN and here is this link that you need just copy and use in that small snippet of code. So after you did it, Sentry will catch all and have the exceptions and also a killer feature. So I think this is the brightest star on the Sentry sky. So you can not only handle unhandled exceptions but you can send the events to Sentry by yourself. So using logger error, logger warning and logger exceptions. I think this is one of the most amazing features of Sentry. In previous versions it used to be adjusted separately and many people didn't know about it. Now thanks for Sentry team, they made it work out of the box and you can use it in the following cases. So why I think this feature is so amazing and what cases it covers and when it can make your life easier. So the first thing is working with third party services via HTTP. So for example, you want to show the where the forecast in the corner of your main page and you are getting it, so the data, the weather data, get it from some other service and so you just need to put this block into try accept and log it to Sentry. So what you get with it, there are several points. The first, you will know for sure if there are 500s and which ones, which internal errors and you can judge whether the third party service is stable enough for you or you need to start looking for an alternative. If that's your own service, it may highlight a bug in this service or a bug with integration and you can work with it. The second thing, you can validate responses from the third party service and catch format changes or some inconsistencies which you need to fix up on your side and you will know about them immediately. So not after some time when you will see that, oh wow, on my page there is no weather, now what's happened, let's check logs and all this stuff so you will know about these changes immediately. Then it's easier to figure out the reason of invalid response from that other side. So when there is no error, just the response is the data that you get is not something that you expected. So instead of spending hours trying to reproduce the issue, you can log and let us see actual parameters the issue happened on. So for example, for our weather service, we might have received an empty response and the reason we sent invalid coordinates and if we only check in logs, we only see an empty response dictionary but it doesn't show why it is empty and we can spend lots of time trying to understand why. Here you will see these parameters that you sent to that side and you will see all parameters are invalid. Yeah, it's my problem, I need to do something about it. Then the second point, the second example is that there are cases when you know that in some places there will be unavoidable errors and you know it for sure and you just want to know how many of them and how often. So for example, you are working with a partner which every night provides you with new data about their products and you're downloading them, parsing and for each product, you need to download an image to your storage. And you know from experience that some images will be unavailable and you just want to know for sure how many. So below 10 can be ignored, above 50, maybe it's a good idea to ping partners and maybe they even don't know that they have problems. Then the third thing, the cases with refactoring. So you refactored some part of your service and want to remove some old code but you are not sure if it's not still being used somewhere, for example, dynamically. You can mark such a deprecated method with a warning and Sentry will tell you for a fact if and where it's been used from. You didn't get any warnings for a month, then perfect. You can now remove this code and forget about it. So the similar example, you deprecated some type of products or subscriptions at your service or change an API. So in this case, you can mark old requests with warnings and see which of your clients still have or are still using obsolete requests and you need to ask them to make changes on their side. So this is the part where Sentry is a way not to be afraid. So with Sentry, you won't be afraid to make changes in your code because you can guard all these places and you will immediately have a response if there are problems. And the last point, Sentry can be used to some kind of monitoring and statistics gathering. So we know that there are lots of tools about metrics, statistics and all this stuff, but if you have no experience and you just don't need some great tool for it and you just need to collect some small bunch of statistics, you can use Sentry as this tool. So you can add login of some variables there, get it to Sentry the same way that you are getting errors and such events can be grouped automatically so you can manually group them. Sentry has this tool, just dive into the documentation and you will have these groups of different events with different counters and you will make, you could make some decisions and have some thoughts about it. And let's go to the next slide, so the front end. So now we are here at EuroPython, but I felt that it is important to mention JavaScript and the front end port as it is all interconnected. I know very few backend developers who never ever touched front end in their projects and it is important for backend developers to at least understand and be able to monitor whether backend functionality is used by front end correctly. And so here is a little part about front end. So we've already talked a lot about what Sentry can do on the backend side. So let's have a look at how it can be integrated in the front end. So let's look at our front end code. So this is the whole integration of Sentry to the front end. Again, it is very simple and everyone can do it. You doesn't need to do any additional terrible things integrating Sentry. So the same thing, initialize Sentry, add the DSN and that's all. So you can use the same DSN as for the backend part if you want all the errors events in one Sentry project. But you can create another project for front end errors only and separate them. So again, it is very convenient and you can adjust it the way you like. And let's go to our project again. It will be the last demonstration of, we are live demonstration. So let me clear everything and as you remember, I've told about these two buttons that they have, they do nothing, just push some strings to console log and it's true. And now just let's choose some user that we have known in database and try to show the user count. And we will have the exception here because, so front end is broken because we tried to get the proper tier that is not available in our response. And if we look here in network, we can see that we have these requests here. So we have requests to our Sentry with some data. And let's look into our project and find this issue. So here it is, can it be proper to some or undefined? Let's dive in. And we will see all the same data. So type, message, when occurred, then some information about the client for front end. It is very important, versions and all that stuff. Then the same stack trace, it's rather useless for front end. But here we have breadcrumbs and this is the golden place because here we can see the whole history of what I was doing on the page. So I click in button add. I have the value 200. Here is what parameter and its values were. And then I have this console output with extra arguments and all that stuff. I can see this string here. And so the idea, you can adjust what is going on. And you can adjust the context that you will have on the backside. So what context will send to Sentry from your front end? And so you will need to make it manually. And there is some work to make it better. But even out of the box, these events are very useful because for example, you deployed a new version of your application and you have a wave of errors from front end and it is the sign that something goes wrong. You need to maybe roll back and investigate the problem. And as I said, with some effort, you can adjust these events and reach the context and get additional information. So let's move to, this is all what I wanted to say about front end, so just use it. It is really important for both sides, back end and front end. And now let's summarize what I've said here. So I think that Sentry is extremely valuable and necessary during development. So this is one of the first modules so you want to integrate into your project, new or old. And if a team consists of developers and QA, there are less back and forth questions between them. And you as a developer don't have to constantly ask for more details, ask your developers for access or copy of the logs, ask QA to add more details to the reported issue and ask your support to get some extra tricky information from the client. So you have all that in Sentry. And more importantly, you have it there before it reported to you. So you have an ability to fix an issue even before it is reported. And because you have all the context to reproduce and investigated by yourself and it is amazing feeling when you get the Sentry exception, you fix it. And after some time you get the message from the support or from the manager that, oh, wow, we have a client that have this complaint and you say, I've already fixed it, just perfect. So I hope that now you believe that Sentry is a good friend and assistant to find bugs and deliver great and stable web services. So Sentry is an easier to integrate and even it's out of the box form, it will be irreplaceable. And with some additional efforts, it gives you a great flexibility. And so read the documentation. I'm far away from description all the goodies of Sentry. So there are lots of treasures and documentation. Please read it and please use Sentry, I hope. And I believe that it will help you greatly. That's all for me, thank you for listening. We do have a couple of questions, but since we're all done, you could take them in the breakout room. Okay, thank you. Have a nice day and have a great conference.