 My name is Rados Afgancharek, but you can call me Rad. It's much simpler and more available for English speakers. This is not my first EuroPython. And this is not my first talk on EuroPython, because I was here. And by here I was in Bilbao on EuroPython 2015. And my main title of my talk was pretty much the same. But now the subtitle is different, the reasonable approach. So what happened? Why reasonable? Because over time your view on things changed. And I wanted to share a couple of thoughts I have after a couple of years of fighting for better quality of Python software. So this talk will be mainly about tools, tools that we use to check the quality or fix things, formatting. And it will be mainly about Python 3 and the tools that are most up to date now that are fresh enough to be safe to use. And it will be only about quality checking, because there are lots of tools in Python for other things, like for checking for SQL injection, for checking the spelling in your code. So I will just focus on code quality and a bit just a little bit about testing. What I want to talk about is IDE-specific and from the framework-specific tools, because there are very many of them. So they deserve their own talks. OK, so let's start with meeting my friends. Unfortunately, they won't join me on the stage because they are imaginary. First of them is the hobgoblin. You know the quotes about the hobgoblins and stuff, so let's sum up this approach. So what does it mean to be a hobgoblin? And do you, are you a hobgoblin? Or are we, or am I a hobgoblin? So hobgoblin is narrow-minded, so he doesn't care about what happens everywhere else, except his own piece of code, his beautiful module that he's working on. And from that reason also, he likes business perspective. He doesn't think about business. He thinks about making his code most beautiful and most according to the standards as he can. And third thing is that hobgoblins are extreme. Ah, OK. OK, no problem. OK. How do you, yeah, OK, that's good. Thanks, thanks. So let's talk about hobgoblins more. So, and hobgoblins are extreme. They either want to follow all the rules or none of the rules because only this makes sense for a hobgoblin. Also, which is a result of all of that, hobgoblins value rules the most. So project shipping the version of the project valuable for client is not important for them. OK, so let's meet another friend, meet Timmy. Timmy is average Python developer. You probably all you know Timmy if you worked in a Python project. Timmy is very skilled. He has some experience a couple of years. And he has his own style of coding Python, which might be in some ways unorthodox. And Timmy, some people say Timmy is mean and they don't like Timmy. But the truth is that Timmy is afraid of changes. And that's why he doesn't like if you come and say, oh, we're going to measure the quality of our code. OK, so the third friend, Zen of Python, probably you all know this. But I won't talk much about it because we are going to talk about this in detail. So for hobgoblin, Zen of Python is like set of absolute rules. And hobgoblin also tends to justify his choices by twisting a bit the Zen of Python guidelines. He said, oh, OK, but the code should be readable. So we won't use these features from Python 3 because nobody knows Python 3. And on the other hand, Timmy is like, oh, it's like pirate code, guidelines, more guidelines than rules. We need to embrace the spirit of Zen of Python, not follow the actual rules. Yes. So both of them are in some way wrong. OK, so let's start with the first one. About a beautiful is better than ugly. What does it mean? It can mean lots of things, in fact. But there is something in our mind. Our mind does a couple of tricks. For example, beautiful people seem to be more honest, more good. And the same is with code. If you look on the code and it looks beautiful, it has lots of space, beautiful line breaks, and it's well divided. You have a feeling that it's well written. It's not always true, but it's good to have a good first impression. And to have a good first impression, we can use formators. Formators that will format our code for us. There are two main players I identified. So first is black. So if you know this quote by Henry Ford, that your car can be any color you want, except it is black. And that's how black works. It has pre-picked formatting rules, no configuration at all. And yes, it will format your code according to the rules that black has. On the other hand, you have yapp written by Google. And yapp is very configurable. You can change many things. But the drawback of this solution is that if you gather your people, gather your fellow developers, and say, OK, let's sit and write configuration file for yapp and pick the rules, you're going to spend a month on it, and maybe even more. So black is a good as a default, but yapp is a good choice if you want to really pick the right rules for your system. And yes, so the hobgoblin approach here is to just enable everything. Enable everything, and let the guys be shocked by what happened. On the other hand, there is Timmy. Timmy is developing Python for like five years. He has his own formatting style, and he really thinks he can't read any other code. But if we have five Timmy's and each of them has different style, it might be at our code might change it to chaos, which everybody is a little shifting to his own preferences. So using a formatter is a good choice. A formatter we all agreed on, of course. Yes, so let's talk about typing. I won't tell you if types in Python 3 are good or bad. You probably have already your own opinion about it. But if you are using types, it's good to check them with MyPy or Pyre. They are the tools that are more or less the same, but you need to check the details to know which one you want. Pyre aims to be faster and more thorough, but now it's still developing. Let's talk about the simple things or complex things. Let's say you have a project when you have very specific needs for static analysis. You have lots of regular expressions, and you have a specific style of regular expression that you use, and you want it to be followed. So one solution is to write your own static analysis tool, which can take some time. On the other hand, you can use BellyButton. In BellyButton, you just write your own rules using the AST expressions. So it's relatively faster than writing your own tool. And adding additional rules, once you learn how to add them, how to write the rules definition, adding additional rules would be even faster. So as you can see, Hopgoblin doesn't like this tool at all because Hopgoblin likes to just check all the rules, turn on all the rules, and then sit and watch how the world burns. And BellyButton just makes you to ask yourself, what do I want to check? What I want to be checked in my software? OK, so let's talk about Pylint. Who uses Pylint? Wow. OK, so that might be not a slide for you exactly, but I know people who just hate Pylint or more exactly are afraid of Pylint. Is anybody here who is afraid of Pylint? OK, see. So people who are afraid of Pylint, I don't mean you because I don't know you, but people who are afraid of Pylint tend to say, oh, it's enough to just take Flake8 and download 50 plugins, and it's almost like Pylint. It's not like Pylint. It's almost like Pylint. It's not Pylint. So I know why Pylint might be scary because it has many rules. It has many things. And the most important thing is that if a formatting tool shows you a violation, you can fix it just in like five seconds. You need to add a new line, add spaces. But Pylint sometimes can show you a violation that makes you refactor all your code, or maybe whole module. And that's a bit painful about Pylint. However, if you follow it from the start of your project, it's much simpler following it. I will tell later about how to deal with old project and the new tools. Yes, and Timmy will say that, well, what are these rules for? They're just good practices. Every developer follows them. But also every developer is a human, and humans make mistakes. Even now, for example, it's maybe much simpler topic, but do you ever make PEP 8 mistakes? Yes, yes, we do it. Even if we work 10 years in Python, we are still doing these mistakes, maybe less often than in the beginning. But it's not something that you can just learn, and it's all good after it. You need to check. OK, so let's talk about big, big tools. There are many of them, or maybe not so many, but they are big. So first of them is Prospector. Prospector is a very nice tool. It's in the GitHub organization with all the cool Python code analysis tools. And also there is another one. There is WeMakePythonStyleGuide, which uses mostly the Flake8, which is a custom configuration with all Flake8 plugins. And there is also PyLama and Flake8. I think everybody uses Flake8. I don't, but it is very common. I have a problem with big tools, and you can see this problem here. Let's say I wanted to use Bandit and PyLint. I don't have it. I might use Prospector and then use Bandit, but in that case, what do I win for using Prospector? Maybe a couple of configuration files less. But yes, but the big tools, they are good for checking things fast if you are doing an audit or you have some old project you want to quickly check things. But they can use outdated tools. Here, tools that are not bolded, they are outdated. They didn't have any new version in the last half a year. So yeah, you must think about it if you want to use it. Also, I tried to use both of these big tools just after installing it, and they didn't work. Maybe Prospector worked, but Prospector couldn't run PyLint for some reason. So big tools may have big errors. And if you have a set of smaller tools, you can deal with errors individually. Also, you don't have any choice of what tools will you have in your project if you use just one big. In fact, you choose what they choose for you. And one configuration file, okay, it might be not a bug thing. After all, we use setup CFG for all the configuration. And yeah, so for Timmy, one big tool is a good choice because it's just one tool, it checks, it's over. But in real, usually you want to pick your tools and have only those that you need. For example, if you don't have dog strings, why do you need PyDogStyle? Okay, and let's say we have a big project and we have installed a couple of tools for analyzing code. And we have code coverage tool enabled. And so what do we do to don't be flooded with all the violations? There is a tool called DivCover, which also has a binary called DivQuality that checks the violations or coverage only on the lines that you modified. Of course, there might be a problem with it because if you modify the line and made an error somewhere else, it won't catch it. But it's good enough for a quick check. And also, in general, in measuring test coverage, there is a trap that you don't... The coverage, in fact, tells you what code is bad, not what code is good. Like if you have 85% of code covered in unit tests, it means that 15% is... There might be something wrong in 15%. And 85% might be good. Okay, so let's talk about readability. Python has a tool called Vulture, which is very... It can't be used very, very often because Vulture looks for unused stuff in Python code. And if you code in Python like for more than one day, you know probably that in Python, there are many situations when you just change the way how the stuff is used. You make a dictionary of functions or call a method via getAt through function. So sometimes it's not visible that something is used. So Vulture is giving lots of false positives. But after all, if you filter them, if you somehow look into them, you can track the really unused pieces of code. So be careful with Vultures. Okay, so what about special cases? Hopgoblin will likely say that, oh, we have special cases, you're not good enough. We all know that it's not true because Python projects have many shapes and many topics. So there are various crazy things that happen in Python code. And if you're using tools for checking the quality, there are many features that can help you. You can, in Pylin, for example, you can use regular expressions for file discovery or even ignoring lines. You can, instead of letting Pylin or Pycode style discover the Python files, you can prepare with a bash comment, a list of files that really needs to be checked. And also, in each that case, when someone tells that, oh, it's a special case, I try to think, is this case really special? Because it's easy to say, oh, it's a special case. Let's disable everything here. It must stay how it is. So don't, going the easy way is not always the good. Yeah, so there is a tool I found, Pydiatra. And I thought it was a clone of Pylin or something like this, but not. It's just a tool with some custom static analysis checks, which is, might noting, there are some checks about regular expressions. So I don't, I can't recommend this tool blindly, but you can, you need to check if the checks that Pydiatra has have something that is really useful for your project. Because if yes, you might waste some time trying to check it or write your own checks. Okay, so what about checking code automatically? In your code, if you have a CI, you can use talks for configuring all the checks. So really, you don't, if you are using multiple tools, as I recommended, you are not, you don't need to run them independently. You have talks, you have bash. You can write a comment that can run it. And in the CI, if you have Jenkins or GitLab CI or anything else, it's really useful that the code checking goes before the tests because if anything fails, you will have a fast answer. Okay, this is about the same as the previous about special cases because we really like just the disabling rules if anything goes wrong, especially if it's Friday, 3 p.m. and we want to go home. There is also an automatic tool for setting types, but it can be used for old projects that don't have types, but not for in day-to-day work. So as to sum up, if you are going to use tools in your project, pick Bandit or Dogey, no Dogey, pick MyPyPy or PyType, PyCodeStyle is a minimum, PyDocStyle, if you are using DocStrengths, PyFlakes or PyLint, iSort and the black for formatting. Here's a slide with Spanish Inquisition. And okay, if you have an existing project that you want to start, you can use tools like Autopep8 or Autoflake to automatically fix issues, but the fixes need to be overseen by a human. iSort for sorting imports PyCodeStyle, PyLint. PyCodeStyle and PyLint, it's very minimum for me. Yeah, but the most important thing is that you should talk to your team because it is very, very important so that everybody accepts the tool you are going to introduce because it might require some additional effort. Okay, I can skip this one, but the advantages are very good. You can reach more readability, maintainability, enhanced refactoring and it may need some additional time and it might produce false positives, but for me, advantages overcome the disadvantages. If you're looking for good tools for Python, visit PyCQA and or awesome static analysis links. And after all, if you want to introduce, by introducing tools for analysis of code in Python analysis of good practices, you want to inspire people and lead them to better implementation of code and not to be a bully. That's all. Thank you, Radoslav. If you have questions, you can raise your hand and I'll go to you with a microphone or you can go to the microphone stands. Okay, so hi, great talk. And do you have any suggestions for third party, like for your requirements, like to check that code, something like NPN audit or for Node.js for Python? For Node.js? No, no, no, for Python, to check your requirements for third party. Our requirements, yes. Yes, for example, I only know there's an online tool available for open source project called PyApp. They are checking the requirements and freezing them in the requirements TXT file. I found the tool for checking the requirements but it was very outdated. So, but I think that there might be something I missed, so I encourage you to look by yourself. Okay, thank you.