 Hello. First I wanted to tell you, you can automatically check if code is good. And I will tell you what we can do. This is called Quality in Python. My name is Erdos Foghancharek. And let me tell you a bit about math. Are there any mathematicians here? OK, no. OK, so we'll go fast on that. So there's a theorem called Rice Theorem in math that has some very obscure definition. But in short words, in words that normal human can actually comprehend, it means that we can't automatically verify if a program has some non-trivial property. So for example, we can't write a dictionary when you will check if your program is correct. So you can't get a dictionary when you will search for your program. Oh, my web application is right here. It's correct. All good. You can't write such a dictionary. You also can't automatically verify if program never fails, if program returns some data, if it is well-written, whichever it means, or the WTF factor, because you can't do this. But you can try. In fact, all we do is try. With all our automated tools, we are just simplifying this task by adding some boundaries and some limits so that this task is easier. So don't worry. It might have been harsh, but it's going to be OK. We can do some things to check if program is good. So I will introduce you to many tools that I use, or at least try it, about tools that are used for checking programs in Python, or making them look more beautiful, like formators, import sorters, and also a bit of the code coverage. There's a small summary of what is this talk going to be about. Are there any managers here? No mathematicians, no managers. I don't get you. OK, so that's right. So I will be talking about developer perspective. So how do we can use these tools? How we can improve our code with using them? Because, of course, managers like charts, like the persons, and like to see that they grow or drop in some cases. But we developers are more practical. We want to see how it changes our code. OK, so let's begin. OK, so here is a graph of various checkers. You probably, or you don't, you probably know all of them, or nearly all of them. There was a talk about Radon yesterday. Yes, so this graph presents how these checkers are contained in each other, meaning that checker that contains other is checking what these smaller checks. Yeah, so now we see that we have one big checker, PyLama, and smaller ones. I will talk about all of them. OK, so let's start with PEP8. Who doesn't know PEP8? Raise your hand. Oh, that's bad. But probably you're not programmers, OK? Yeah, so PEP8 is a Python standard for how code looks. So it's just about how code more or less of looking and using proper idioms than of the semantics. So PEP8 is a checker that in most cases you can afford confirming all these rules. There are no rules that you will find difficult to use. Yes, so that's a style checker that's the must-have of all the in most cases. So next one is PyFlex. PyFlex is a small checker, even smaller with PEP8. But it checks many, many nice things that you would usually use PyLint too. But PyFlex is faster. It's checking for unused variables, unused imports. So if you have a guy that, when he's creating a new module, he's copying the wall of imports from another file, you can check it with PyFlex and actually give him PyFlex and say, hey, remove these imports. OK, next is Maccape. This is a very, very small tool. It's just checking the complexity of functions. And what you see here on the screenshot, on the shell shot, in fact, are the complexity for functions. And you just assume which level of complexity is OK for you. For example, here we have 18 level of complexity, so that's very, very bad. OK, and Radon. Radon is like on this graph, a bit off of the whole ecosystem, because Radon is more like a metrics tool, so a tool for managers and tool to see the big picture of the project. And so if we are going to look if overall the project is doing good or doing bad, we will use Radon. But I think it's no use using it every day. Yeah, but it also counts the Maccape score and other metrics. OK, I told about PyFlex and PyFlex, and here is Flake8, which incorporates PyFlex and Maccape. So this is a tool that, in most cases, it's enough to use it if you have dirty things in your code base. I will talk on this later. Flake8 is also a tool that, in most cases, it will be OK to follow all these rules. And here comes PyLint. Here comes PyLint. Who doesn't know PyLint? OK, so PyLint is a tool for analyzing Python code for a bit more, there are many more violations checked, not just the Pep8 or something like this. There are lots of, lots of things that PyLint checks, PyLint is very thorough, and that's its blessing and the curse. Because if you are, for example, creating a class and later importing it via some import or something like that, PyLint will not like it. If you have a class that generates its fields, PyLint will not like it. It will say that field is not defined in the class. Why are you using it? And you are using it because you know this field is there. So PyLint is a tool that you can't just use without configuration. You will always need some configuration. There are modules to ignore some common things that happen in Django on Flask. But PyLint is also very, very useful because it also points some places where you need to refactor something, where the function is too long or class has too many public methods or has no public methods which in fact should not be allowed. Class needs to present some interface. And PyLint is pointing it out. It sometimes is relevant, sometimes not, but always gives something to think about. So I recommend PyLint. And here is PyLama. PyLama, as it was shown on the graph, it just gets all the checkers that I was talking about and runs them. And yeah. And PyLama, it's easy to think that, wow, we'll just install PyLama and it will do it all. But if you want to configure all these tools, if you want to install some extensions to them and you want to have some specific settings, it's difficult. You have one big configuration file which for some people would be good. But for some people would be a hell to edit it and search for some things. I personally like smallest tools I can actually have. With the example of Flake 8, because PyFlakes is not as big to be treated as a fully separate tool. And Pep 8 and PyFlakes are integrated very tightly. OK, so there are also other tools. The tools I was talking earlier were a classic tools for static analysis of Python code. But here are some other tools you can enjoy. Pep 257 is like the document, Pep 257, is checking if your docs mix are valid. So it's very thorough. You can see it's going to say that you don't have a point. Sorry, a full stop on the end of the sentence. And many other, and enforce that the summary line should be, in fact, one line and many other things. So PyLint only tells that the docs mix is missing. But if you want to be a docs mix nazi, Pep 257 is something for you. OK, Vulture. This tool checks for that code, for code that is not used. And as I said, when I was talking about PyLint, it can always fall in a trap of class that is defined but used by some magic or some attributes that are set with magic methods. So with Vulture, you need to be extra careful, because it doesn't have many configuration options. So you just can't ignore all the things you know will fail. It's just between art and science, checking which functions are really not used and which are used, but it's not obvious. So I recommend, but you can't follow it blindly, because it will not end well. And last of the additional tools is iSort. iSort in its default mode sorts the imports in all the files you can find. But it can also check for imports, check if the imports are well sorted. Yeah, that's not very meaningful. But if you want more meaningful results, you can install a plugin for Flake 8, which makes some real hints how to sort imports. OK, and to some of these tools, you can write extensions. In Flake 8, it's also usually it's an extension with another track for doc strings for imports. In the Piling, it's usually something that makes some projects specific changes like for Django, Flask, and so on. And to Piling, I can add new trackers. And when writing an extension, you can choose either to analyze a row code or abstract syntax tree. Who doesn't know what are abstract syntax trees? OK, so when you have a Python program, and this program is parsed, and the special words in this program are changed into tokens. And from these tokens, you form an abstract syntax tree, like abstract syntax tree if under this node is the logic value, the command for then, and so. So it's your program in the purest form. And you can see here an example. It's the code for the most relevant part of code for the Maccape extension to Flake 8. So we can see that it just retrieves the graph and goes on this IST graph by every node, checking some thing. In fact, here it's simple because the path graph IST visitor gives the complexity function. So here it just counts the max complexity. OK, so next thing, formatters. So if you're lazy and you want your code to look beautiful, you don't code Python. Because Python requires you to work with all these white spaces. And if you are not respecting it, you are not coding Python, in fact. So yeah, but there are some tools that are to help, but they won't do all the job that you should do. The first of them is AutoPep. AutoPep, I don't recommend because how you can see there, it fixes Pep 8 violations, not all of them, but leaves very ugly line breaks. So not. It's not that, unless you want to check and repair after it. Yap. Yap is a rather new project from Google. And in the docs, you can read that Yap is also not meant to do all job for you, but it's meant to unify the style among whole company. For example, they wrote, in fact, that you can add a pre-push hook that will format your code with Yap so that the format on the master branch is always formatted the same way. Even if you have your personal preferences, Yap will do it its way. And I sort, I was already talking about it, it sorts imports. OK, next thing, test coverage. Who doesn't know what is test coverage? OK, test coverage, it's about that you run all the tests and in the meantime, you check which lines in your code were executed so that you can be sure that all your code is executed. Coverage is a module that you can use also in other cases. For example, if you are running your application, just to see if the code is used. But its most common use case is just for testing. And the common use case for using coverage is that you check coverage on your branch. If you are working on branches on Git, for example, and you compare this to the coverage on the master branch so that you don't do worse than it was before. One thing to remember about coverage is that the extensions to NOS tests or PyTest with coverage sometimes don't have the good results. And near the end of the presentation will be shown a better way of running coverage. It should be just run before the actual test runner or just run test runner inside the coverage, not the other way. OK, and other articles that you can use, DivCover and DivQuality, they are tools that are running. You give DivCover and coverage XML file, and it checks if there are some missing lines on your branch, but only on your branch. And DivQuality works a bit the same. You give it a checker, and it checks but shows the violations only on your branch. DivQuality supports only a limited set of checkers, but you can add new. Now it's not possible by extensions or configuration, but maybe it will be in the future. OK, GitLint is also a small tool like DivCover, but smaller because it just runs PyLint on your branch compared to the master. And if you have a colleague that leaves orthographical mistakes in code, yeah, this tool is for you. You can check if there are any errors in your code, like what we have here, absolute instead of absolute. Yeah, so you can catch these things. OK, so a couple of words about automation. Who doesn't know TOX? OK, so if you don't know TOX, you want to know. Because TOX is a marvelous tool for running tests, and it handles its own virtual hands. So you just don't need to bother about anything. You just configure TOX and run it, and it's done. You can also enjoy some plugins to PyTest that check PyLint. And what I recommend and what I use in my project, we are coding on GitHub, and Jenkins has a plugin to build every pull request. In our case, it means that this plugin for every pull request is running the test, the PyLint flake8. And if the results of tests, if the coverage is lower or if the PyLint fails, the pull request can't be merged. So I recommend. However, you might not want to run Jenkins because it's Java software, it's big, and it's consuming all the RAM it is given. So maybe you can use Travis or another CIA tool. OK, and it's an example of configuration of TOX. You just give what Python versions you need to check against, and you write commands. Here we see just clearly that coverage run minus mnose. So it runs mnose inside coverage. It generates XML, fetches the latest master, and runs DivCover, DivQuality. It's an example from the DivCover project. You can find it on GitHub. OK, so a couple of words about why. I probably have one minute. OK, but in most of these cases, you just want a uniform style. You just want that when you go to some module that nobody worked for a month, and a man who worked on that lived the company, when you go there, you want to be in the environment that you know. And now one minute, thank you. And PyLint and Coverage and other tools give you that. Also, test coverage, if you have high test coverage, like 90% or better, it gives you confidence to refactor. Because if you refactor and you break something, you will know about it, because all the tests will run, and they will say, what's wrong? OK, if you don't have time to do this, you can go to managers with these buzzwords. Maintainability, readability, extendability, it's always better. Productability also, yes. So as a recap, check your code, run tests, run coverage, all you will find, all things will go terribly wrong. And are there any questions? We have time for two questions. One in front, and then we'll go to the back. Wow, great. So OK, first of all, thanks for showing PyLama. Great, but the one point I'm missing from your presentation is SonarCube, and it's plug-in for Python. Well, let's be silent about checking rules it uses, because it's its own custom Java implementation of analyzing AST trees. But the cool thing about this is that it maintains a list of issues in your code. Yes, that's helpful. But SonarCube is running only PyLint. And a couple of custom checks, but that's not enough, at least for me. Yeah, you can use it. And if you have enough RAM to run SonarCube, go with it. Yeah, it's very nice. It creates issues. If you actually want to have every violations as a separate issue, that's the other thing. OK, but if I may, it wasn't actually my question yet. So I would like to ask you if you know any other tools for virtualizing or maintaining a list of issues that come from PyLama or other tools you mentioned. Because I bet you use continuous integration and so on. So in my current projects, we just had a master job on Jenkins. We checked master with all the rules we really want to be followed enabled. And if someone had some free time in the end of the sprint, he just was taking some of these rules and fixing them. Some of them also required some serious refactor. So you can fix all the PyFlag violation, for example, I hope, in one day if it's not a Gargantwijk project. But for the more severe refactors, you will need to really think of what needs to be done and the bug for this won't be just fix violations here and here because it might be a bit harder. OK. Thank you. Unfortunately. One more question. OK. Yeah, my question is if you have some good tip for enforcing the rules in a team of five or more people without being too rigid about, for example, the PyLama rules and stuff like that. Without need for what? Sorry. Without being too rigid without having a push which refuses the push. If you have some other, because in my experience, that's like a big problem. You have rules, but always you have people and circumstances and other problems which when you break the rules. So if you have some good tip, how to avoid that. OK, my approach was to enable the rules that are checked, not all of them at once, but the ones that can be enabled. And we weren't using PyLint and Flake8 from the very beginning of the project. We just started using it with the core PEP8. And later we expanded it with consecutive rules from Flake8, from PyLint. And I think that's the path. That was my path and it worked. People weren't angry about this because in general, people are most angry when you make them fix someone else's mistakes. And if you have deep quality, deep cover, they will just need to fix their own mistakes. So if you will talk with them and tell that you are going this way and they will agree, so it's going to be OK, I think. OK, thank you very much, everyone. Thank you, Radislav.