 Thank you guys. It's really nice to see so many people interested in static analysis here and This year is a special year for piling because it is it's 12th birthday. So Yeah, basically, it's the oldest static analysis tool in Python that is still maintained, which is really nice so in this talk I'm going to talk a little bit about piling's history and we'll go through a Detour in static analysis and we'll see where where that goes So what is this lint thing? I'm talking about basically a lint is like a tool you use to analyze your code in order to find bugs or potential errors or Style checks or stuff like that. But Saying this about piling is an understatement since it can detect a lot more than a Normal linter. It's also a style checker which enforces Pepe roles It's a structural analyzer looking at your Classes and at your special methods and checking if they are implemented correctly. And it's also a type checker so piling works by using a technique called static analysis, which is basically the act of Analyzing code without actually executing it And if you don't use static analysis right now, you should use it You should use any other tool not just piling Because it can really help you in your day-to-day job For instance, you can use static analysis if you have a lot of tests and it takes a lot of time to run You just want to check that There are no obvious errors in your code or stuff like that Or if you have big legacy systems, which you don't have tests at all in fact as My first job experience I work in a company where we have a lot of big legacy systems But they didn't have tests. So using static analysis was paramount for for checking that Every everything works before going in in production and you can use it as a form of Doing reviews. Well, of course, it's not equivalent to a manual review, but It's better than no review at all right so Here's a piece of code which has a couple of problems And I'm going to show how piling detects this kind of stuff for instance If you can see I Imported OS, but I didn't use it across my program and I also define a variable which is just there It's not used of course. This is this is not a problem per se, but It leads to ugly code or unmaintainable code or stuff like that and more important is This block here from line 8 to 10 Let's let's just say that I invented that code Incorrectly and that code will now will not run because it's after an array statement. So basically it's useless code also Applying 10 I intended to call the execute method, but I didn't so that's also a problem And as you can see down down here piling detects this kind of stuff and properly emits that There is something fishy in your code Okay but we can detect More serious bugs than this like using undefined variables or trying to access members that don't exist or Calling functions that aren't functions like calling an integer and As you can see here piling detects this kind of stuff And it's really nice for a static analysis tool to tell you that you you made a mistake in your program And as I was saying Before piling is also a structural analyzer checking that your special methods are properly implemented or that your classes Are written correctly and in this case the dunder exit method Needs three parameters in order to work properly and piling detects this kind of stuff It emits that that warning right another nice Error that piling can detect is this one basically here I have a function and then I have an if statement with with funk as as a Conditions conditional for for the if statement and what I intended here to do initially was to call funk but I forgot to write the parents and This statement will always be true and that if statement if statement if branch will always be taken So if you write this for this for this piece of code you might How to say if you write this for this for this piece of code You might overlook overlook it if you if you didn't test it properly So piling can can help you in this kind of situation as well Take a couple of moments to figure out. What's the problem here? If you know to answer Yeah, really really loud anyone Sorry Var is being changed in the loop. So what's the output of this code? Who say that? Yeah, that's it babe, sorry basically what we did is we created a closure here and Var was looked up into parents scope when when called Yeah, doesn't work and Var was was Looked up in the parents scope and called and for this particular case the parent scope was the least comprehension at least on Python 3 so When the for loop was executed and callable was called Basically Var was the last value from the for loop which was 9. So that's a bug and If you try to do a review you might have missed this stuff and piling can help you Deal with this kind of situation Okay A little bit of history about piling it was created in 2003 by a French company called a logic lab they used to Maintain it for a while for 10 years, but now they are not so involved anymore. In fact, they are not involved anymore in piling development Even though many still Think that about about them, which is wrong I know that Google uses a Modified version of piling internally for their use cases because they had an A maintainer for G piling that was also maintainer for piling and they used to push upstream a lot of changes and some statistics According to oh look net we have over thirty five thirty thousand lines of code Which is pretty big but not big enough because we can detect all the problems in your code and unfortunately Piling is gpl. I says because gpl was really cool back then in 2003 It still is Yeah And here's my involvement with piling to basically I started working on it in 2013 when piling 1.0 was released and up to piling to 1.3 I Become a committer a maintainer the only maintainer and the guy that pushes the thing forward I'm planning to release piling 2.0 in 2016 if my time allows it and I'm going to have some Advanced techniques in it. It's still right now pretty advanced, but it's going to be even more advanced than that Basically, I'm going to use a technique called abstract interpretation where I'm going to Interpret statically your code not actually executing but statically Also, I'm going to add support for paper 484 what Widows talk was about and Changing the things internally like having control flow graphs and having a symbolic execution engine for for it Yeah, right And at this point you might ask yourself, okay, this is cool, but how it works and why does it work? statically First of all piling is split in actually in two components in the real checker Which is pilot and it's inference engine and the component that understands Python, which is astrayed When following the general pattern of building a linter basically we're using abstract syntax trees Even though our abstract syntax trees are augmented with a couple of functionalities that help help us in in building the inference engine And internally astrayed uses the built-in module AST Which kind of looks like this when using it you import parsed and you give it a string and from there you get an abstract syntax tree which Basically looks like this so if you can see at the top of the tree we have a lambda which has a couple of arguments and the blue the blue stuff out there the blue thingy is is a node and the rest of them are just attributes of those nodes so as you can see it's pretty structured and You can just by looking at it. You can reason about code Even though AST is great. It's built-in. We don't have to write our own parser it's not perfect because The it's not backwards compatible not even for minor versions of Python releases like 3.4 and 3.5 basically they they are going to add new nodes or remove nodes or change things out there and astrayed strives to be a backward compatible Rapper over AST that you can use for Python 2.7 till Python 3.5 and as well for Pipe I or gyton or any other interpreter which supports abstract syntax trees and DAPI is quite similar with the with the AST as you can see it doesn't differ too much We have basically the same functions Yeah, so if you're using AST right now and you want something more capable you could use astray and as I mentioned our Astray nodes are quite similar with the one from AST But we augment them with a couple of capabilities Specific for for our purposes Like in this example for instance You can actually retrieve the parent of a over node and you can you can walk up in the tree starting from one node Also, you can walk down by using the get get children method, right? Also, you can retrieve the lexical scope of a variable for instance in this particular case the lexical scope of foo is the function where it is defined and Down there at the least comprehension basically I'm doing foo for foo in range 10 and The scope of of foo at least on Python 3 is the least comprehension itself. So foo doesn't leak outside Yeah, and pilot can actually know this kind of stuff and astray Some nodes are augmented even more or more than that like in this example, we have a couple of classes and We're using a meta class we're using ABC ABC meta and we're defining some slots for for our class and Yeah Let's see what pilot can do what astray it can do with this code So we can retrieve the slots of the class using the slots method or the meta class using the meta class method or Using the MRO method We could retrieve the method resolution order of your classes Which is actually the method resolution order when running Python over that over that script or over that class Which is nice Yeah But the most important part about astray is not the AST itself But the capability of doing inference Basically inference means the act of resolving what a node really represents like if you have a name in that name It represents something like a variable or a function call You want to infer that and you want to see what's at the end of the of the tree for that particular node Our nodes basically Implements the Python semantics for instance a call func inference rule will be the return values of the call func or for list list Comprehensions its inference rule would be the actual list that is returned when evaluating the list comprehension and our inference also does some partial abstract interpretation, but it's partial because It's not working for all the cases in your code Right, and in this particular example, we have a function which adds Two values the argument And as you can see in this particular case When in in foreign what this function actually returns in fact when in foreign, what's the result of the function called? Astray knows that it's 48 because 24 plus 24. It's 48 Yep, this is a more complex example Involving binary operators and if you know The rules of binary operators in Python goes like that if you have Two different objects and you try to add them first the Dunder add method of the left-hand side will be called and if that doesn't return not implemented then I think the Right-hand side object no, no, I think that the Dunder are add method of the right-hand side object will be called And in this particular example, we're having in the left-hand side a Supertype which is a as you can see a is a super type for B because B has as its base class a and in the Right-hand side, we have B which is a subtype of a and the rules are a little bit different in this particular case because if The semantics would be the same as for for the different classes a rule then the first rule then to first method a Dunder add will always be called which is wrong and in this particular example What will happen is that first the Dunder are add method of B will be called and in this particular example it returns not implemented So it will fall back to the other one, which is a Add the Dunder add method from a and if you do the calculations you can see that as read this right The result of that operation is actually 45 Yeah Okay now As for it is great pilot is great, but they can't really understand your code I mean they can't really understand your full code. So we have to deal with this kind of situation and We provide we're providing providing a couple of APIs that could help you like having node Transforms basically we're with a node transform We can modify a part of the AST to be some something else Like you have a function call and instead of that function call you want to do inlining Replacing the AST node with the result of the function call or anything you'd like and we can do that with this API Basically that API is a function Which should accept one parameter that parameter being the original node and should return either to the node modified or a new node and You just register that function with an internal manager So an implementation detail, but anyway, you just register your transform function and when doing asteroid dot parts That the transform function will be called for for that particular type of node in this example, I'm registering a transform for the class node and Also, you can apply a filter function because you don't actually want to change all class nodes in your code We you want to change Something in particular and as you can see There I'm just Giving it filter functions for filtering any other class that it's not six at meta class The same thing can be used for inference rules because At some point really you want to have different inference semantics than the Python Python offers you so using the same API you can you can provide a custom inference rule for I don't know Want to infer these comprehensions differently or you want to infer function calls differently? So you're using this inference inference rule and you're changing basically the semantics of Python in your AST and We're eating our own dog food in this example Not in this example. We're eating our dog food with inference rules because we have inference rules for built-ins as I'll show you immediately Basically with inference rules We're understanding existence is subclass. We're understanding the author has a Type callable list set whatever we're understanding not in particular with inference rules binary arithmetic operations really really Good logical operators or comparisons Also, we're understanding context managers and list set string indexing slicing whatever Yeah As earlier take a couple of moments to see Where are the bugs in this particular code? There should be like three bugs, but if you can find more Join piling sting Also, if you know the answer yell really loud and as a hint when How super works basically the first argument of super specify no the The last argument of super specifies the object From which the method resolution order will be retrieved and in this Sorry Yeah, that's good because in that particular case Super see and self would would be to method resolution order in this particular case I don't know what it is, but super see self will call Be because a doesn't have boo. Yeah, and it's multiple cooperative inheritance if I Not mistaken and be boo will be called and be boo doesn't accept two arguments Anyone? Yeah There's no spa and There is a full Sorry, sorry, there are too many voices one at a time Foo is not cool. Exactly so Piling detects this stuff because asteroid knows Python really well And as you can see what what piling says there are too many positional arguments at line 14, which is actually true Foo is not callable of course because who is an integer and Super V has no spam member, which is actually true because it's spam not spam Yeah Here's a more complex example of asteroid capabilities and as you can see here asteroid understands List indexing understands has author callable yet after and at that particular line where met is retrieved what will happen is that has utter call will return true because a has the method called met and a met is callable and Get out from get the met from a will return the met object so it will be like true and true and a dot met and In this particular case the last value, which is also true will be returned so met will be a dot met and then the context manager is invoked and The context manager will return a real funk which has no argument whatsoever And we're going to call that and piling will say that hey you use too many positional arguments in your method call You should change that so this kind of stuff that asteroid knows about Python code and Concec and Piling knows knows about it as well Yeah I don't know. I'm going really fast or How much time do I have? Okay, thank you Piling is not so complicated asteroid is the most complicated thing Basically, it's like a fancy walker of the AST and it has a couple of patterns more than a couple because pilot can detect almost 180 type of errors or Verifications And basically we are we are using the visitor pattern to visit each node because that pattern really decouples your data structures from your algorithms and A small example of how visit Visitor pattern works Let's say we implement in our own checker visit call funk And in this particular example, we're importing collections and we're Obtaining the default attribute from collections, there's no default attributing collections by the way it's default dict And what will happen is that visit get author will be called with the with the node which will be the get author node Afterwards we'll infer what that node really represents. I mean what the parent of the statement really represents the parent will be The name the name collections and we need to infer it because we should know what that node represents at the end of the AST Afterwards we're we're calling the data through method with the result from the inference And if that doesn't raise Any error such as not found that yay collections has default to default attribute Else we're going to emit a no-member error But before that we're going to have a couple of Filters like it's the owner is a mixing class because you might not have that attribute in a mixing class or I Don't know what the owner is a class with unknown bases like Bases from extension modules And we can't really understand extension modules by the way Yeah I'll go really fast from now on because I don't have much time left Basically Abstract interpretation will have piling for for this kind of code in this particular example The dunder date is updated with the dictionary of I don't know what kind of arguments will be passed and the at line five We will call some arguments set in dunder in it or whatever other attribute and that attribute might not exist and What happens right now is that piling says hey you don't have that attribute But we're actually having that attribute because we just inserted it at line two so That's what abstract interpretation is going to fix because you will just interpret every line Every logical line in in your code and it will know at the end what side effects each line had okay, we have multiple type of checkers and errors such as conventions for Pepe tools refactorings Various warnings which are necessarily bugs in your code and actual bugs like Like no member or not global things like that and we have two types of checkers a C based and Token based the token based checkers are for I don't know Line is too long or bad the indentation or other similar examples Piling has a really vibrant community. There are a lot of External packages for improving in furnace for improving piling checkers out there You could write your own if you want to as the problem is that They are pure Python so you need to write Python code in order to have a custom checker Yeah, it comes with a couple of features Extra features like you could generate you a UML diagrams from your code or you could spell check your doc strings or comments these are disabled by default or You could use the Python port Python 3 porting checker, which is a checker when when activated All other checkers are disabled and it will emit stuff that it's not going to work in Python 3 anymore like using remove syntax or Remove built-ins or special methods or my favorite is using map filter or reduce in non-interval contexts As you can see in this particular example on Python 3 map is lazy evaluated and In this particular example download your role will never be called because at least on Python 3 because You should evaluate it first Yeah, it will return a generator stuff like that Okay, oh There are a lot of similar tools like pyflakes my pie Pie checker pie checker basically is the forefather of piling And I should say a couple of things even pie checker is now defunct and dead for a couple of years It was way more advanced that many of static analysis tools that exists currently in Python As you can see in this example We are unpacking three items in two variables or we are having a constant check or We're catching an exception, which is not really an exception through and pyflakes as you can see detected almost nothing While pie pie checker detected all of these errors. Yeah, there's also Jedi my pie Okay, yeah Final note is that users at least a part of them actually expects this kind of code to be understood but Really that's not actually possible. So if you want static analysis tools to understand this Just don't do it. Yeah Yeah, and this is basically the future of piling We'll have piling 2.0 next year. We'll have full flow control and analysis better data model like understanding the scriptures or Having the proper attribute access logic it's not the same in piling as it is in Python and I'm interested in bringing more contributors into the project and My final slide is Hey, what your Talk title is how do I stop worrying and start loving the bugs and so on Well piling helps if you if you're going to use it and if you were going to write as many tests as possible So use piling Thank you I've got any questions or many questions Hi, I actually have a comment or a word of caution and the question The comment is if you're going to start piling It will take you some time to configure it to make it shut up about things which don't interest you But it is well worth the time to do so And the question was how did you get into? this kind of parsing Python and ASTs and I mean I tried to write piling plugins and because there isn't really much Documentation for piling the best rate it found it's very hard to get into this topic I was a user of piling before and It had a bug like it didn't detect Stuff like it didn't take unbalanced couple unpacking like unpacking three items into two items and I had a bug in production with that stuff and I just wanted piling to hey tell me about this So I implemented that check and I started from there Okay. Thank you One other question. What are your thoughts about pylama? Pylama, yeah, if I'm not mistaken is a rapper over over multiple check over multiple linters like pilot and flake eight Yeah, so it just takes pilot and other stuff. I never use it More questions There's one here I'm not sure what is can you be more visible At the front again Hello, so regarding the the the first question the most common complaint complaint about piling Is that he has too too too many checks active? So maybe What if there will be like some sort of configuration wizard that will show you examples and rescue do you want this or not? Yeah That would be nice to have but I don't have time to write it myself I Sorry, yeah That's the question is there a sprint and piling there is there is Saturday and Sunday Thank you, and can you join if you're a beginner? Sorry, can you join to the sprint if you're a beginner? Yeah, of course great More questions. Well, I think it's fair enough. Um, they thank you again Clujie