 So, hello everyone and welcome to my making your Python code, write your Python code presentation. It's very nice to see all of you. Thanks for coming. Let me introduce myself. So, my name is Marcin Sopczyk and you can reach me at this email address. I work for Red Hat and I'm part of the over project. So, who is the stock for? If you've ever done anything with the AST module, you might be bored, but otherwise I think you'll find it interesting. So, abstract syntax trees, that's today's subject. But why? Why would we even bother? Well, who knows PyTest? Yeah, almost everyone. So, a very simple example. We create a class definition using named tuple and then we have a test. We compare two of these, whether they are equal or not. When we run this with PyTest, it shows us the names of the fields. It shows us which field actually was actually differed. How does PyTest do all that cool stuff? The answer is it uses ASTs to transform your assertions into something completely different. So, abstract syntax trees allow us to analyze Python code, transform Python code and then generate and execute Python byte code and all of this in runtime. And today, we'll try to write a decorator like this one. So, a decorator that will measure the execution time of each line in your function. So, if you want to find out how to write something like this, stay with me. So, how are we going to do this? Well, first, we need to know what ASTs are, then what do they look like, how to create your own AST, then how to make byte code out of an AST and finally write the tool. Simple as this, right? So, what are ASTs? First of all, they're trees as in tree data structure. So, we have the root of the tree, we have some nodes and we have edges between them. Second of all, they're syntax trees. So, syntax trees or parse trees, same thing, is a family of tree data structures which are created by extraction of information from some structured text, Python code in our case. And abstract, why abstract syntax trees? Well, that's because once an abstract syntax tree is created, it doesn't really care that much about the syntax. It doesn't care about the columns, it doesn't care about the indentation, it's all about the logic that's behind your code. Okay, so now we know what ASTs are. Let's find out what they look like. And to do that, we will use two tools. The first one is the parse function from the AST module. This one converts a string of Python code into an AST. And the second one, the pprint function, comes from an external library called AST pretty. And we will use this one, you know, to print the tree in the console so it looks nice. Okay, so let's parse some code. So, if we take a look at the parse function signature, we can see that it has three arguments. The source argument is, of course, the string representing our Python code. Then we have the file name argument. And this one is actually only a helper. So, if we do something stupid like this, we get a traceback, right? And the top of the traceback points to a file named unknown. So you can see the correlation between the default value and the file name, right? So it turns out we can type in here whatever we want. And this is our file. Okay. So, there's also the third argument. So, parse function can operate in three modes. The first one, default one called exeq, will give us a module like a Python module. Okay. The second mode is called eval. And this one will give us a single Python expression. So the question is, what do we get in this case? Yeah, we get an error. Why? Because an assignment is not an expression in Python. It doesn't produce any value, right? We get a syntax error. So, let's fix our code a bit. And there we have an expression. So, there's also the third mode. And this one is called single. And this one produces an interactive node. So, that's something we get when we type a command in Python interpreter. But we won't, it's not interesting for today. Okay. So, we did some parsing. Let's check out. Oh, no. So, so far we actually only saw the roots of the trees, right? There was only the expression and the module. But we didn't see the whole tree. So, to see what the whole tree looks like, we will use the pprint function. And we will try to analyze something as simple as this, like assignment, just assignment. Let's do this. Okay. So, we can see the whole tree printed out. And we can see some line numbers, column offsets. And these, of course, point to the places in the string from which the node was extracted. And regarding the nodes themselves, going from bottom up, we have the num node, which holds our integer value with the n attribute. Then we have the addition operator. Then we have a name. Actually, we have two names. And these names identify our variables, right? X and Y. And they differ in the value of the CTX attribute. And that's the context attribute. So, we can see that for the left hand side values, we will have the context attribute set to store because we're storing the value. And for the right hand side, we will have the context set to load. Going further up, we have the assign node. And the assign node is the assignment operation. And the assign node has the value attribute, which points to our binary operation. And it also has the target's attribute. But there's something weird about the target's attribute, right? Question. Does this work in Python? How does it work? We just said a moment ago that an assignment is not an expression. So, it turns out that the target's attribute is actually a list. So, we can have more than one name to assign. But that has some interesting consequences. In languages like C or C++, where an assignment is an expression, it does produce a value, we can do something like this. But in Python, we can't. That's not even valid. Python code, these brackets look so innocent, right? So, multiple assignment in Python is actually a syntactic sugar. You can see that even by analyzing the simplest snippets of Python code, we see the clever ways in which abstract syntax trees implement the syntactic features of Python we all love. Okay. But we're done with analyzing. So, we know what do they look like, roughly. So, now we need to find out how to create your own AST. And we will try to recreate an AST for something as simple as this. Two is greater than one. Okay. Let's do this. But before we begin, another question. Does this work in Python? Yes, it does. Okay. So, you won't be surprised in a moment. So, two is greater than one. Where do we begin? Well, first we need to create the numbers. So, we have number one. Then we have number two. Then we need the operator, which is greater than. Then we need the compare note. And the left side is number two. And then we have a list of operators. Not a single one, but a list of operators. So, it's OP. And then we have the comparators. And that's number one. Okay. So, we still need to wrap the compare note into an expression. Let's do that. And the body of the expression is compare. Okay. We got our expression. So, let's try to print it out. Yeah. So, we've built an AST by ourselves. Okay. So, now, so, now, let's try to convert that AST into byte code. So, how do we do it? Compile and eval. Okay. So, we need to compile this expression. So, there's this built-in function called compile. And if you look at its signature, it's similar to the signature of the parse function, right? There's the source, there's the file name, there's the mode. So, it turns out that the parse function is actually only a wrapper around the compile function. The difference between those two is that the compile function don't have any default values. So, we do have to provide the file name and the mode by ourselves. But the file name is still irrelevant. Okay. So, let's try to compile our expression. And we will use the single expression. And we get a type error. Required field line number missing from expert. Well, yeah. When we were printing out the tree before, we saw the line numbers in column nonsense, right? But does that mean we have to, like, calculate them on our own? That would be crazy. Fortunately, we don't. There is this function in the AST module called fix missing locations. And it does that for us. Okay. So, let's try to compile once again. And we get a code object. Great. So, what do we do with the code object? We evaluate it. And there we have it. True. Great. So, we've just executed our first AST. So, we got this one. So, the only thing left to do is to write the tool. But before we do, I would like to introduce you one more class. And this one is called node transformer. So, the node transformer class looks something like this. We have the visit method. And the way we work with node transformer class is we define our own class which inherits from the node transformer. Then we define the visitors for the nodes we want to visit. We call the visit method from the base class on the AST. And it works like a dispatcher. It just takes the node, checks its type, and calls appropriate visitors. So, what we return from the visitor will replace that node in the tree. Okay. So, we're ready to write the tool. Let's simplify the problem for now. Let's say that we don't want to measure the execution time of each line. We just want to measure the execution time of the whole function. So, that's our function, benchmark me. And we have the benchmark decorator. And that decorator has a wrapper inside. So, the wrapper inside measures the start time. It calls our function, measures the execution time, and prints out the message. Right? So, we actually need something very similar. But we don't want to benchmark the whole function. We want to benchmark a single expression. So, we actually don't need this wrapper. Let's delete it. Okay. And what will F be, the F argument? So, the F argument should be something like lambda, print, fuzz them. That makes sense, right? If we have a parameter less lambda and we run it here, then we will measure the time it takes to execute the print function. Cool. So, what we want to do with our function is we want to go line by line and replace each line with something like this. Lambda, print, yada, yada. Makes sense? Great. So, let's do this. Okay. We still need our benchmark decorator. And let's write a dummy method for now. And let's see if that works. Yeah. Still works. Great. So, how do we make the new function? First step is to get the sources of the old function. We do that by calling inspect get source. Okay. Let's see if that works. Yeah. We got the sources for our function. So, there's one thing, though. That function is still using the benchmark decorator. That will be an issue in a moment. We'll see. Remember about it. Okay. So, we have the sources. What do we do next? We create the ASD of the function. Okay. Let's see if that works. Yeah. We've got the ASD. So, what do we do with the ASD? We transform it into a different ASD. And, again, let's write a dummy method for now and see if it still works. Oops. Transform is not defined. Transform. Oh, yeah. Still works. Great. So, now that we have our, let's say, transformed ASD, we need to fix the missing locations, then compile the whole thing. The file name, again, is irrelevant. And we're making a whole module. And then that will give us the code object for the new function. So, what do we do with the code object? Previously, we were using eval function, right? But this one works with single expressions only. If we want to define a function of the fly, we will need to use the exit method. But there's one issue with the exit. Right now, we're inside the make new f function. So, if we run exit here, it would be like defining the function in here. So, the issue is that if we call it down here, we won't see it because it's like created in the scope of the function. So, there's a dirty trick. Yeah. Let's trick exit into thinking that we're actually in the global scope. Yeah, that should do the trick. Okay. So, right now, after calling the exit, everyone, like the global scope, will see the new function definition. But the f variable still points to the old function. So, how do we get to the new function? Well, we can refer to the globals and just use the old function name. And that will be our new function. Okay. So, it's time for the issue I've mentioned before. Let's try to run it. Yeah. And we get a stranger. Source code is not available. Why this is happening? So, what we're doing right now is we're in the middle of decorating a function. And we're trying to create another function that uses the same decorator. So, actually, we're like in an infinite loop. So, the only thing that prevents us from like hanging forever in that place is the fact that the get source function fails because there are no sources for the function that we just created on the fly. So, we will have to fix this. We will remove all the decorators from that function. And this is where the no transformer class kicks in. So, let's do this. So, right now, our new function should work. Let's try this out. Yeah, it prints out for us then. So, right now we're executing the new function that we've just built. There's only one last thing to do. We need to change each line of that function so that it calls our benchmark expert with a parameter less lambda, right? So, let's do this. So, what will be the new expression? Right now, denote is an expression that has a value of the print call. And our expression is also a call, but it's a call to the benchmark expert function. So, we need to create the name of the function. And the context is load. So, we got the function name. Oh, yeah. So, we need the call. Yeah. So, let's define the call. So, it's a call to a function called benchmark expert. What will be the argument? The argument, we'll have a single argument, and that will be the parameter less lambda. And we also have to provide the keywords. This one is just mandatory. I can type lambda because that's a keyword. So, we use just, you know, a name for the variable like this. Okay. So, let's define the lambda. So, it's a lambda that has no arguments and has the body equal to the value of the node we are visiting. And we're done, I think. So, let's try this out. Okay. Looks good. Let's try to add some more stuff in here. Yeah, it works. Right? So, that's all I have for you today. Questions. For each question, I have an overt flashlight. So, you can get some. No questions. Okay then. Thanks, guys.