 So hi everyone, and this is my Twitter account and this link here that you can see you can already download my All the material I'm going to show that will be the the code I'm going to show and the slide source So you can have a look and play around you're welcome to open Python and Interactively try the examples. I'm going to show you because that's what I will do as well So I come from Italy but I live in London and I work for the company called depop. This is just a small advertisement slide where I tell Just telling you that we are hiring. We're looking for API developers and If you're interested just have a look there depop.com jobs. We're using Django Sellery all nice things MongoDB Postgres, etc, etc All right, so what are we going to talk about today? So first of all I try to define what is metaprogramming and why is useful and Then we are going to talk about least a little bit, but not too much. So don't be scared and Then we're going to talk Python about Python and how some techniques we can use for metaprogramming in Python So this is the definition. I found Wikipedia about metaprogramming. I don't know if it's the best one But let's try to go with this one Metaprogramming is the writing of computer programs that write or manipulate other programs or themselves as their data Or that do part of the work at compile time that would be otherwise than at runtime So this is kind of still not very clear But I think we're gonna see now in some examples how this what this really means I manipulate the program itself Manipulate themself. Well, I don't know. I'll show you later what that means So why on earth would you do that? That's the most important question because that sounds really magic and really strange There are some scenarios where this really makes a big difference and normally it's used it's not really used for performance reasons or for Making a better product is maybe it's normally just used to be closer to the domain you're working in So you write an abstraction in a more powerful way and in the end you you manage to minimize the line of code you have or you manage to Make the old program more close to your domain and more easy to understand for the people in that domain So what's the cost of that? Well, if you do it? Well, nothing if you do it wrong You might screw yourself up a little bit So next slide just as very small introduction to Lisbeth, so he was invented by McCarty in 1958 so there was a Very long time ago. It's the grandfather of languages in a way and what people think about it is that there are lots of parentheses That's how people see it normally There is a reason however for all these parentheses so and then going to try to show you What it is? This is the simplest possible well, the simple factorial function and This how is the fine? There is a special form the fun, which is the way of defining functions Which takes an argument which is the name of the function the argument of the function and then there is another special form if which takes three arguments the The conditional the then and the else branch here and also all these things are expressions as expressions to be precise I'm going to do a lot of kind of live coding. So maybe I'll try now for the first time to just show you how things actually work So we go in Emacs here, of course and I define this is the repler for SBCL, which is a common Lisb interpreter and Then I hear a defined factorial. I Computed everything works fine Interesting thing to note is that even if I give a very high number it doesn't blow up like Python would do Even if it's a cursive because it's It's smarter about recursion than Python So let's continue so Lisb has very small very limited syntax and It's another property, which is quite uncommon and not of many languages, which is homo-iconicity that's a very difficult word that Not easy to define but in practical means that the program itself is a valid representation of the data So here for example what we have is the computation of the Product between the sign of one and the cosine of 2.3 And what this actually means is that it's like setting an expression to a list where the first element of the list is a symbol Star the second element is another list with sign symbol sign and one-on-one and the third is another list with cosine and 2.3 And then you evaluate that expression So why is homo-iconic that that word? This because you can actually Transform and pass around a program and manipulate it easily as it was data So to make it more maybe more clear I try to say how is Python homo-iconic, but no, but what if how would it look like if it put people sample? That might be something. I don't know if that makes sense But so suppose you can write suppose this is valid Python This is a way to define a function you oh you create a list and Then first element of the list is def then you have function then you have arg and then comma pass So this is a list and this is also how you define a function. It's not true, of course But that's how what it would look like if it was So why this in this thing is so important well first this is just to show how devaluation is done in Lisbon just to keep it clear as you know every operator is in fix and Then everything is a cons in the end. So you have here The star the multiplication sign then you have the two then you have another Branch where you have the plus three the four and here nil a nil because there is no other element in the cons So why is this property important because in Lisbon you can make make a programming to a very Deep level you can make using macros. So assume Which you may be ever seen but in Lisbon to set To do an assignment of a variable you do set q x 10 for example, which sets to the symbol x the value 10 suppose I want to do something like Set 10 to two variables at the same time like assigning two things to one value in one Call is that possible? Well at the moment is not possible, but what if I write a function? So I write this function the fun set q f x y z and then I do this I use this special form progen which concatenate to multiple Expressions but within them But that doesn't work. So we can try it out See what happens So now I do set q to f Do a a well a b and I say 10 it fails. This says the variable a is unbound So well how I was all that but that's kind of understandable in a way because I'm trying to set To create a special form with the function. So that's not really possible. So however If I rewrite this thing with the macro Where the only difference that here I use def macro and here I use this back quote and these commas here in front Instead everything works fine See how they find the macro And then I do set cool to a be a Be everything works. So what's the difference the difference seems small, but it's quite relevant because here This macro is basically a creating a compile time a new A new function that will get be templated when you call it. So this is like It takes the symbol it doesn't try to evaluate it doesn't try to evaluate a or b It just pass it around to this Quoted thing which then gets evaluated at runtime and these runs normally as you should So a very powerful technique. Can we do the same in Python? Almost well, I mean for to do this kind of thing you could do something like that, which is horrible and Yeah, almost works But so basically you pass two strings like a and b and then you pass an expression You evaluate the expression and then you update the globals with this thing Yeah, it's not even the same exactly, but it kind of works But yeah, don't do that because So, yeah, the least pre introduction is over like even we can pass to something more interesting Python So what can we do in Python for metaprogramming? Well, we can have function decorators class decorators Meta classes and then if you really want to be fancy You can manipulate the is the astra syntax tree. I'm going to show them in order. They are also in order of Difficulty in way What is decorator first decorator is just a syntactic sugar for a function that modifies the behavior of another function or a class so this Simply means this below You you do a at decorator or as the function and the same thing as defining the function and then redefining Assigning it to the return result of the decorator itself and Let's see the first problem that we try to solve with the decorators Assume I want to measure the time spent in a function and print it out to the standard output and see how long everything takes This is decorator that does it. So it takes the function He uses this nice wraps decorator Defining in func tools and this is just to make sure that all the like underlying Attributes like doc and so on gets passed correctly and they don't get lost along the way He would work anyway. It's always good to put this So, yeah, and then you define as an inner function which will call the original one store the result somewhere Store the time before and after and then print out The time spent and return the original result and then here you return this Inner function Can show this how that works So now we go to the shell I Open my python. Can you see in the back bigger? Maybe Okay What was it called? Creators by print. Okay. So now I create another function time it's print There's something time me. So I do anything useful Then I call this and I get this message here function time me took They should have said something else. Ah Because I changed before this the talk of course and I made a mistake No still the same Yeah, yeah, yeah, thank you. All right. I Can reload the decorators does work. No What? No, just Okay, let's do that. I showed you the next example is more interesting Okay, so we have another problem I was still with timing But slightly different because I want to know when I when I mean by anything something new like a new feature Or I change algorithm I want to know that a new one runs faster than the previous one But I just I don't want just to know it once I want to make sure this always like that So I want to have that running in a test Somewhere like integration test and say my new fancy algorithm is still faster than the crappy old one After six months of hacking on it and adding things So well, I mean in theory with the previous approach we couldn't do that because We were just printing it out so we can't just check all the time how things look like But with another decorator which also changed the signature of the original function then we can do that So what I have here is a test To check the the first implementation is faster than the second one and the way it works Is that the new this decorator here will change? the function return values and add the time spent in the function as a second and as a second Return value and so I do this I do this for the second one And then I can check that the time of the first one is less than the time of the second one So now let's see how we actually implemented them So Still quite simple Actually exactly the same almost the only difference is that here we When we do the return we do we also add the time here So this is an interesting case and the difference is that you can't use it in a import time way because if you If you decorate your function in the module like that then it will mess up everything because you don't want you don't care about the time spent in production of course So you can only do that in this way, but it's perfectly valid thing as I was saying since it's just in touch sugar to just Replace the function at runtime in this way and then call it. So let's see if this I have more luck with this one So Oh, yes, that works. So, yeah, I see I get a tappel and The first element is the result and the second element is the time spent Very nice To actually see how things work. It's interesting also to look at the code here of time me and Yes, and the wraps function did the nice thing to also make sure that the source code That is returned is the original one not the wrapped Not the wrapped in the decorator All right. So another example. This is how you can decorate the classes then Suppose I want to add always the same method to many classes and this I mean I don't want to go in every class and change it and I don't want to make a super class to May to make this. This is not a very simple way to do that You just this is the test where I have a class Which adds a response method and then when I essentially this class and I call the response I get the return I want the value I want So this is the simple way you do it very clear simple. You just take the class here You assign to response an anonymous function, which takes self Elec turns this and then you return the class itself Yeah, that's all for the credit also now. So let's pass to meta classes meta classes is the next Way of doing meta programming in Python. It's more powerful and there's more It's also a bit more difficult to understand in the beginning, but it's really not that hard as hard as it looks as I'm going to try to show you so what's a Meta class well meta class in Python is just the same as a Meta class tends to a class like a class tends to an instance. So it's a type of a class If you create a class in this way here is exactly the same thing as doing in this way with the type function I'm going to try to show you now a little bit For a sample if I now do this I Forgot to mention in the beginning that all the code I'm showing is a Python 3 only well the difference does not many but yes, if you try to write it on Python 2 It will my frame or might not do what you want Okay So what's the type of C? Type and then if I try instead to create a type Let's see what this takes as I can make it takes an object or name a bay list of bases and the dictionary So for a sample suppose I want to create a runtime a class like B We has a as basis all the object and then as dictionary I have Like a class attribute a 10 Must be a tuple. Yes mistake Okay, so this is now class and I can instantiate the class and I can access the attribute a So this is another way of creating classes at one time and this is how in the In the background it works Let's see the simplest possible meta class implementation and how to use it So we have a class simple meta which subclasses type and Then to use it in our own class. We only need to create class Subclassing object and then passing meta class equal to simple meta here in Python 2 it would have been Undescore and score meta class here equal to something, but it's just the same thing and now Let's pass with some more interesting examples. I think In every ORM or every jungle framework or something it's very easy to define models that talk the database store data keep some Information about the type and so on and so forth and this is kind of possible Syntax that is used which I just made up, but this is quite similar enough. I think and It looks like Magic, but it's not that hard to do in reality and There's simple way to do it in this case if I want to Accomplish something like that if I don't have meta classes I could just have a model for this particular case which takes the argument X and Y and then initialize them right there So well, I mean if you you can do that if you have some very simple scenarios But if you want to define many models If you want it's very verbose because every time you have to declare everything and assign it There's a lot of boiler press and also the very important thing I think is that you have no information about the types and While Python is duck type Databases are normally not so you may still need to have the information somewhere So how how can we accomplish this kind of very nice thing in Python then? We start by looking at how we can implement a fill the type So which is this one? so a fill type will be a base class for things like integer or string and You can implement it. So this is the test that will this the requirement for this kind of type I want to have three different things I Want to be able to I'm not this does not write test. Okay. This one is right test. Oh Sorry, sorry. No, take it back. This is the next way So this is the the model I want to have and these are the three Requirements that I have for this model to behave like I want so I want to be able to create an object Like this passing for a sample Y And without passing X X as the full value Y doesn't have one So I checked that after I pass this a Y is set and X is also set to the full value I want to be able to know that if I pass something wrong like unknown. Hello It will give an exception and if I pass something of the wrong type It will always given a type error in this case So first of that we can look at how fields are implemented and here again this requirement for the fields So if I create an integer where I pass Zero as value and the fold one the value will take the precedence if I try to set at any time Something which is the wrong type I get a type error and also very importantly if I in the beginning create The element correctly and then I change the type to something else And then I also get a type error Let's say an implementation of fields first Which is not you done using metaclasses, but it's not done in this way So they need the most important thing is that we have a property here, which has a Also a setter and This property is the actual representing the value and Whenever you try to set here the value, which is also done at this point. It will always check the type He checks if the type is not an instance of this base type, which every class Will have to define then I raise a type error Let's see how is for example the implementation of string or integer So very simple here we have The base class Here is integer and here is string so that's all all you have to do is to find the base type and if the Object your passing is not an instance of that then it will give a type error Yeah So let's go back now to the model the models metaclass And this is the actual instead the solution for all this which fits in the slide So it must not be that difficult is the solution for the original problem The way is done is that I subclass type as I've done before in the previous example And then I I override the underscore underscore new method, which is used to During the construction of the class itself instead of the initialization initialization of the object like in it And Then I get four arguments here. I get the class. I get the name I get the list of base classes and I have this important the class dict which is all the the Attributes and methods of the class basically which we can modify So the way The way I want to do this is basically just replacing the in it of the new class that I will create now with another in it that will do some extra checks and extra Thanks for me automatically every time first of all I loop over the attributes of the class and They check if they in if the attribute of the class is an instance of a field Then this is a field. I just stored it in a dictionary here and then I redefine an in it here Which takes some kwx so it can take x equal 1 y x equal 2 and so on and so forth and Then it will first of all check if there are some arguments which I pass in that are not known so if they set of the difference between Decade the arguments past and the fields that I found looking inspect in the class is greater than the zero then given exception Otherwise, if not if everything is fine, then loop over this and then generate Set an attribute to this object self with the as a key the the name of the Argument and there's a value the value which I just now Created by calling the class and passing the value So sounds more complicated than this maybe but I try to show maybe now How it works and then after that the most the important step is that I redefine the in it of the class take to the thing which I already defined here and Then I return Just a super call and Trying to show you just very simple how this behaves maybe So if I now yes Now the only thing I need to do is to define the model in this way class model yes, and then I will simply If I define now another model simple to Let's say for for model and then I define x equal to models string Okay, now suppose. I want to create an instance of this, but I pass the wrong thing Complaints and no arguments. Why if I pass This it checks the type and this another if I pass this instead Everything will work correctly. Oh, sorry, and the interesting thing if if I look at the unit of this method It tells me where actually it comes No, sorry if I look at this But look at the unit of the class simple to it tells me that it comes from here function models meta model new locals in it So that's where it finds it Another is another example, which is a bit more simple Is how to for example make sure that we always call the super of the unit in the subclasses of a Superclass that we define I've seen this doing for example in the module threading dot pi if you forget to call the unit It will blow up and complain and if you look at the implementation It's basically just this it has an initialized false and in a niche in it It will set initialized through and in every method that has to be Working on a initialized object. You did there is this check So that's a bit annoying and that's a bit too long maybe to do so. This is what I want Is that I call or something? And if I forgot to call the super in the unit it should Give me an assertion error And that's it. That's the implementation. Very simple. I do I store the original in it somewhere and then I Call it here. I will replace the unit with another in which I'm doing I call it here just to make sure I don't lose anything and then I also add in the set an assertion Looking for a flag variable that is set So this is kind of doing the same thing but is in a more generic way And so all you have to do in your class to use this is to just go Here I Define a class base passing the meta class check in it and all I have to do is to define a flag variable Equal var and this is it a variable that I know that I always set in the in it Here so if something tries to subclass base and forget to call in it this here Will be not set and then this Will give an assertion error. Yeah And last thing on the show very quickly is that just you can do even much worse than that if you want and This is a very simple function that Analyze the AST of a given function passing it passing the source and then trying to find out a Particular value a string and replacing it with something else But to show you the actual example is probably easier to understand this is This is the function here, which I want to replace around time and what I want to replace is to remove the hello from the string So and this is what is doing. I call it. I accept this Return of this and then I call it again So what is going to happen? Is that Anna? Yeah, the first time it calls with aloe world the second time all you want Well, everything works fine So the thing is that of course, this is very hard coded and this not really The good idea, but in theory AST is just a tree so you can traverse it in any way you want so you can look for all the things that look like strings that have this content or you can look for all the You can do anything you want and you can put that in a decorator or you can and in this This is the only way. I think that you can reach something like As powerful as Lisp does in Python and there is a library which I'm going to I don't have time to show because I knew it was too much There is a library called macro pie where you can actually do things like that and this is valid Python Even my syntax I like that doesn't think it is But it will run and it will work and you can make a case class where you declare a class like that passing X and Y and this automatically will create in it with X and Y and you can just instantiate it like that and everything will work correctly and that's how it's implemented So if you want to look it up, it's called macro pie So, yeah, that's it for me So any questions and then just the conclusion, sorry Well metaprogramming is fun You can make in just one fitting one slide the code to create a very simple style or M and then just be careful But yeah Yeah, yeah, sorry if I knew that the maintainer might be a psychopath and serial killer would I still be using metaprogramming? Yes, but I would comment it maybe leave out my name or Or change the history of gate to make sure doesn't come to me. Yeah Another question I think the philosophy of Python is that you shouldn't do things that are too unreadable and too hard to understand So it would kind of go against the the language Design to to add more and I don't know what really you could but there are things You can't do crazy things like macro pie does so it is possible already It's just I Think it's enough and I wouldn't think you need more than that