 is Python 3 metaprogramming macros madness and more the session will start at 4 p.m. so I guess we are like five minutes remaining for the four right yes so we'll be starting in five minutes or the I would ask the people who are sitting on the first floor if it is possible for you you can just come down so that the the PPTS or the promo thus the slides or something would be visible if you are comfortable in there and then it's okay but okay so we are starting with the session the session is on Python 3 metaprogramming macros madness and more the speaker for this session is Suhas S.G. Suhas is a data scientist at Grammar and was previously an engineer at Verisign so I would request Suhas to start the session cool thanks for those of you who were here five minutes before sorry that you caught me red-handed editing the slides in the last minute okay let's start metaprogramming and all my slides are I've given the link to these slides on my Twitter wall so it's at j-a-r-g-n-a-r and if you go to the link and click on that link that's on my Twitter wall you can follow my presentation live as I change the slides and do follow me on GitHub as well j-a-r-g-n-a-r this talk I hope you had good tea and coffee because not much slides and a lot of dirty coding I hope there will be no errors let's go so what is metaprogramming so metaprogramming is writing code notification cool metaprogramming is code that manipulates code so traditionally code is you think that code is something that manipulates data but then metaprograms are programs that manipulate code themselves so a classic example of a metaprogram is a compiler itself so what a compiler does is it takes code from one source language and converts it into one of the source language so compiler is a classic metaprogramm anything that involves manipulating anything that involves inspecting anything that involves doing something with code itself and not data is a metaprogramm but why do we need to metaprogramm why do we need code that should manipulate code can't people do can't people do it themselves good question let's look at why we can't so the biggest reason is people are different and what I mean by this is my thinking is is invariably different from your thinking and let's see let's see an example of how people can be different cool so before going to the tab that's on my right there a good indicator of difference in people is Google searches right so if I go to Google.co.in and if I type how do I type something Google gives me auto suggestions and these auto suggestions are Google kind of puts these auto suggestions to the region and what people are thinking in that region so if I go to Google.co.uk and search how do I do something it's going to give me different set of auto suggestions so we kind of did a hook into Google and saw what people are thinking right so if I type in how to in Google.co.in Indians want to know how to kiss if I do how to in Britain Britain British people want to know how to make pancakes the United States the Americans want to know how to tie a tie you can explore right you can see what people are doing Indians want to know if Indian girls want to know if he loves her and the Britishers want to know how to renew their passport we have different priorities in life clearly so what people definitely think differently right so one of the example of how people think differently this is a recruiter search that we at Grammar have built earlier many of you might have seen this in some of our earlier talks but let me go through this what this is so it's a network graph of all the people that are on GitHub in a particular region for this instance we've taken a network graph of all the people who are in Bangalore and the color here represents the language of their choice so we see a lot of reds in the middle a lot of greens in the middle so that's a lot of popularity for Python and Ruby for example so the software ecosystem is diverse this choice of languages are diverse the choice of ecosystem is different everything is different right so people are different right why do we met a program people are different another reason it's ridiculously fun to do and let's see how that is today let's have some fun cool directly to code now functions how many of you have written Python functions I'm sure most of you have written Python functions this is a classic Python function what it does is returns a sum of two numbers quite simple to understand so everything in Python is an object when I say everything everything is even function is the first class object so what this means is regular operations that are possible on an object like addition subtraction everything is possible with the function to yes it is possible so function is a first class object and we'll see how it is possible so keyword odds and args so for those of you who aren't aware functions accept args star odds and star star quads so what they are basically is when you have add 2, 2 and as float equal to 2 here basically as float is taken as the keyword arg 2, 2 is taken as the args so you can for that same function add if you pass as float is equal to true it's going to return it in a float so it checks if as float if the key as float so this is the key right so it's a classic Python dictionary this is the key and this is the value so it checks if as float is in the dictionary keyword arcs it's going to return a float of sum so sum of args basically args is a list so I can just do sum of args and it's returns me a sum so far so good functions can be passed functions can be returned so the first example up there is apply func which takes a function takes a set of arguments and then it returns it basically returns that function applied on itself and then there's closure so what's closure so I have a make adder and then I define it's a higher order function and then I define a function within that function and then return that function so you can do some things like add three equal to make adder three and then add three of five add three of ten it keeps on adding on to that so if I make add four is equal to make adder of four and then do add four or five it's going to return me nine and so on cool quick exercise let's go to coding let's write a function that only executes functions whose names are cool okay so this keyboard shortcuts has me windows 10 cool this is a console and let me open a quick editor wait let me go to the live directory and let's say create a new cool func only dot by right so what we do is define a function no cool funcs only and which accepts a function and arguments you can see that right let me increase that one cool let's say it accepts these two and then there is a property in Python three that's called underscore underscore qual name before I go there let me quickly show you what qual name does right so some function let's say oops I'm not good with the regular python shell let me go to the cool so what qual name does is basically it returns me the name of the function in the string format itself now this we are starting to meet up program already we have gotten the name of a function so now we can use this to manipulate stuff right so I can check the function that is passed here but how do we decide if a function is cool or not so let's write a function for that def is cool it takes as func string func name let's say it takes a func name and functions are only cool if if they start with Python as a prefix let's say all right so we have a function which decides if a function is cool or not so I check if is cool function I apply that function okay so how do I call this function now so there's a function called PyCon India which takes let's say which prints so this is Python 3 by the way so I'm gonna use print as a function PyCon is awesome and then this let's say another function called some other conference which is apparently not cool at all some other conference so I want to apply only a cool function print let me create a list of functions PyCon India is one function and some other so you remember functions are first-class objects right so they can be part of a list too so what I can do is for funcs in my funcs I can say what was my cool function applicator again cool funcs only can you please increase the font size cool I can is that okay oh so this is Python 3 thank God I remember that syntax error invalid syntax my funcs no it's on line 9 which says oh okay so yeah function object has no attribute starts with of course it doesn't it's a string right so I have to pass a string name to this so is cool func dot qual name so remember qual name which has the qualified name cool so PyCon is awesome can you see that yeah so PyCon is awesome the other one says no sorry dude so you you're not up you're not applying that function at all so this is a first step to metaprogram you can do such kind of things and we haven't even entered decorators and metaclasses so let's go to metaclasses no sorry let's go to decorators let's just take a wild code snippet right so we have def add sub mal and dev and let's take that to avoid typing I have already created no I haven't let me just take let me have a function let's say def add a comma b and which prints a plus b and let's say one other function called def sub a comma b which prints a minus b right so now you want to debug this right so this is a lot of things are happening here and it's so complicated and you've lost yourself in a lot of code you want to debug this you want to start debug this how do you debug you obviously write print I'm here yeah and you say print I'm now here things like that all sorts of things which we developers love to use to debug and of course when I call them you have so you get where things are going some reason it has decided not to print oh it has decided to print sure so what we do now is look at this code and gasped the horror of how horrible it is right so this is not the right way to do things and that's where decorators come in so what a decorator is is it's just a where's my presentation here's my presentation so a decorator is a function that wraps another function and returns a function so what this means is it takes a function as an argument it's a higher-order function and then it up it you can do things with that function and then you apply that function and then you return back the same function remember we wrote a higher-order function earlier so let's just do that with this now I'm going to create a new file called decor start by where I'm going to define the decorators so this is my higher or let's say debug which takes a function name so because it's a higher-order function it takes a function as argument and then I write a wrapper which takes arguments and then what I do is print hey I am here or in this sense I could do hey I am in let's say what's the function name yes call them remember call them it's a cool thing now I then apply that function pass that args and star star coax and return the wrapper so we have now defined a decorator so all this does is accept the function do something and then return the same function back applies that function and returns the same function back now instead of print here a and here a I am going to from because I'm gonna import what was my decorator debug debug and debug so this at is just a syntactical sugar for passing the function explicitly like func is equal to debug of add it's just syntactic sugar for this but let's not because it's so much cooler to write at debug we'll just write at debug so yes higher-order function again so debug is our function which is accepting add as a function now hey I'm an add as simple as that and if I do sub of two comma three here I'm in sub so you can just get the function name and start debugging like that so it's a powerful tool when to avoid code repetition and so there's a principle called don't repeat yourself and this is where we start in not repeating ourselves right what it comes next let me go back to my presentation cool so that's a decorator we've already seen a decorator it's a function which takes a function wrap cert and returns a function and that's how decorators are applied but the problem with decorators are let me remember our decorator name it was diva and we had there was our functions cool so when we want to it's getting cut right for some reason so for some reason when I do help of the whatever function we've written since we had added that at debug it's convoluting namespace and it's doing some kind of surgery within and then it's messing up all my metadata about that function so this is not cool add is not help on a function wrap in module because wrap star arcs or quarks or whatever add is a function which adds two numbers and this is not cool so what I have to do is there's a tool called fun tools import wraps which if I apply let me do that maintains the metadata so what I'm doing now is writing a decorator for a decorator cool so it's just I import wraps from fun tools and then add this wraps for the wrapper and now if I redo the cool it says help on function add in module main add a comma v and so on so we're good there sort of few nuances in python and we will definitely get over it this sort of exposes a powerful set of API's now for us to explore what if I want to do things like this now obviously developers use decorators to follow don't repeat yourself that's right at 12 HR SH we developers love to do things like this right so all sorts of crazy star star star star plus plus plus plus kind of debug messages and now how do we handle that right so what if we want to have print star star star star star here and print plus plus plus plus here and our decorator will not be able to do star star star star or plus plus plus plus based on our function name right so we have function dot call name here so we can do that what we can do is kind of write a decorator which accepts accept some kind of argument so we can do star star star star and plus plus plus plus here but we haven't implemented this yet and how we implement this is a little tricky with arguments so what we have to do is basically wrap the decorator itself accept the message and then wrap the decorator which actually wraps the function and oh looks like my last minute edit hadn't come here so I was if those of you who saw that inception slide in the last five minutes yeah I wanted to put that here doesn't make any sense now let me do that here this is a debug wrap which accepts that function and now this whole thing is encapsulated in debug which accepts a message now what I could do is of course I have to return the wrapper debug wrapper itself that should do and now I can just say hey I'm in whatever I'm just gonna format it and say message print message here that should probably work and let's see if it does yes so we have now almost a comprehensive framework to start building on your own debugging module with your own set of decorators with your own set of nuances and there are a lot of more possibilities here you can start adding more arguments you can start accepting special you can probably accept a function in your decorator as well and the whole universe of possibilities are involved for 20 how are we doing on time cool I think so let me not code this but it is possible to decorate a class as well so what if you had add sub mull and as a part of a class right so you had class foo which has add and sub and you don't want to decorate each and every method of that so you could just do at debug class let me explain what's happening here so at debug class is decorator and it accepts a class so remember decorators are functions which accept either functions which accept functions or other things so a class is a first-class object in Python as well we'll see that later class is an object to we'll see that later and so debug class is going to accept that class and what it's going to do is for key comma well in bars of class dot items where is an inbuilt Python method that's going to give you a class dictionary what a class dictionary is basically is a set of attributes that are part of the class so we'll see that later too so a class dictionary might have add and sub in it it's going to take all of those class dictionary the items of that dictionary and it's just going to say if that well is a callable that means if my ad is a callable or sub is a callable I'm going to do something there set at a set at a class key and debug of well so I hope you're getting where I'm going so debug is my original decorator what I had written earlier so debug of well is just going to add debug of ad and debug of sub so I'm just I'm just walking through all the methods in the class and prepending or wrapping the debug in each and every method of that class so this is a classic example of a class decorator but there is a problem with this no actually there is no problem with this it's quite powerful let's go to metaclasses so decorators were one part of probably a powerful metaprogramming API so we were able to intercept function calls do something with the before the function after the function and so on now we'll go to a step now we'll go a step further and try to modify a class itself the behavior of a class itself right so there comes metaclasses quick advertisement again that's my Twitter and GitHub handles so this is like those mobile apps where and after each level you have those advertisements coming in so yeah this is a Python class again so a classic Python class let's say class foo it has an init function it has to do something and so on and it's basically a set of entities right it has a class name the class name is foo it has a set of base classes here it's nothing and it has a set of methods so here it's deaf in it and deaf do something so a class is basically a composition of these three things it's a class name set of base classes and the set of methods or attributes for here we don't have any attributes let's just consider the methods now what we have so what we do in Python when defining a class is very simple right so we just write class foo deaf in a deaf do something so believe it or not this is syntactic sugar as well internally Python doesn't translate this directly to abstract syntax tree what it does is something like this it calls the type a type is the supreme metaclass that sits on top of the Python class ecosystem now let's see what that means and before we come into the slide let's do a little bit of live coding things right so we all know that by we all know about the Python type system right so it has an int and then we have a visible enough yes let's say 3.0 then type is float oh sorry wait I was coming to that so type of a is float but what is the type of float itself it's a class and the type of float is type the type of every class that you define is basically type let's say class foo oh I hate this Python thing Python class foo and let's just say pass and type of f is equal to foo type now type of f would be foo because we've just defined the class foo and similarly int is a class float is a class everything is a class but what is the type of foo itself it's of type type so every class in Python is an object of the class type so what this means is type is the meta class and all of these classes are basically the classes of that meta class so meta class is just the class of a class it's as simple as that right so what is happening here foo is equal to type of class name basis and class that's how you instantiate a class and that's how Python instantiates a class as well let's do something let's do a quick let's say our class has the class name foo again somehow I like the word foo and it has a set of base classes I don't want to inherit from anything and let me define a function in it which has can you see my code so I'm gonna do self and x or whatever and then say self dot x is equal to x and that's about it and let's say do something and I'm gonna say self comma no y and self return self dot x minus y something like that so these so but these are the prop these are the elements of my class basically we remember that we figured out that a class was basically a class name set of basis and then a set of functions now we can go ahead and create a class from these entities right so all I need is to create a class dict oops which has a function which takes my init right so init is the function and it is assigned to in it and do something is my function name so do something is my function name and it is assigned to do something and there you go that's my classic now we have the class name we have the basis and then we have the classic so now let's create a class foo from type which is a class name basis class dict and we have created foo oh wait it accepts some argument cool now type of f it may not fool so this is a class and we've created a class with how python does it internally so for 30 15 moments let's do some so we've looked at what type is and now essentially if you are not if you're defining a class like this essentially you are by default the meta classes type itself so you can see that type of foo will be type now it is possible to modify the meta class and define your own meta class you can create your own type in python and this is extremely powerful and let's look at a quick set of examples let's do a quick exercise let's do some diabolical stuff one such hmm a very interesting talk on meta programming thank you whoever tweeted that okay let me just open a new console let's say Python let's create a okay let's let's create a class Python right so which has let's say the set of talks that were initially proposed and let's say selected you get selected from the set of talks so for some reason let's say the python people decide to choose exactly a third of the talks that were approved random logic I hope that's that wasn't how it happened but anyway so this is my class python right and I want to do something with this and I want to make sure that yeah so let's create a meta class of type conference I mean obviously by con will be a conference and there are a few hooks in python right magic methods we saw underscore underscore name like that whenever a class is instantiated in python we have the new thing is called so whenever a class is instantiated internally python calls a new method of that class or of that meta class every type has basically a new method so what we so new basically accepts the class name then it accepts the class also known it accepts the class itself class name and the set of basis and the class dick how do I know this I looked up the documentation let's just see what happens when this thing is going on let's just print the class dick right and the way you do this and the way you do this in python 3 is just say meta class is equal to conference now my the meta class of Python won't be type but it will be of the type conference on instantiating type or a none type object is not callable where is this happening oh correct you're right brilliant still says none type is not trace back file line p equal to python then none type not callable let's see if my debugging skills are good oh okay so this I think should be of I don't know if this will work no it's not working brilliant that's right so what I have to do is who said that brilliant thank you so what I have to do is no super dot new class class name basis right so remember that every class is an object now when I'm doing of the type type I have to return the class object itself right so Python is an object and here it's not callable so that means I hadn't the type or the conference meta class had not returned anything so it wasn't a callable yet now oh it's still not callable no I got to return it no no no I have to return it yes cool so when I return that class object what it does is it shows me what I'm doing is just printing the classic whenever instance of PyCon is being created so this is the class dict it says qual name is PyCon there's a init function and then there's a module main and then there's a selected thing so I could do crazy things here right and let me take cool and uncool conference things again so in when the beginning of hold on in this meta class itself what I could do is take the class name cool and check if class name dot thanks so yeah do stuff like this and I'm I'm intercepting the class name here and doing stuff on the class name itself and this is extremely powerful again so there are a lot of hooks that are available def let's say call so call is when your object is instantiated new is when the class is defined and call you call basically gets called when you do P equal to PyCon of 10 or whatever now a call accepts the class and the arguments now you do know that the meta class conference if I define for if I had some other class called class some other con or whatever you could have told on yeah or not cool and you could have not created the class itself at all you could have thrown some error over there so you can actually inhibit people from writing classes of the type meta class whose name is not cool a lot of things are possible now something that's even more cool is the call function which is extremely powerful right okay so let me just see what's happening there print class print args and print quarks whatever okay so it says do you see that it says 10 comma and empty dict so that's the args and quarks that are basically being called when I do P Python 10 so when I call 10 so my args are 10 and when I do 100 my args are 100 and so on so let me intercept this so a very important concept of metaprogramming is interception so you some someone defines a class and someone's running it and in runtime you intercept stuff and you modify the behavior of the code itself and what I could do here is check if args of 0 cool conferences okay if args of 0 is let's say less than 10 I'd say print ouch too less participation oh okay so it was 100 right now if I say 5 says ouch too less participation 10 minutes remaining cool yeah so it's quite powerful and you can do all sorts of things like this you can just you know create your own class yuck to few people and let's say it's exception pass and you could just do so it says yuck too few people yuck too few people exception and so on so you could do a lot of powerful things now one final thing that I'd like to do here is oh ideally to the whole thing is it it's back thank God for control C and control okay no I'll just show you the already created ipython notebooks so I have only 10 minutes remaining so I could take some questions so by the way all this code is on my GitHub and in the form of ipython notebooks so you can create singleton classes singleton is a gang design pattern you could just say if you are it is a meta class of singleton a every instantiation would be the same and right so this is something extremely powerful so if you've used rails it has something called active records and arbitrary so even if the class does not have any property you'd still basically be able to return something for that so get at a rest that exactly so when you do class name dot property let's say dv dot whatever dv dot some random property which does not exist clearly here but what will happen is dv's meta class query maker will be intercepted and query maker has a get at a hook that is going to capture this and then it'll it's it's going to just basically return lambda it's going to return a function what I'm doing here is return a function that basically writes that SQL query so what I could the idea behind this was let's say dv dot column name of 10 or whatever how do I execute this shift into yeah so just this would give me select column name from database limit 10 so things like that right so I could have any arbitrary column here dv dot whatever you wish so it would select select whatever you wish from db limit 10 so now we are actually going towards building some kind of domain specific language so just dv dot whatever of whatever and you can create powerful construct so it's returning a function here returning that lambda which is returning a SQL statement so you basically writing code that is writing code for us yes that's a bit hold on let me get back to my presentation function of 11 yeah and the final frontier would obviously be abstract syntax trees and I I thought I wouldn't have time to cover that and I don't and Python has every language has an internal representation of the code itself code is translated into abstract syntax trees while execution and the abstract syntax tree is then converted to your low-level language and in Python it's so there are two kinds of languages right there are homo-iconic languages whose abstract syntax trees or the code itself is basically written in the same language as the higher level language so in Lisp for example the abstract syntax tree itself is in Lisp so that's something cool that you have to explore and a lot of powerful things are possible with that yeah thank you it's about having fun really that's about it thanks okay anyone having any question you can raise your hand so that I can get a mic for you in a class you gave the argument for a meta class so is there a way like we can make it default like in a module all the classes will use the same meta class yes absolutely so a meta class is inheritable that means that if you have class a which has meta class is equal to some type you have class b which inherits from a classy which in its from b it propagates thank you where we can use meta class like what is the need to use have you used Django yeah so you have something called models in yeah it's completely meta classes you could look up the implementation so a lot of frameworks uses meta class to make things simpler lesser lines of code okay thanks for functions and the meta classes for classes you could write decorators for classes to okay we did that quick example where we went through all the methods of a class so decorators and meta classes are different ways of achieving probably the same thing capturing the hook and intercepting code so you intercept at runtime or compile time you intercept code and change behavior that's the intention pardon me Jenna I just need to make a small announcement and continue with your question session after that so we are having the community discussion that was going to happen in the open space at 430 is happening in odd e3 at 445 so those are few who were interested please come to odd e3 thanks a lot continue with your country thanks one more please sure like this meta classes do they reduce the complexity like you are reducing lines of codes so but we are still writing inside a class meta class so are they reducing the efforts or cost if if you are comfortable enough with any concept I think it would be a powerful weapon meta classes definitely a powerful weapon but if you're not comfortable with it you're I mean usually meta classes are like the far end of Python and nobody talk about them in general so if you look at any language meta programming is somewhat similar they are a set of principles and they are a set of concepts and Python exposes some behavior by the next was some API's like meta classes for instance for instance if you take ECMAScript 6 it exposes something called proxies and if you take Lisp you can do a whole lot of macros within it so it's a set of principles again thank you so in what cases will you use the call instead of an in it considering that both of them will be called when you initialize it in it is when you initialize call is when you so in it happens after call that's the only difference yeah in it happens after the call you define some self dot x equal to x or whatever call is when the class object is being called is being instantiated that transition within a function there are many conditions like many calculation how can we debug or using so with I understand so you want to modify the function code itself within the function so you have a set of statements within the function and you want within a state within a function there are loads of condition and calculation I want to debug within what condition it will sure so two answers to that one is don't write functions that have many conditions write small functions that's a good practice second is if you really want to go inside the function and do stuff then you have to use abstract syntax trees abstract syntax trees expose you to the code itself so you can extract the code and modify each line how much of these is specific to python 3 quite a lot so for example the sin the syntax meta class is equal to within the class which we did was python 3 so in python 2 you have inside the class you write underscore underscore meta class underscore underscore is equal to something and usually you're better off using python 3 today and there are a lot of underscore underscore get at her underscore underscore all those new hooks are python specific three specific some some code is definitely portable so you can there are a lot of APIs in python 2 as well look up the documentation I guess you have hooks so yeah actually yeah yeah that's right thanks hello so one question here was what are the difference between meta classes and descriptors descriptors okay well I won't answer that because I'm really not sure yeah I just want to quickly understand the meta class you defined you forgot to do the new thing right so what was the need for that I mean it's not clear to me if you don't the super class should or anyways be automatically called by your interpreter when you that's that's a base class so now we're defining our own type okay so that won't have the object as the root is it no it'll have type as the root not object as the root right so the type super should have been called automatically right so I did you why what was the need for doing a class object equal to something dot new and then return the class object is that how python automatically does as well when you yes it's fine your normal and we modified the code there so we did something else one thing when we were doing writing class decorators was we did debug of class method and the second when we were doing meta class example what we did was we did print something and then we returned thing usually by default so it'll just create the class and give you we intercepted that we did some other operations there and then we instantiated then we returned it okay and meta class is more close closer to like the introspection kind of thing one time you're basically figuring out the exact thing java also had that introspection long long back so I think using this abstract syntax case I'm just trying to understand like so I can actually compare if my earlier version of my code and the current version logically if they're doing the same thing or not is it possible using the diff of the abstract yes yes yes so abstract syntax please are trees basically and then if you have two trees you can compare which nodes are different so it's just three operations data structures okay thank you thank you sure thanks