 That's actually for that's actually for Chris. Okay, so let's get started. We're starting a little bit late This is Python gone bananas monkey patching in Python We are at pi Gotham 2016. It's Sunday, July 17 2016. Hi everyone. I'm James Well, that was really good that was a really good response before I get started with this talk I wanted to give you a disclaimer. I won't be upset if you all walk out This is not one of the usual types of talks that I give usually I give talks about don't use this code You can see that's what I am on Twitter, and that's my email address I tell you about really weird wild things you can do in Python This talk is something different I thought I'd go back to my roots and I'd give you a talk about something genuinely useful Maybe educational something practical something that many of you can employ in your actual work This is a novice to novice intermediate talk It's very similar to one of the very first talks ever gave in fact My very first talk conference was in 2012 at pi Gotham the conference organizer that year Gloria Had met up with me after NYC Python event and we were at Burger King and she said, you know Pi Gotham's in a week and a half you want to go give a talk and I said sure and I spent the next week and a Half furiously preparing as much material as I could I Took that talk and I presented at pi data that same year and I went to pi, Arkansas I went to pi, Texas Since then I've spoken at a number of other events and you can see I think I'm on my 40 This is my 40th conference that I presented at I think it's the 50th or 60th talk I've ever given for many of in the audience this conference you have ever attended and You've seen the kind of energy and the kind of life that you can you see what can be done when you present your materials in this forum To this kind of audience you see how powerful that is many of you are Nervous you think I could never give a talk that could never be me on stage But I want to tell you it just starts with one powerful things about pi Gotham We love to give you opportunities to speak and we really love to see up-and-coming new speakers because who knows Any one of you might come back next year give your very first conference talk and end up like a conference junkie like me Four years later given given like 70 talks now. It's okay to be nervous I'm nervous before every single one of my talks. I can't even I I tried to put together the list of all the comments I've spoken at I can't literally not remember all the talks I've given and yet. I'm still nervous every single time It's just the way it is and it's okay so This talk is about monkey patching is kind of softer term for what other people might call gorilla patching Gorilla patching and monkey patching are essentially changing the runtime state of a program Using some mechanisms now this talk may be a little bit different than some of the other talks You've seen about core python materials this talk is not a why but the how I Won't tell you when you want a monkey patch or why you want a monkey patch I would think that when you have a need to do this This mechanism will probably be the most obvious way to solve your problem There are probably a lot of other talks out there that can explain better than I can some of the when you want to use This mechanism instead. I'm going to focus on the how just the nuts and the bolts It's very similar to the very first talk I ever gave which was the nuts and bolts of generators and context managers and decorators One thing I want to do in this talk is try to put forward a couple of mental models Some simple ways to think about how the Python language work that will help you in your everyday Python programming and will make What appears to be a very complicated very dangerous thing to do just seem obvious I? Told one of my colleagues about this talk and He said oh you're just going to talk about the mock library and it is true The monkey patching is very widely used when people are doing testing you could think that in a production system You have code that does actual things and you might want to mock that code You might want to replace code that does actual things like save to a database or load from a database What simulated code that just checks to see if that behavior behaves correctly and I'm not going to talk about mock I'm not going to talk about testing. There are a lot of good talks out there on that material I'll tell you that when I Started to write this talk at around and see I got home last night at 11 o'clock and I think there was a new episodes of suits So I had to watch that and then it was like 1215 and I was like oh my goodness I have to talk tomorrow so about 12 about 130 last night because I Did all my laundry I clean all my house Procrastination is really good for getting things done that aren't the thing you wanted to get done But everything else in your life. I thought okay after write my talk I started was what is a monkey patch? I wanted to include this link for you because this is a stack overflow link that tells you what is a monkey patch and one Of our organizer is actually a very well known person on stack overflow And he has a very nice answer here just explaining a little bit of the usage of munching packing So if you see Aaron Hall, he's one of our organizers for the meetup group You can ask him and he'll point you to his stack overflow answer One note for all that I'm going to show you is I'm only going to talk about Python 3 I could talk about Python 2 and Python 3 and there's a political choice there and whether I present this information in 2 or 3 What I'll tell you the city's sake. I'll show you all of this specific to Python 3 Some of the things that I'm going to show you have actually changed quite dramatically between Python 2 and 3 They've been made easier in Python 3 and I want to show you that because hopefully most of you are using that It makes my job easier and to show you both the Python 2 case and the Python 3 case Well my time allowed So what I want to start with is a couple of puzzles and for each of these puzzles Unfortunately, we don't have enough time to get feedback from all of you So I'll just give you a couple of seconds to look at the code and to think what the answer is for each of these Puzzles, I want you to guess what the output or the error is I will tell you there are no tricks So you defining a class? I'm not going to do some weird trick where that class has a bad name or I've The semicolon the colon is actually semicolon each of these are actual problems that you might run into and I'll start very simply Here I have a class. It's called my class. I do a single assignment in that class I say ID equate an instance of that class called object. I Do my class ID and as many of you know the ID function in Python just gives you the ID of some object So it takes one argument and it gives you the ID, which is just a number I do my class ID and object ID First line this matches what you think will happen It worked and it worked now in this case the exact output you might not be able to guess But that's not important if you guess that both of these worked then that's good Let's take another example. I wrote a custom function called my ID. All it does is it calls the ID function I have a class calls. I assign my ID equals to my ID. I create an instance of that class Does my class dot my ID work? No, it doesn't oh we have to evaluate these Yes, it does does object up my ID work. No, it doesn't so immediately we see something There's a puzzle here. What's going on? Why is there a difference here in the next two examples? I'll choose two mathematical functions sine and cosine and so I'll just show you just just to be absolutely gratuitous Here's my little notebook just to remind you of what the sine and the cosine function look like the sine function is in red It starts at zero it goes to one at pi over two it goes down to zero at pi Here I import pi from the math module and I create a new class and I say from math import sine Is that legal can I do that in a class body does that work? I create an instance of this class. I call the sine method on my class with pi over two Well sine of pi over two should be one on the instance Does this work? Well, it seems to does this line work? Yes, does this line work? Yes, it does Let's do something else This is in jupiter notebook. So some of the syntax at the top of the cells May not be familiar to all of you file that file is called mod dot py. It's just a python module in that mod dot Py python module. I'll import cosine from the math module. I'll write a function called my cosine, and I'll just call cosine I'll create my own class, and I'll do the import. Well, we saw that work before so it should work here I'll call my ass. Does that work? Yes, it gives me the answer. I expect does it work on the instance? No, why not? Here's another puzzle. I have a class and I assign some variable to a lambda x and I return x times two Well, this works. There's nothing strange about it. I call this method on the class. Yes, it works Can I call it on the instance? No, it fails and one thing that you might immediately see is well, isn't there a self parameter somewhere in here? That's a good hint We'll try an even simpler example here. I have a class and it has a function in it It takes the first parameter self. We call this implicit self and it takes another parameter x and it returns x squared This is very standard code. That should definitely work. Does my class dot square work? No, it doesn't it's missing a parameter This object dot square work. Yes, it does Here's another puzzle My class it doesn't have anything in it. I have a function called message. All it does is it prints out pygotham 2016 I assign message to my class dot message. I create an instance of this class Does my class dot message work? No, it does object dot message work Yes, it does Let's try another example instead of assigning message After or before I create the instance. What if I sign it after I create the instance? I create an instance of this class then I assign message to the class itself after I've created it This still works, but does this work? No, it doesn't It's missing a method or it's missing an argument. There's a second one work. Yes, it does Lastly, I assign the message function onto the instance Does this work? No, it doesn't here's one Two more two more examples, and then we'll talk about why some of these work and why some of these don't work. I create a folder Called folder in that folder. I create a module mod a member I have two modules called mod dot py one in the folder from which this notebook is running and another one in this folder called folder Maybe I should have called this my folder, but I didn't I'll create this and it just writes a file Here in a separate Python interpreter, so I don't screw around with anything else to look I Append this folder path to my sys.path, and then I try and import message from my module Remember only the module that I just wrote has that message variable in it the one that I wrote a long time ago Didn't have that at all. It only had my my cosine function Does this work because? Python is going to look for that module when I try and import it Where is it? It's gonna look it's gonna look at sys.path the first part of sys.path is the current directory It's gonna go to the current directory. It's gonna find the module that I created in my previous example It's gonna say there's no message variable in that module, and it's gonna blow up on me So obviously cannot import name message if I and To show you why that is if I were to Import that module and look at where that module is located relative to my current directory It's in my current directory when I do that import It finds the mod.py in my current directory not that's the first place it's going to look So you could think if I do insert and I put that folder path at the very beginning of the path That'll be the first place. It'll look It'll find that module in the folder directory and then this should look So here I say the first place you should look is my folder That's probably a little bit more confusing than I thought it would be at 2 30 a.m. Last night because I didn't say I started working at 1 30, but I didn't say that I actually got that far dragging my feet This should work and it works as we expect the question is Does this work? I say path where brackets folder plus path instead of using the dot insert method I do this does this work? No, it doesn't. Oh, there's a type. It still doesn't work. Don't worry Here's another puzzle. I have a for loop in a class body. Can I do that? Is that okay? Is that moral? Is that in good taste? I'd say it's probably not in good taste the morality of it. Some people say code has no morality to it I don't believe that but I wouldn't say that this is a particularly odious piece of code But the question is does it work? Do I can have a for loop in a class body? Let's see what this for loop does well It iterates from the numbers from zero up to nine and it Amends this local variable it creates a local variable x underscore that number and sets it equal to that number times ten So if I say object dot x zero x one and x that give me the number zero ten and twenty. Does that actually work like that? Yes, it does. That's kind of weird Here's another question. Can I do it if else in a class body? Does that work? Here I say x equals ten if ten is greater than five Define message is this otherwise define message is this because there's that work Probably not something you want to do, but let's see Well ten is greater than five. So it prints out. I got them as awesome If I change this to one and reevaluate these two it prints out the other one. So that seems to work Why does that work? Lastly the last puzzle I want is tell me the difference between these two And anyone tell me the difference other than what they look like the difference between these two Structures from x import y printing y or import x and printing x dot y That's the last puzzle that we will answer in this session So the answer or the all of these and the key to thinking about how and why monkey patching works the way it does in Python is to remember that the Python interpreter is a very simple imperiper It's very straightforward if you look at a Python file and you execute in your mind the lines of code from the top to the bottom of The file you're doing almost exactly what the Python interpreter does Many variants with other languages like C or C++ or Java languages have a complex compiler step And first the compiler does some magic It produces some binary and in that binary you may not always be able to predict line by line What code is being executed? Python does not work in that fashion at least see Python does not It's some code and you can execute it in your mind line by line from top to bottom And that's very close to what the Python interpreter internally is doing. I Don't think we had a Introductory bytecode talk, but I've given introductory bytecode talks before but you can also you can often see the close Correspondence to actual lines of Python is to a very close Correspondence and that's done in order to keep the language simple in order to keep the language Flexible and in order to make some of the things that I'm going to show you here very easy to do where they're otherwise actually fairly Difficult to do in languages like C++ and Java So we know that the Python interpreter operates linearly from top to bottom It doesn't have a separate definition or declare step as you have in C C++ or Java a deaf statement of Python just says Figure out what codes under this create a code object Create a function object call it this name. What happens in a class body? It sees a class body It sees some code under the code under that class body looks at all the local variables that were created says these local variables are Attributes on that class and that's it. It's all that happens One question that is relevant and you should ask yourself is Does Python have a step that we could call compilation? Who thinks Python a couple people who thinks Python doesn't have a compilation step? Who's ready to go home right about now? Nobody that's good. So let's let's think about what that means Oftentimes when people talk about Interpreted versus compiled language Singuishing between is the way in which we use that language in a language like C or C++ or Java The way we use and develop in that language is different than in Python We often write some code. We subject it to Verification by a compiler we go for a coffee break we come back and then we run some tests and we see if that code works We're used to a very interactive method of inter of working with our code We write one line of code and we immediately interact with that. There's not that strict Difference that's not that strong line that stark line between writing the code and trying out the code You work in a more iterative Way you write a little bit here and you test it out You write a little bit more and you interact with your code live use that to find some bugs Incorporate that back into your script and then start the cycle over Python however does have a compilation step If we were to say Python had a compilation step We would say that it's the step which takes all of the source text that you write and Generates bytecode from it back almost nothing interesting or useful it catches almost no errors at the most It could catch some name errors, but it catches almost no errors unlike in a language like C or C++ or Java Where the compilation step can catch a certain class of errors? That compilation step does one thing which is it how to generate bytecode from source text and one interesting thing That it does in that process is it figures out where variables came from it sees the name of a variable that you're trying to Do something with an arithmetic expression or printing it out to the screen and it says was this a local variable? Was this a global variable did this variable come from a closure this variable came from and it emits special bytecodes in those cases We'll see this become relevant in just a few minutes The other thing I will tell you is that the Python come because even though Python doesn't have a compilation step one thing that we can think of and how we interact with our code is that time is Equivalent in many cases to a compilation step that is to say people who write code in Java or C++ are used to Running their code by a compiler Having the compiler tell them some things about whether the code is correct or not and using that to determine the production worthiness of their code in Python if you can add certain checks or certain constraints to the top level of a module when a module is imported the code is run from top to bottom linearly Well module import is actually something that you could do ahead of time You could take a code base and import all your modules see if they worked and any checks that you included at that top level that module Could be used and so module import time in many ways is very similar to compile time or a little bit closer to compile time For those of us who are used to that So let's take a look at a module. This is a little Python module and it has a variable called x it sets x equal to 10 If x is greater than 5 forms this class definition Otherwise it performs this class definition Most of us who are looking at this code say this makes pretty much this makes sense If since x is 10 we'll go down this branch and my class will be defined to this thing If x were less than 5 we define this and we look at this we'd say that just works That should work fact it works and does exactly what we expect if I say x equals 1 then the branch We enter as this branch here my class is defined to This version which has a different message function pi got them as fun So if I re-evaluate this cell and reveal re-evaluate this cell the behavior we see is exactly what we'd expect Take that exact same behavior and put it inside the class body Nothing changes just as that cell which is equivalent to a Python module in the Jupiter world Runs all the code from top to bottom and performs all of those definitions live the exact same thing happens inside the class body We run all the code from top to bottom and we perform the depth so the definition of this message function It's performed live based on the state of this x variable When this code runs, so when I run this It does exactly what you think and if I do a object message here And if I change what the x variable is it changes exactly in the fashion you think the if else From outside the class the inside the class But the way that we can think about how the interpreter moves through that code Is actually identity Before I talk about the next puzzle I want to discuss briefly what this locals function is Locals is a function that tells you all of your local variables all the variables that you have here Here's a simple python script that creates three variables x y and z I print out my locals and in my locals I should see x is here z is here and what and y is here And you can see that this locals function returns a dictionary mapping the names Of my variables values are Here's one quick question. Does this work if I amend this locals dictionary set x equal to 20 Does the value of x change when I print it out? Is that value reflected on this line? In fact, it is when python sees this code it sees all of its scope It has to answer in that compilation step. Where did x come from? What does it say? Well x came from my global scope Or it came from the variables that I have It came from my global scope and it emits a bytecode called probably low global I'd have to check the bytecode to tell you for certain, but it probably emits low global What it does that it'll actually reflect all the differences in this that you make to this locals dictionary Here if I were to just create this variable The python interpreter when trying to Produce bytecode from this code says I don't know where the x variable came from It's probably some local variable. I was introduced somewhere and it gets picked up from this Amendment from this mutation that I did to this local state. So both of these work exactly as you might expect However, if I take exactly the same code and I put it inside a function body It no longer works and I want to tell you why that happens. So here I set x equal to 10 I try to change and then I print out x x is still set to 10 Why did that work that way? Well, what happens is inside that function body python said where did this x variable came from? It's probably a local variable because I see it defined here It emits a special bytecode instruction called load fast This is done for performance reasons because these very rare to do in python Most code doesn't need to do this. There are other mechanisms to use. So they said it breaks that use case But that's not a very good use case When you load fast you look up that variable from a different place Then where you're amending it by changing this local dictionary The greatest step that I can go into with the time that I have allotted But if we have time for questions, I'd love to show you a little bit more about what the bytecodes are This is relevant However in our next example Here in this next example The python interpreter sees a variable called x. It has no idea where it seems. It's a global variable Here I try to create x as a local variable I get a name error. I can't find x even though this line said, oh, no added in there as a local variable Nope, the python interpreter cannot find an x variable and at the compilation step at the parsing step It gives me a name error We can see the behavior that occurs inside a class body is somewhat different The behavior inside a class body is very similar to what happens inside a module So we'd expect mutations to the locals variable to be just like they were Inside a module so we'd expect this to work if x is equal to 10 and then I amend it I mutated local dictionary. I should see that When I look at the x attribute on my class Similarly, if I were to create a brand new x in this scope, I should see it reflected here Let's go back to our example in this example I just create a bunch of variables called x underscore something and that's why this works Just as you might naively expect for some of these puzzles I feel like there's a Little valley between if you don't know anything you might look at this and say, of course it works And then as you learn a little bit more python, you say, well, I'm not sure And then when you learn a little bit more python, you say, of course it works A little bit nudged in that direction And that's the example to our or that's the answer for why this works Before I talk about the next puzzle, I want to briefly discuss with you Ids and objects and mutability and immutability Here I have a variable called x I used this id function earlier What is the id? It gives me a unique number for every object In c python, if I look at hex of id, it gives me the hexadecimal representation of that number This hexadecimal representation should trigger some memories for you Especially those of you who are c or c++ programmers This looks like a memory address In fact, the id of an object Standard doesn't guarantee this, but the id of an object is basically the memory address of that object So x is a variable that lives at this memory address In my running program Here I have x and I add one to x Notice change the one from c to e Why is that? Numbers like strings and tuples Are immutable in python. You can't change them in place Every time you perform an operation that changes a number by doubling it or adding one to it You actually allocate a brand new number And so the x variable here referring to 10 The x variable here just had its name reassigned But that 10 that little piece of memory there that stored the number 10 Is still sitting around the memory somewhere. Just nobody's referring to it. There's no variable that refers to it There's no name that refers to it That's why this id changes If I have a list however imputable we can change them in place You can see I create a list with three values. It has some id I append to that list. I add an element to the end of that list You can see The id does not change at all. I've mutated that list. I've changed it in place And if I look at that list and say it's the number four in inside Of course it is. I append it in the last line Let me try something a little bit funnier x and y are both names that are referred to the same list in memory One two three They have the exact same memory address because they both refer to the exact same list If I appended four to x you'd see four and y The numbers to x you'd see those in y as well because they're both referring to the exact same object If I do this line, I say x equals x plus square brackets four. What happens? In many cases in the python syntax Instead of changing something in place, especially with this arithmetic syntax this augmented assignments Instead of changing x in place what this says is create a brand new list by taking the elements of x putting four at the end and call that x y still refers to my original list x now refers to a brand new list that happens to be all the original context copied Plus a four at the end. But the idea of x is totally different than before What we'll see is if we add five to the end of x That five will not be inside of y because Y is the original list with the original id So look at If we take a look at our original question path dot insert What is this doing? The sys module has a variable called path The path variable is a list of strings those strings represents paths directories on your system Where you'll look for python modules and packages to import dot insert It takes the actual path variable the actual list that's used inside the interpreter That's used inside the import library And it'll add an element to the beginning but it'll mutate it in place So this should work If I say path equal square brackets folder plus path This will say in my module. I have a new variable called path That's equal to whatever was in sys dot path was something appended to it But the actual underlying sys dot path was never changed. So when I run this And I'll remove this code and I'll show you From sys import path And I'll show you from sys import path You can see there's no folder at the end What I might want to do and what you're encouraged to do is write this code like this sys dot path equals folder dot sys dot path here I'm actually changing The path list in the sys module it added folder at the beginning That's why that example didn't work And this is something that you'll see in a couple of different circumstances Here I have three functions that ostensibly do the same thing In the first one, I take some list called x and I call dot clear on it In the second one, I use a del operator and the third one I assign it to him if I run this code And I try with clear will it change the x variable that's outside of this function? Yes, x dot clear closed a clear method on the original object What about del del deletes all the contents of the original object? What about the assignment? Note to a variable called x in the local scope of this function and empty list So this will not reflect the change that I tried to perform inside my function The last set of puzzles can be answered by thinking about python in a in terms of a set of ad hoc protocols One thing that you're deeper into python and you look at the data model methods in python All those methods that have two underscores before and after and seem to conveniently Correspond to top level methods for len. There's underscore underscore len. I hate dunder. I don't like that word For repper you have repper what 200 for for the addition operator you have Addition of what 200 scores before and after I add and so forth and so on these protocols The two protocols that we'll talk about in the time that we have remaining are the get attribute protocol and the descriptor protocol Because they can answer a lot of the how for why monkey patching works the way Here I have two classes and an instance one class is called a the next class is called b and b derives from a a is the base class for b b is a derived class for a I have an object that is an instance of the class b I set an attribute on the instance itself. I say object dot message equals python Pygothem is awesome If I try and print out object dot message does anything get printed out Of course it prints out pygothem is awesome. I just added that there I added that information to the instance variable if I want to mention that's been added to an instance of a class I can look at the underscore dict item of the class And you can see this is all the information I've added just to the instance I could do the same thing With the class itself Oops And you can see I didn't really add anything interesting to this class Let's take a look at this example again here. I assigned message equals pygothem is fun to my base class Let me run that code and when I look at object dot message what happens Behind the scenes I invoke the get the get attribute protocol is actually fairly simple Now there's a little bit of complication when you look at underscore underscore get adder and underscore underscore get attribute We'll put that aside for the purposes of this talk What happens when I say object dot message? I first go to that I this line this piece of syntax is saying python And I want it on the object use the get attribute protocol to figure out how I get that variable out What does that protocol do it goes to the instance? It says instance Do you have a variable called message defined in your dict? If the instance does it'll give you that if the instance doesn't it'll say no, I don't check my class If the class has to turn that if the class doesn't have one It'll say check my base class and we'll keep checking until we hit the end and I'll show you where the end is So here that works fine. I pulled a message variable off of the class called a If I said b dot message equals pygothem has cool talks What'll happen when I say object dot instance? Do you have a variable called message? Nope, I still don't it'll ask its type b. Do you have a variable called instance? Yes, I do It was added on the line right above And so this will give you pygothem has cool talks including this one You might wonder what's the order in which the python interpreter looks for these things There was a talk given I think right before this session by one of the people sitting in this room amit on MRO the method resolution order I encourage you all if you're interested in this to look at his talk when it's posted to youtube The MRO tells you the order in which we look for these attributes So here if I'm looking for Something called msg on some instance I'll first look in the instance then I'll look at b then I'll look at a then I'll look at the end the top level object If top level object doesn't have it it'll emit a warning saying attribute error. No such attributes However, there is one complication. There is one additional protocol that intersects with interacts with this protocol It's called the descriptor protocol When the get attribute protocol? Moves past the instance and starts asking the classes up the chain. Do you have this attribute? So most I do And then I'll ask it one more question That second question will ask is does that thing you're about to give me back? Have a have a method on it with underscores before and after it called get or set or delete In this case, we'll only look at get And if that thing does have a get method it'll say don't give me the actual thing itself Call that get method with a couple of special parameters and give me what that returns I'll show you an example of that in practice Here I have a class it has two things inside of it a variable called message and a variable called descriptor Because this is the descriptor when I say object dot message. What happens? I ask the instance. Do you have this variable? The instance says no, I don't ask my class. I ask the class the class says I think I do it looks at what it has. It's just a string. It returns a string outright When I ask it for this descriptor attribute, what does it do? It says So I'll ask my class the class says do I have it? Yes, I do It'll pull that thing out and it would return this thing here Except that one extra step it takes is it says the thing I'm about to return Does that have a get method on it? Oh, it does So instead of returning that thing return what happens when you call that? And so what you'll see on this step is we'll we'll perform this print and we'll return Pygothon 2016 is awesome You may wonder how the Property decorator works in python. This is how the property decorator works in python When you ask for something that looks like an attribute instead behind the scenes you call some code This is done using implementation of the descriptor protocol Let's take a look at the id function. We looked at the very beginning of this talk The id function is written in c. It's part of the built-ins module. It is not a descriptor. It does not have this get method on it Let's look at the my id function that I wrote myself It's written right here. Does it have the get actually doesn't have a get method? Yes, it does In python 3 all functions are descriptors all functions have a get method on it What that means is if you put a function into a class And you look it up through the instance Instead of right you'll return something slightly different. You return whatever happens When whatever whatever's going to happen whatever is defined by this get Method and i'll tell you what that get method does Here's a simple class. It has one method in it called message If I do object dot message oops I get this I get what's called a bound method one of the nice things that python 3 did is it removed what was often of Very confusing and hard to describe Situation in python 2 with bounded on bound methods. There are no unbound methods in python 3. There's only unbound methods What's a bound method a bound method is just a very simple wrapper over a function And all it does is it says when you call me I'm going to call the function that i'm wrapping i'm going to pass in a self as that first parameter And that's all i'm going to do. That's how the self gets implicitly added whenever you do a function This is a bound method. It's bound to this object So this self that will be put in is this instance. It's is this instant called obj And that'll be passed in Let's see that in use but before we see that I want to show you something kind of weird Whenever you do a Attribute look and that attribute is actually a function and that function invokes its get its get descriptor protocol component It'll actually return a brand new bound method So if I say x equals object dot message and y equals object dot message They'll both be bound methods and they'll actually be different bound methods. There's I'm not sure why this is And there's there's some there's almost certainly a good reason behind it But this actually has very real implications if any of you are working with especially gui tool kits And you're trying to set up callbacks And those callbacks are trying to use instance at times when you set up callbacks They're added to what's called a weak dictionary a dictionary that does not make sure that the object stays Stays around as long as the callback stays around Otherwise nothing would ever be garbage collected Well, those instance methods have nothing referring to them and they're created fresh every time you do a data lookup Every time you do an attribute lookup So what you'll have or try and create callbacks on instance methods And they'll disappear the moment you add them in there. This is a real implication And there's some weird work around you have to do in order to make this work But let's get back to our puzzles. This was the very first puzzle I showed you My class id equals id and my id equals my id What did I say at the beginning when you have a class body you just you evaluate the lines line by line You see what was in a local what was in a local scope? What are all the variables are and you package those up so saying id equals id just says Here's a local variable id and it's assigned to the id function that we're all familiar with Here's a local variable called my id and it's assigned to this my id that I defined a couple lines above I created instance of My class id I look up the id attribute through the class the class says I have it immediately It spits it out What it spits out is the id function on molested and that id function only takes one argument obj So this works If I say my class dot what does it do it says I have it myself Spits it out i'm molested and this works If I say object that id what happens object the instance does not have An id attribute so it asks its class and that's remember when you start asking the class is when you start It asks the class. Do you have an id? Attribute the class says yes, I do The class says but before I give it to you. Let me just check. Does this id attribute have A descriptor is it a descriptor that have a get method on it? No, it doesn't because id is a c function and none of the c functions have descriptors have the descriptor so This just works this is just The id function that you know and love Return to you on molested However in this final case when we say object dot my id We ask object. Do you have a my id attribute? No, I don't let me ask my I'd ask the class the class is Yes, I do but before I give it to you. Let me check if it's a descriptor Well, it's a pure python function. So of course it's descriptor So I'm not going to return to you on molested. In fact, I'm going to wrap it as a bound method I get a bound method back I try and call that bound method with one argument bound method automatically inserts an extra argument for me self What happens here? I got two arguments when I expected only one and that's why this puzzle works the way it does Let's take a look at the next puzzle with the math module Well the same in the same case the math module is written in c the signed in c So exactly the same behavior occurs When you ask the class for the sign method you get it out on molested When you ask the class for the my cosine method you get it out on molested But when you ask the instance for the sign method You first check does this have the descriptor protocol? Yes or no No, what does it? So you get it out on molested when you ask for the my cosine method The class says hold on a second. This is the descriptor. I'm going to run this special get function before and I'm going to give you what that returns And you end up getting a bound method That's missing an argument Before we have a set of puzzles I want to show you a fairly simple example And I want to work through it line by line and I want I want you to see Something about in binding something that should hopefully be obvious in this code at the very top line I create a class called my class It has a single attribute and it called message and it's just a string pygotham is awesome I create an instance of that class object I say some message some variable called some message is equal to obj dot message You may have heard me use the term bind name binding Name binding is often how we refer to the process of creating a variable a name And saying what it refers to so I'm saying the some message variable Where first got returned when I looked up the message variable on the instance called obj I change What's inside that class I change my class dot message on this line Will some message be effect? Of course not I looked it up on this step and this step here Is too late according to the protocol on this line and any changes I make to the class aren't going to re-invoke the protocol This line doesn't say re-invoke the attribute lookup protocol It just says give me whatever the value is that was set before and no code is run It just spits out pygotham is awesome. This is distinct from if I wrote object This says invoke that protocol and give me the latest value, which will include the change I made on this line So then these two lines should be obvious or these two These two form formulations should be obvious and why they're different Let's assume that there are many lines of code separating the import state In the very first example, what do I do? I do an early binding I say give me the y attribute out of the x module as it exists right now And for the duration of my module, I will always use whatever that variable was set to As of the time when I imported it and I won't reflect any changes people have made To the y variable in the x module On the in the second example this line says something it says x dot y x dot y is a dotted lookup It says invoke the attribute protocol every time I print out x dot y I'm saying I'm going to go to the x module Give me what the value of y is invoke the protocol and maybe that might have changed somebody might have patched something Something might have swapped something out here. You'll in the very top You'll never reflect a change that was made after the fact after this import line to the things in the x module Here you're always looking up the live value And there's a distinction here This is why both of these syntax some cases you want one in some cases you want the other in many cases because very few actual modules in the standard library or on pi pi actually use this They usually have getters or setters at the module level for setting state But you can think these are actually two very different things to ask for and they're in meaning In between these two or distinguishing these two So let's take a look at two final examples Here I have an example where I'm going to take I'm going to show you some actual monkey patching I'm going to take the length function and I'm going to add 10 to it So when you ask me for the length of the string abc 3 I'll give you a 13 That's kind of weird and probably not something that would be very useful Now in order to make this work, I need to make sure that inside this my length function I can call the actual length function If I were to say my if I were to call len here And I were to overwrite what length was set to then I might use I might have some infinite recursion here Where I keep trying to call the same method. So what I do is I use some I use a closure in order to bind the value of len It's not that important for how that works, but you can see that it just works here This len function instead of returning 3 returns 13 What I'm going to do is I'm going to take that exact same into a module But because this kind of binding using the closure is a little bit obscure And it's not something I've covered in great depth in this talk I'll just rewrite this to be a little bit simpler I'll just say at the very top line the old length the original length function Is assigned to this variable called old len and that's what I'll call here and I'll just add And on this line, I say len and built ins dot len equals my len Now notice one thing that I didn't say explicitly, but every time I have a cell which has a Percent percent python on the top that's running in a brand new python interpreter And it's not reflecting any of the I've done in this notebook So here in this script in this standalone script. I import my module called len mod and I print out one two three I've overridden What the length function does To add 10 to every value and this just works This is an example of a not very useful but a fairly straightforward monkey patching in python You can monkey patch all of the built ins in python Anytime you call a function of python. It has to see where did that function come from Most of the time that's defined in the built ins module Some of the time you can patch those Historically before the introduction of things like import books Built-in functions like the import function in order to add custom ways to import code Like importing code when you have to go and download it from a url This kind of patching is not that useful. That's not that common But it does show you how it works and why it works one thing I would like to say is Well, actually I'll I'll show you the next example a more practical example of this is somebody has Written some code for you and instead of my class is now your class and they define a method on it called msg I don't like the way you wrote your code. I want to change. I want to change that class I have an instance somewhere in my program of the class that you wrote Let's think let's take a look at some ways that I can change this Using the same code as before Let's create my own function my message that says py got them is awesome And it has great. You're not enthusiastic enough for my purposes So I want to patch your class. How can I do it? Well, I can just say your class dot message equals my message I'm just swapping out one function for the other. I don't have to worry about bound or unbound methods This will work exactly the same way yours worked and now when I call object dot message called the one that I patched However, there's a qualification because you might not want to change The existing class because if there were 10 objects that were created as instances of that class You'd be changing all 10 of those You might only want to change in a one very small very narrow case in the case of only one instance Well here if I say object dot message equals my message what happens This seems to work But there's one there's one corner case and one important qualification Let's say I did something I used a cell variable. I said this is an instance method. I'm swapping out so it should look the same It should have a cell variable as well. When I do this patching I get an error not enough arguments. Why not? I patch this on the instance Remember the descriptor protocol Only kicks in to create a bound method once you move past the instance to the class Because we never move past the instance to look this up because the object itself in its dot dict Had this definition This doesn't work and I want to show you how this works and then we're done So one way you can do this is you can perform the binding yourself in the types module They're in type and you can just do the binding yourself So here I say object dot message is a bound method of the my message function Where the self parameter will be filled in with whatever object is and this just works And I want to show you one last way actually. I'll keep that on the screen for just one second Then the very final set of slides. I'll show you one last Here's your class. I create my own class And I swap out the classes at runtime. I say this object is not of your classes of my class And this just works, but if you want to find out why you have to come back to pi gotham 2017. Thank you so much I don't think we have any time for questions. So we'll probably take questions outside the room. I hope you enjoyed this talk