 Thank you. Hello everyone. So I'm really gonna be really excited to see how this talk goes because I got asked to do it yesterday and I said I would do a talk on the condition that I didn't have to do any preparation So my goal is to teach you the Python object model The method resolution order for multiple inheritance and meta classes in 20 minutes and have five minutes for questions left We'll see we'll see if I get any of that. My name is Michael Ford. I've been a Python developer Well for about 15 or 16 years a professional Python developer for 12 years I started my career in London working on a desktop application written in an iron Python There weren't many people using iron Python at the time. So I wrote a book on Python in action I called the passion for testing Wrote a library called mock which some of you might have heard of and is now in the standard library as unit test mock And I'm now working Having made a few jumps fire both canonical and red hats I'm now self-employed as a Python contractor and trainer But please don't take this as too damning an indictment on my training abilities Okay, so Can you all hear me all right by the way? Okay, I will stop where and can you all see the script? Can you all see the screens? Can you read the code on it? Is it good enough? Okay, great right. So this is a very standard Python class We you can see the code there. We have an initializer not a constructor. It takes self as an argument So the object has already been constructed at this point if you want to over Python has to phase object creation if you want to override The constructor the construction of the object you need to turn off caps lock on your computer and Then you override new which takes the self and any parameters the same the same as in it And then should Dunder new return. Oh, what Tom? I'm sorry I do get to unexplain Dunder the Python magic methods the Python protocol methods methods that typically you don't call yourself But the interpreter calls for you under certain circumstances they start and end with double underscores This is something that Python inherited from a C convention And it makes them nice and recognizable if you see a method starting with double underscores ending with double underscores You know it's one of the protocol methods one of the magic methods And you know that it's something that the the interpreter is going to call for you and typically you don't call them directly yourself But double underscore new double underscore is a mouthful and so I and some of the Python community like to call them the Dunder methods Dunder Mifflin right, so Python has two-phase object Creation And if you want to override the constructor That should work. I didn't test this or plan this at all okay so long as Dunder new returns an instance of the class or a subclass of The class the initializer will be called Dunder new is free to return whatever it wants if it returns something That is not a subclass of the class being constructed It will it will not initialize it and this is actually really useful mock makes a useful Makes good use of this technique Mock objects allow you to customize their behavior including customizing the behavior of some of the magic methods like Dunder get item and Dunder set item so that you can use a mock object as if it was a dictionary now These magic methods they're typically looked up on the class rather than on the instance This is one slight area of confusion in the Python object model. So if I do If I index or technically subscript into an object X there Python is going to look up The Dunder get item method on X, but it's actually going to look up It's not going to look it up at all there are fast paths inside the interpreter for that That is how it's going to find get item This means if you want to customize the behavior of Dunder get item you have to do it on its class Which if all mock objects were the same class if you changed Dunder get item on one the behavior of every mock object will be changed so what actually The mock constructor does is it creates a new subclass of mock or magic mock Inside the constructor and returns an instance of that subclass and then you're free to tinker with the class of the mock object customize the behavior of the magic methods without interfering with any of the Without any of the behavior of the others so it looks something like this it doesn't actually do it this way And we delegate the actual construction To object Let's see if that works. It's a long time since I've tried Messing around with anything like this, so it was called thing wasn't it? We should actually find our tea is an instance of New not many use cases for doing that Okay right So what I was intending to talk about is the instance attribute X here We have an initializer which doesn't do very much It sets the attribute X on the object on on the instance and Perfectly standard Python when I look up the attribute X on the instance t we get back the value three Actually the full rules of how Python does that look up are way more complicated than you might imagine Mostly implemented in object dot Dunder get attribute Okay, so that's straightforward. We have an instance. We have an instance attribute. We fetch the instance attribute What about when I do something like? the also entirely Un Astounding tell me X. I'm looking up a method on that after on that object We get back an interesting object But so let's take a step back and look at a bit of what Python objects are So you've probably heard it said that Python is built on top of dictionaries dictionaries hash tables associated for arrays keys mapping to values In Python namespaces are particularly important and namespaces are names mapping to objects and We have an object here and We have a set of names some of which are valid and some of which aren't So Along with this object we have a kind of namespace that goes with it which stores this mapping of attributes to the corresponding objects behind it the The corresponding object that the name t.x there points to is the integer object three So under the hood and this is probably not news to many of you There is an actual dictionary Storing these attributes and it's the Dunderdict attribute. It it's not considered Particularly private. There are lots of use cases for playing with this directly particularly for working around the descriptor protocol We can see this is a real dictionary. I can look up the X attribute And I can do things like set arbitrary new attributes and now my t object Has a y attribute so operations on the attributes of a Python object correspond Directly is very much the wrong word because there are lots of slips in between But it's not too much of a lie to say that they correspond directly to operations on the underlying dictionary So this is if what I like to call When I've had a bit more time to think about what I'm going to say Python from the inside out This is not thinking about Python objects when we write classes when we're designing code classes and objects they their ideas they Represent something conceptually it might represent a socket it might represent a file it might represent a worksheet in a spreadsheet and we think about the object in terms of the concept it represents and a well-designed object will have an API that corresponds as Closely as possible to the thing it's trying to represent If the idea your object presents and the way you work with that object map well together It will have what feels like an intuitive API But actually still what we're working with is still Python objects and all objects have the Python object model in common So this is this is objects turned inside out. This is looking at their behavior from the inside So that's straightforward But you'll remember mere seconds ago. I was looking up a method and I'm pretty sure That my dictionary here has no tell me x entry It's not that So the attribute look up is not as straightforward as just look look at the object dictionary find what's in there and unfetch the method and in fact It's not a problem because we know where that method lives every object In Python we say Python is fully object oriented. That means everything is an object That means everything in inherits from objects So what do we got? We've got our thing class here A class is just an object. It's just a Python object The consequences of this is if I ask Python is thing an instance of object It will say true that will be true for anything you can find in Python any first-class object And this is not actually this is not just theoretical. There's certain object behavior You see that Representation they when when you display an object at the interactive interpreter you get a representation of the object Which includes the class name some information about where it came from and an ID which is a proxy for a memory location That is actually the magic method done to repra the representation And it's inherited from object. There's a certain amount of behavior including things like object dot get Done to get attributes which implements the whole attribute lookup protocol that your Python objects get just by virtue of being objects That's what it means in Python to say everything is an object. There are some really interesting consequences of that You can do Really dumb things So here I'm creating a a list It holds a built-in function apps. It holds the math module and it holds the type error exception And please don't ever write code that looks like this right and it all does What you would expect and this is the bit where you're going to put your hands in your head Exceptions are matched at runtime when the exception is raised So Python will fetch the exception type. It's looking for from the list Everything is an object Where have we got to? okay, so I Said that's Instance attributes are stored in the instance dictionary, but we can look up attributes which are not in the instance dictionary So the rule for attribute lookup is well if it's not in the instance dictionary, where is it and? every object Has a link back to its type Everything is an object and everything has a type and classes are just objects and they also have a dictionary And this is where class level attributes live class level attributes are attributes that is shared by all instances of the object so you can have arbitrary attributes and Methods now that we're in Python 3 and there's no such thing as an unbound method Methods are just functions that live in the class dictionary. Oh, what's it called? What did I call that silly method? Right, however, because I did this up on the subclass trick I Get to do something like There are many this is the thing with Python. There's only one way to do it is a lie Okay, and you see I've just pulled out a function object interestingly if I look up the method on The from the object I get a bound method back again back back something different and this is um That's the descriptor protocol functions are descriptors. They have a done to get method I'm not going to go into that but what it means is when Python finds via the attribute look up when it finds the method What it does is it it? Wraps in self and returns you a method where self is already bound in partial application. So tell me x Takes no parameters However, if I do this which gets me a function object What's the first parameter Self obviously you have to bust in explicitly because we're working directly on the underlying function So the rule becomes a little bit more complicated first We look in the the instance dictionary then if it's not found there. We check the the class dictionary Not quite true the descriptor protocol makes it more complicated. Okay, right Um We don't have a lot of time left right so meta classes in five minutes So the rule is you check all of the base classes up until you find it the descriptor protocol overrides that so properties override instance attributes So if you just add the phrase it's more complicated than that to everything I say we'll get on fine Okay, right everything is an object every object has a type Classes are objects. So what's the type of a class? What is its type? The type is its meta class and how do we instantiate objects? We instantiate our objects by calling their type. So how do we create a class? We create a class by instantiating its meta class and the default meta class is type and what will boggle your minds is That the type of type is type type is a type Okay, so type is the default meta class type is what creates classes What you need to do to create a type is you need a name you need a tuple of base classes and you need a dictionary of members And so that I just created a cloud. This is what Python does under the hood It executes the when you enter a class statement it executes inside the body of the statement the bot sorry inside Executes the body of the class creating the functions that creates a dictionary of the mapping of all the class attributes And then the meta class is called with these things the name the tuple of base classes and the dictionary of attributes Created from executing the class body. This means that that class body is a very interesting scope That doesn't behave quite like other Python scopes, and I'm not going to go into that What it means so there that's a meta class in use a class factory I don't think I'm going to get to the method resolution order probably So the typical way to create a new meta class Is you just inherit from type you override whatever behavior it is That you're interested in So here I'm underwriting done to get item on a meta class Now we have in Python this funky syntax. Is it meta class equals? now There's nothing inside the body of the class because I've set a meta class as long as I've remembered the Python three syntax correct Python is actually going to call my my new to create the class now I could have made that more explicit by By putting them something in in the call But the interesting thing is remember when I said earlier that the protocol methods are looked up on the class So here we have an object its type is new when I subscript it The type of foo is New done to get item on new is going to be used So here we're getting behavior from the meta class on the class object and the meta class is not in the inheritance hierarchy Which is an interesting corner of the Python of the Python object model? It's interesting to know that class decorators and done to in its subclass particularly are Good reasons to never write meta class But this is the machinery that's at work under the hood everything in Python is an object every object has a type classes Objects their type is typically type the default meta class responsible for creating types But you can override that and there are many reasons why why you might want to override that but I can assure you far more Why you might not want to do that? Okay, I Hope that was understandable. We're going to be coming up for question time Very shortly. I did promise. Let's look at the method resolution order So the the ret the attribute look up order that I mentioned before First look on the instance then check the class and then maybe check any base classes until you find the attribution It's returned that's straightforward in the presence of single inheritance But here you can see I've got a class a a class B a class C A and C both define a foo method and class F Inherits from D and E when I instantiate F and call foo what is going to happen? Which foo method is going to be called the fact that it's even slightly annoying to work out is another Reason why you shouldn't use multiple inheritance mixing classes are the only valid Reason so in this case It was actually the the a that happened I'm Instead of taking questions what I think I will do is I think I reckon I can possibly explain that in five minutes Let's have a look so F is an up a class class object We've explained that in the face of inheritance Python has to walk up the inheritance hierarchy in order to find the method How does it do that in the face of multiple inheritance where you have an inheritance graph? And the answer is that Python constructs a method resolution order the MRO it linearizes and the algorithm is actually called the C3 linearization algorithm, and I'm hoping I haven't talked too much nonsense If Mark is at least not scowling at me. I'm not doing too badly and pipe This is a linearization of the method resolution order Python will walk up until it finds the first one What this means is it's perfectly possible. Oh, right. How does it construct? What is the linearization algorithm the linearization algorithm has essentially two rules the rules are that children must be checked before? Parents and that order must be preserved. So in this example D should be checked before E and D should be checked before Before B and B should be checked before a because children must be checked before Parents Okay, what it means is it's perfectly possible to create inheritance hierarchies Convoluted had an inheritance hierarchies that aren't possible so If I reverse the order of these you have E and D you'll see an exception which you will hopefully Never see in practice. Oh No What did I do? I'll make it I'll make it easier. I'll put a at the front and rather than spending at precious seconds debugging this Right to type error cannot create a consistent method resolution order the two rules that I described are that children must be checked before parents But that the order they are declared must be honored having the order They're declared honored is useful because it means you can have a mixing class that you mix in by multiple inheritance the only good reason to use multiple inheritance and your you declare it first and your Overridden one will always be called before The main class and you use super this is cooperative multiple inheritance if you're mixing class uses super to to make the call up the inheritance hierarchy super will actually jump sideways because the The method resolution order in order to honor these children must be a be called checked before parents and the order Order matters you will often find that the next in the method resolution order is not up the hierarchy But a sideways jump that was fairly disorganized But I think we maybe kind of got there Thank you very much Do we have time for any questions? We have time for one or two very quick questions Hope it's I hope it's quick Abstop based classes and met classes usually considered black magic. Yeah. Yeah, they are so My question is do you believe that? Python programmers should leverage them use them. I think I think it's very useful to understand them Because you're understanding Python. It's this the same with the instance dictionaries and the look-up rules It's it's useful to know when you do foo dot X It's useful to know what's happening if you want to understand and similarly when you instantiate a class It's useful to know what's happening. So it's more of it. So no not something you should leverage Particularly, but something that's worth understanding Well, thank you very much Can you hear me I can hear you that was entertaining and illuminating I have a question about the Attributes of the object which you showed were Correspond to a dictionary. Uh-huh dictionary is itself an object, right, right? Yes. Yeah, how do we get objects all the way down? I guess you could say that it finally ends in type where if the type of type is type What creates type and the answer is somewhere the interpreter has to cheat and it creates a C-struct and pre-populated and type is never instantiate it so, you know, there is a bottom to the house of cards So please let's everybody thank Michael. It was extremely great talk last minute live demo worked perfectly so Thank you