 Thank you Dugal. First, a quick survey. Who knows what introspection is about? Could you just raise your hand? Okay. Who has already used it in his own code? Okay. For the people who have not raised their hands, don't worry, you are not alone. It was my case a few months ago. And that's why we had this idea of talking about introspection. How did this start for me? We were working with different people on writing a test for hiring Pitanista. We had already tested for other languages, but we had known for Python and we saw that it was important to have such a test. Among the tests, we wanted to verify that the candidates could write specific type of functions, typically like a generator. We wanted also to enforce typically the use of least comprehension by asking to modify a code by changing just a line. The problem is that this test is of course automated and that we wanted to check if the candidate had indeed followed those constraints. And all the things we had are the three lines that you see there. So roughly we could import a module called answer, including the function that the candidate had developed. Then of course, how could we do that? No idea. So who should you ask? Of course your best friend. And who is the best friend of any software engineer? Stackoverflow. And of course, we found the answer there. And the answer is, oh, just use the library called inspect. Just call get source and is generator. And then you have the possibility to answer that original question. What is exactly introspection? So let's go back to dictionary and ask introspection, two possible meanings. One in psychology and the other one in object oriented programming. Okay, let's select the first one. And the first one is just ability of a program to examine a trend time, the type or property of an object. The ability of a program to examine a trend time, the type or the properties of an object. Hey, that's cool. That's cool because in Python, everything is an object. A function is an object. A class is an object. Everything is an object. Isn't that cool? So you can in fact go and look everywhere in the different elements of your code. In fact, I was already using introspection without knowing it. You probably recognize that snippet of code. It's the code which is used in order to separate the code which should be executed when you are running your module directly or when you are importing it. And the difference is made just by looking at that specific attribute, which is the name of the current module or similarly of the current name space. So by using that, I was in fact using introspection. But now let's look at what the standard library can offer. And let's start by the built-in. So this is a list of functions which is part of the standard library which can be considered as being introspection. You have things like ID to get back the exact identity of the object you are looking at. Dear, which is providing the list of attributes and methods type, which is providing the type. As attributes get attribute, relatively explicit. Is subclass is instance callable, providing you information about a specific characteristic of your object, as well as different functions for accessing variable depending on the scope. The standard library provides also a lot of attributes. There is a really huge list. There was a lot of effort done by the core developers in order to provide a uniform interface for the object. And if you look at the documentation of the inspect library, you have the full list of all the attributes you can call. Typically if you want to get the documentation of a module, a class, or a method, it's important. What does that mean in our case? So you have a function, a stupid function, nothing important there. What can you get from it? You can ask, of course, for the type, and you get the type as a function. You can ask for the name. So you could ask what the interest of asking the name of something I am providing. Well, this can happen when you try to associate a different name to the function. So it's important to be able to get back to the original name. You can, of course, get the documentation. You can get the module it is part from. You can ask all the methods and attributes that that specific function has. So this is a list. But the standard library provides even more than that. It provides a dedicated library called inspect, whose objective is to inspect live objects. It's a pretty big library. It's around 60 different functions. But providing four main services. First one, type checking. Is it a module? Is it a class? Is it a method, et cetera? Getting the source code. Get the module. Get the source file, et cetera. No problem to ask that. Inspecting classes and functions. What is the signature of a given function? What are the different parameters in signature? And examining the interpreter stack. We are not going to discuss about the last one. I refer you to the presentation of Alessandro Molina from Monday on Python code and introspection for that specific point. If you want to know more, there is, of course, as usual, a pretty good documentation of that model. Let's go back to our example. What does that mean we can do with that library? We can get the exact parameter. Inspect.signature. You provide the function and you look at the attribute parameters and then you have an order dict which contains the different parameters part of the signature. And, of course, you can ask for the complete source code of the function. Okay. That's very nice. We have seen a lot of function that we could use. But in practice, what could be the use of such a stuff? And so it's time for a demo. Thanks. Now we know a little bit more about introspection. More importantly, we know how to do basic things. We know which function to use, for example. But what about doing a bit more? Doing real things and having a bit more fun? For example, we could check that our doc strings in our code are always up to date. It can be very interesting because, I mean, you communicate to the outside world with your documentation. And the guy who will use your library will see your doc and will know what parameters to call your function with. So it's important to have a documentation that tells the exact number of parameters to give and what type of parameter to provide. As well, when we will be able to get valuable information from our doc string, maybe we could check that the value provided at call time are indeed of the right type according to what's documented. Let's begin. First, we have to agree together about one syntax for doc string because there are actually different syntax that we find in open source libraries. Let's take this one. I write colon, then an optional type name, then the name of the parameter, then colon and followed by an optional description. What is relevant for our use case here is to be able to retrieve the whole list of the parameters, names, and also if they have types documented or not. Here is how you just get the documentation string. Nothing new here. You just invoke the attribute doc. If you have the documentation string here, it's how it looks like. You just have a string, the row, original string, and if you have no documentation on your function, you will have none. We have to extract the parameter. Here is a piece of code that is not very funny. We just have to parse the doc. I have nothing very interesting to learn. To teach to you, just create your regex for your syntax if you don't have already a library to do this. Here we extract, I don't know if you see if I select, but here I'm looking for the keyword param in the documentation string. Here I'm getting the name of the parameter and here the optional type. Then I will just return all of the couples containing the name and the type of each parameter. But just let's run it to see how it goes. I have, I can see here my four parameters as expected, A, B, C, D. I can see that C is assigned to the type, to no type of parameters. Here I get none, which means there was no document type. And for the other one, I get just what was written as type name in the doc and that's all. Nothing very smart here. Now we have to do a little bit more. If we do validation afterwards, we need to load the type. We don't just want to have the name of the type. Why? If you just wanted to use the built-in type, which gives you the type of an attribute, you could check that the representation of the type, which is its name, fits the type name in the documentation. But if you want to have a number as input and you write this in your documentation and you provide an integer, an integer is a number, but the type name of an integer is not number. So I have to load the actual type that was written in the documentation and then check if the input provided fits. And so we will use instance for that. But you know that. I also need another thing. It's to import, I mean, being able to import a module because in my documentation, in my doc string, you saw that I was referring to a custom type called example type that I was just defining just above in the current module. So I have to be able here to import the current module and use the function get at just above to load example type from the current module instead of from built-in, for example, as it was above. So here I can see that it has effectively loaded the type example type. And here a basic function I will have to use at some point. It's instance to check that value, for example, three is actually an int as I told us before. Let's put all these little pieces together in function that will take the name of the type and optionally, not optionally, but name of a module and I will optionally use the module to load the type from this module. But if there is dot in the name of the type, I don't have to because you know that if we have a dot in the name of type, then it's a fully qualified name and I just don't need anything else to know from what module it has been, I mean in what module it has been defined. So just we see that I just used a built-in function I used before get at instance and the import module function from importlib. And now I can see how it behaves. I'm able to load a type from the module main, which is a current module. And here I try to load a type I didn't actually define in the module main and I can see that I have a proper error saying that this is at least not a valid type. If we just look at this module. Okay. Now let's put again what we just did together in a function that will very simply just extract what we are able to extract from the documentation. Get the name and this time load the type from the type names we get from the doc string. And here it's how it can be used. It's very similar to what we saw before. But this time for each parameter name we have an actual type here. The type function and here still none for C which has no type constraints. And you can see by the way that the built-in types and the custom types have been imported without any problem. Okay. So the goal for a reminder was to be able to verify that the doc string is updated according to the signature, the actual signature of the function, what parameters it takes. So now the second point is to be able to get the signature. The good news is in Python 3 we have a function that does that very, very easily. It's called signature and it returns an object of type signature. If you get a representation of the signature of the object, signature, you see between the parameters that your function takes and what is useful among others is a parameter called parameters inside this signature object which is interrable. Actually, I think it's a read any map, I mean, addict, read an addict. And you can, for example, get the number of parameters very intuitively. You could also get the default value of a specific parameter by its name. So it's very easy as well. You just give the name and use on the parameter object the attribute default and here I'm expecting to see 42 because it was a default value I gave to parameter D in my signature. And now it's a big piece of code but it's really not to be afraid of just read the doc and it's really enough. Basically, we just want to check that if we provide a default value in the signature, it has the right type if the type is provided in the doc string. So we are just checking that the doc string is consistent when it comes to the types and the default values in the signature. It's quite straightforward, actually. We just use what we did before, getting the parameters names and types, getting the signature and comparing them. Here I ran what I just did on my function of my testing function and I'm expecting to not raise this function is supposed to raise an error if the documentation is inconsistent. So here I can see it's consistent. For the example, of course, I provide another function that is badly documented. You can see that it takes only three parameters and I just copy paste in the doc string of the other function which had four parameters. So obviously, the last line here which tells that it takes parameter D is not relevant. And here, when I run the function, check the committed parameters on that function, it tells me that it's wrong because signature takes just ABC. The function just takes ABC and documentation tells about ABC and the present B. Okay, so the good news is we just achieved our first goal. I don't think it was too hard. So now I can verify that wherever in my code I want to check that, you can check that doc string is up to date. Side note, doc string is a doc that is assigned to a function but also to a class or method or other things actually. So you can do that everywhere if you want. So second goal, yeah, I'm sorry. I show that the second point is yet to be done and now it's a bit more at runtime. It's a little more an actual introspection because I really need now to know the actual state of my input at runtime to be able to check it against the expected types according to the documentation. So we see here I'm invoking the function called signature provided by the module inspect. So nothing new here. But now I'm calling another method on that object which is called bind. Very simply, bind you just give it all the arguments you would call your function with when you want to just call it. And bind will tell you that this value would be bound to this parameter and so on. Just look. Here it tells that other functions that I provided as first parameter will be bound to the parameter A because A is the first parameter. And then I explicitly invoked bind with the parameter C which was actually the third parameter and you can see that there is no mistake. It correctly bound the value 2 to the parameter C and the instance of example type to the parameter B. So it's no magic but it's very easy to have that and not have to implement it yourself. Now that we have that again and again we just create a function that invoked what we did before. I am able to get the names and types from my documentation string. I'm able to get the signature and now I'm able at runtime to bind the values that are given to the function to the parameters and the parameters to the documented types. So with this again it's not very big deal. Just it's a decorator. If you don't know what the decorator is very simply here. My use case of a decorator is I want to add a check step before doing the rest of my function behavior. I don't want to touch what will do my function but I want to decorate it in order for it to just do a check on the input before doing the rest. So the usage is that on the first line I just add the decorator to any function that is defined just below. My function can do whatever I don't care. Here it does nothing and I just add the decorator to tell at least my input should be evaluated first. And that is what I will do here. I'm calling the function that I was decorating just before. The function still has the same name and when I call it despite the fact that this function was not doing anything I have an error because the decorator added a step to that which was a validation step on the input and I also have a valuable information to work with because I know that my mistake when calling this function was to provide an integer as parameter B which was expecting actually an instance of example type. And that's it. We achieved our two goals and so well done. Okay. Then we promised to try to answer the question when should we use introspection? It's a tough question. The general principle should be relatively clear. You should be only using it when you don't know the information at compile time. Well, that's the principle. No in practice. We don't pretend that we have the truth but we have tried to identify some good candidates for places where introspection could be used. The first one is about exploration, learning and debugging. It's the idea that introspection allows you to have an in-depth view of what is happening with your object. And that's very nice if you want to learn typically the Python internals or to better understand what is happening inside your code. The second one is related to IO. That's typically the case when you receive from outside an unknown object and that you want to adapt the behavior of your code depending on the specificity of the object. On the output side it's more about when typically you want to serialize any objects in order to log it or store it for future use. Then in that case you have to adapt your behavior also depending on the specific type of the object. The next one is when you are missing so much the polymorphism of other languages or typed languages. So you would like to have a single function which is processing whatever type of object that you receive. And in that case of course you will have to adapt what is happening inside your code to the specific type and specificities of your object. The next one is if you are missing also interface and typically an architecture of plugins then you are supposed to use classes or modules that you don't know and where you would be interested in checking that well indeed I have all the methods I need or I am expecting. And the last one is close to the first part of the size of UG is about metaprogramming. It's all the things around your code like I would like to check some convention, I would like to check some coherence within my code. I want to be able to do auto completion when I am at command line. I would like that the type of things where you can also use introspection. Just three words of warning. The first one is about performance. Of course when you just access an attribute of a class it has almost no cost so it's fine. But you could have some cases where introspection is more expensive. And also as it's a good practice in programming don't check multiple times the same things through your code. So check it once when you receive the object from outside. As far as portability, depending on the Python implementation some objects could be not inspectable. So that's something to take into account if you want your code to be working on the different Python implementations. And finally maintainability, depending on the ID you are using it could be you could receive more or less help when trying to do refactoring because of course everything is just a representation like a string of classes when you are calling them. That's all for us. We would like to thank you for attending this session. And yeah, by the way, Criteo is hiring if you are interested in working on really big data. Thank you. Okay, we have about four minutes for questions if there are some questions. So I have one question. So when you're doing introspection for some of the examples you have it means you're executing the code when compared with using the ASD module or some other way of evaluating static analysis. I was just wondering if you combined the two approaches or, I don't know, for example, validating the doc streams you could do either way. Yes, true. Actually we created a slide for showing how ASD works and how it's easy to create a visitor. And first it didn't fit in 30 minutes. And it was also very, it's another subject already. If you do ASD you're not introspecting because you're analyzing the very structure, the static structure of your program. And in fact when you visit the ASD in Python you have to load a source file. You are not inspecting the actual code that is running. You really are loading something else even if it's the current file you are evaluating. But you are loading something and creating the ASD and visiting it. And you don't have the memory state of the current execution. You don't have the state of the data that is going through your functions. So it's not introspection actually. But when it comes to doc stream, yes, the first example was to check something that truly is static. And it was just to show here that a way to check it was to use introspection in this very case because it's very easy with Python to just load your module. And because you just loaded it you have something, you are running, you just have a state and you can see the state of function. So it's just a way to do this. But the second example, the one where you check the inputs is the true, I mean the pure theoretical example of introspection.