 All right, welcome everyone. So the schedule's a little bit different. You might have been expecting frienders, let's put Wi-Fi and everything talk, but unfortunately she's sick. So thankfully we've got Graham here and he will be giving his talk, here no evil, see no evil, patch no evil. Please save your questions to the end. We'll have 10 minutes for questions then. So please give a warm welcome to Graham. Thank you. So as you hear, Brenda is sick and one of my colleagues at work knew I was coming to this conference and I was chatting to him earlier. He pointed out how the president of New Zealand had recently said something about the problem of New Zealanders being too lazy and bringing in foreign, cheap, unskilled workers. Well, I'm the cheap, foreign, unskilled worker that's been brought in at the last minute to fill this slot for you, okay? And really good idea to plug these things in when you need them. So I'm gonna start off with decorators to talk about monkey patching. We'll start with decorators. You should all know what they are. It's that little at symbol at the front there, which they stole from the email addresses. If you've used decorators, you have to say decorators are useful, yes, no? Yes. They're easy to implement. They're not easy to implement. Yeah. So typical decorator, it looks simple. You can write them using a nested function or function closed, what do you wanna call it? We just create that internal function there and we're gonna use it with our bit of magic decorator syntax and we apply that, that's a typical pattern for implementing a decorator. But for those who said no, this breaks introspection. That means that if I try and get the name attribute or the doc string from that function, that will no longer work. But you might say, but doesn't func tools wrap else help fix that problem? Now the idea is we've gone and used this func tool model and we're gonna put an actual decorator on our nested function and then use that. So yes, it solves the problem of name and doc but it doesn't solve all the problems. There are still issues with introspection related to things like getting the signature of the function and various other things. It doesn't work that general thing of using a function closure when you're doing wrapping a decorator on another decorator where that inner decorator is implemented using a descriptor. And there's a whole bunch of other things as well where that sort of used a nested closure, the typical way to do things, it doesn't really work. Now, a few years ago is in Auckland. Who was in Auckland? I went to the park on a few years ago. Did you go to my talk then? One person did, I'm very sorry, okay? I killed various people's brains that I try not to make my talk so intense and full on nowadays, but I did in that one. I went into the whole details of why decorators have done wrong. And yeah, I had people come up to me later, mum's later and said, yes, I very much enjoyed your talk but I had to watch the video six times to actually understand it. So I'm not gonna do that again today. There are lots and lots of complicated details about why that basic pattern for decorator doesn't work. If you go to my blog, off on the right hand side, there's a quick links section and there's a little thing there for decorators and monkey patching. You'll find a series about a 10 or so or a dozen blog posts there which go into that whole problem of why that decorator pattern doesn't work and also talks about some more in depth the monkey patching stuff I'm gonna talk about today. So please try not to implement decorators itself. But what is the solution? The solution is this module I've written called wrapped. Now you may have used one in the past called decorator module. Don't use that one, use wrapped, okay? It's sort of taken all that lessons learned from what was wrong with that one and done it correctly and sort of hides all the details from you so you don't have to worry about it. So basic decorator in wrapped, you can define your function which is gonna be your function wrapper and you're gonna put wrap.decorator as a decorator. And we got a few magic arguments here which is the thing that was wrapped, an instance which I'll mention in a minute and args and keyword args and you can do what you want with that. You can change the arguments that they go in or change the result as it comes back out, substitute things, put timing things around that or print things, you can do whatever you like in there just like you would normal decorator, it's just we're gonna use wrapped and wrapped to worry about all the details of getting it right for you. Instant point and that's where what wrap does is actually very, very interesting. If you familiar using decorators, you might have come across the problem of having to write one decorator which you can use on a normal function versus writing a decorator that works on a method of a class. A method of a class has a self argument so it's actually quite hard to in some cases to write a decorator which you need to apply in both cases because well, how do I know that I'm being called as my wrapper function and I'm applied to a method of a class? How do I know there's that magic self argument in there? And wrap takes care of that because it gives you this instance argument and if you are called on a normal function, if your decorator was applied to a normal function, instance will be none. If you're called on a method of a class, then that instance argument will be the instance of the class. If it is decorator was applied to a class method, that instance argument will be a type object and it will be the type of the class that it was applied, the class method was on. And finally, is that all we run out of options? So function, static method or function, and the class that's top on is the class itself. You can apply decorator to a class itself and you'll get, what will you get in that case? Decorator is a class, this instance, I've marked something up there, but you can determine the difference. Ah, the wrapped is a class and that is a type in that case. So you can actually work these things out and it's what's called a universal decorator and you can make those calculations of when a thing is applied to and that becomes really useful in certain things. And one of those things is in wrapped as a bit of a bonus. It's mainly there as a sort of an illustration of what you can do. I've got this implementation of a decorator for doing mutex locking, synchronization. Now, if you're familiar with Java, they have an actual synchronized keyword which you can just essentially whack on a function and it'll do all that locking for you. This synchronized decorator I've got here works in very, very similar way. So I can apply it to a function and it'll apply a lock to that function. I'll apply it to a different function while that function's got a separate lock. But if I can also apply this to a method of a class and this is where I can use the fact that in wrapped you can know that it's a method and I can do something special. So if the synchronized decorator was applied to an instance method then I can lock that instance of the class. Now, if you've come across these synchronized decorators in the past, often the problem in here in that context is that they're only are still applying it to that one function or you have to provide an argument somehow of what you're gonna lock and all sorts of weird stuff. We can apply it to a class method in that case it'll lock the class or we can apply it to a static method in which case again it's locking on the function. And one of the key things I've done with this particular decorator is it's not just a decorator, it's actually a context manager as well. So we've got our synchronized decorator on function IAM went up there but at the same time and that's locking on the instance of the class but same down further on if I need more fine-grained locking than the function as a whole, the method as a whole, I can within there go with synchronized self and that'll lock on the instance just for that bit of block of code. And similarly down the bottom if I wanna lock on the whole class because I wanna block out something that's happening in a class method then I can do synchronized on object on the type of class. So it's a bonus thing and that shows you the power of what, well you won't be able to see it because there's a lot of magic happening underneath and the implementation of that but this would not have been possible with a standard decorator pattern but I can do this with wrapped. Now, so do you wanna trust me when I say that you should use wrapped? Well, I've had many, not many but a few core developers of CPython come up to me and say can you please work on getting this in the Python standard library? So if they're saying that, believe me, it must be awesome. So if you do do decorators, have a look at wrapped but wrapped, the primary purpose of wrapped wasn't actually for writing decorators. Most people know it before that if you have heard of it but the primary reason for it was for doing monkey patching and this is because I used to work at a company called Nurellic which did performance monitoring and who used Nurellic, who's heard it, use it. Jeez, I don't know why you guys wanna use it. If you understood the horrible stuff I'm doing underneath with monkey patching into code which you're running in production, it works. But oh my gosh, you just, obviously people trust me. They'll do the right thing. I don't work there now, but anyway. So decorators rely on similar principles to monkey patching and that's why even though this library wasn't designed for decorators, all these problems you have to solve for monkey patching are relevant to that problem of decorator because when you monkey patch, you need to have all those similar things to things. You need introspection to still work. You need all those things. You don't wanna break descriptors that you may be wrapping in that because as soon as you do something like that and break the code, I'll break your production code and you don't want that and that's why RAPT has to do this very, very well. So now, what's the difference between decorator and monkey patching? Who's used Python 2.3 or earlier? A few. So back in 2.3 and earlier, there was not the concept of decorator syntax. That at there didn't exist. In Python 2.3, you had to do this down the bottom here. Function equals function wrapper function. So instead of applying at function wrapper, you actually had to do this explicit call to apply that wrapper. That could have been a global scope or even in the body of the definition of your class, you're gonna do that. So you're actually making an execution call while the definition's been read in to apply it. Now that down the bottom is what we were more commonly referred to as monkey patching because it's like more obvious, right? It's not the special decorator syntax. But the difference is decorators are applied when the code is defined. So that at thing was a part of when it was being read in. The Python language interpreter actually worries about it for you. You'll notice with that functional call, we did it after the function being defined. So that's sort of site difference. So monkey perfection is performed after the fact and essentially we can't use that decorator syntax. But why would your monkey patch in the first place? Now I mentioned performs monitoring, which is what New Relics doing is one down the bottom there. But you may need to do it if you're using a third party package, say Django. And this man here is lazy and they taking a long time to put a fix in Django and get it released. But you need that fix straight away. And you don't wanna carry the burden of running your own fork of Django code for a while. But you need this fix. So one way you can do that is use monkey patching to apply your fix in place into the code. So import normal Django, apply monkey patch and you're good to go. And then you can wait for them a few months later to get a new version Django. Is that right? The other one where you can use monkey patching is testing. Who here uses mock or whatever it's called in Python to free our current one that's called now. So it's actually a version of it in Python 3.4 onwards something. So mock is a thing where if you wanna do testing of your code and imagine your application does an actual call out to a backend web service. But you wanna test the bit of code that does that. That web service may not exist when you're doing testing or you can't have it do that. So what you can do is use monkey patching to stub out a bit of code. So rather than actually go and do a call out you'll just return some dummy data. And that's enough to then satisfy your test. And you can compartmentalize everything. And so test a little bit of your code and not really be doing end to end testing where you're having to involve all services. That's another use for it. So monkey patching with wrap, how would you do it? So what do we have here? We have our example class. We wanna apply a wrapper around this name method in here. So we're gonna define our wrapper. I'm just gonna parse the result. We can import our example and we're gonna say we're gonna wrap function on class example with that name and we're gonna apply a wrapper. And that's all we need to do. And you think, well, but I've done this in the past. Why can't I do that? Please don't do that. Okay. That sort of direct patching of methods breaks in certain corner cases. And this is again gets back to decorators things. Is Raptors designed to worry about all these things that you probably don't know. And okay, 90% of the time you're not gonna encounter them but when you do and things don't work, you're gonna go, what the hell's going on? And in this case, wrapped worries about it for you. If you wanna know what the corner case, you can ask me later. Now that particular example, we imported our class module for the thing we want to patch and we applied that patch using that wrap function wrapper. We can start to make that a bit simpler again. There's more decorators than wrapped. So we can essentially just, in this case, we're not even going to import that module we wanna patch. We're just gonna say patch function wrapper on this module, that class with that method name. And it'll go and apply it for you. Nice, simpler. So what about testing where we do not want to apply permanent patches? So those ones go and apply a permanent patch as if we wanna fix Django, cause he's messed it up. But if we wanna do testing, so for example like mock does, where we actually only want a patch applied when we're running a particular test, we wanna apply this patch, run the test, and then take it out again. Because otherwise, if we leave it in there, then it might mark up our next test and so on. So we wanna apply patches in a certain context. So there's a decorator in there for that as well, wrapped transient function wrapper. Very similar to the patch one. But the difference is that it's not actually applying the patch at that particular point in time when it encounters this. What's gonna happen is that that creates an actual new decorator, and we're gonna put that decorator then on our test method. So when we call our test method now, it's gonna go, oh, valid storage, lookup decorator is gonna get called, which is gonna go apply our patch. When it exits, it's gonna remove the patch. And that's how we get our transient decorator. Wrap does even more than that though. If you've used mock, you'll know that, yes, you can patch a particular function, but the other thing you do is you can mock out objects. Wrapped is ways of doing that as well. And there's a thing in there called what I call a transparent object proxy. So this class called object proxy. So you can apply this to another Python object. And anything you do on that wrapper will be applied to the thing that was wrapped. The inner one. What that means is that if there's only one particular method, say we're wrapping an instance of class and we only want to intercept one method call, we could do that. We could just do this one, do what we need to do, and just call the original. Anything else you do on that will just get cast through transparently. And you don't have to worry about it. So in this case, now test, yeah, we're just gonna create our storage, this time we're gonna put a wrapper around it, and then we'll call it. So this one, look up here, go through this one up here, and clear will just go through the original. So there's all sorts of magic in there for doing that sort of stuff as well. Now, all well and good. And it doesn't solve all the problems. Now, one of the problems you come with doing monkey patching is this one here. If you import a module using this syntax here, from example import function. Anyone see where I'm going yet? Got a couple of nods, yep. So if you just go import example, and that's all you do, and then go reference example.function, the original version of that function, bad name, I got tripled up this last one, as soon as I go back and change the names, is it still inside of that module? So when I go and patch that function inside of that module, then I put a wrapper around the original one. If someone does this from example import function, they've essentially created a copy of that in another module. So if I go patch the original, I've only patched the original, I haven't patched this copy. So you end up with all these sorts of problems with monkey patching, that you don't be very careful about the order in which things are done. You need to get the monkey patching in before things like this happen. The other problem is that when I had that before example of patch this function, patch function wrapper on this thing, I had to import this module that I want to apply it to. So if I've got a huge code base where I've got a whole lot of stuff I want to patch, I've imported all those modules already. But what happens if my end application doesn't even use 90% of them? I've still dragged in all those extra modules into memory and it's going to consume memory. So what's the solution to that? So a while back someone proposed PEP 369, it was never accepted. So it's not been implemented. Don't go looking for it as part of the language. But one of the things that defined in there was this concept whereby you could specify that when a module was imported at some point later in time, that you could do something. So RAPT provides implementation of that. So in the example here I can say, RAPT when imported example, and that's my wrapper there. And as long as I get this in first, then later on whenever example is imported, if it is, it will call, it will apply that wrapper at that point in time. Or it'll call my function, then my function will then apply wrapper. So it's better, but it still does require that bit of code that I had there, the patches to be done very, very first thing in your application code file. And we're still modifying something. We're still going to modify your application code, which is not great, is it? So we need a way to trigger monkey patches without actually modifying any of your application code. Okay? And you may want to do this. For example, if you're using Gunicorn, for example, and Gunicorn's broken, well, if Gunicorn you run in the command line, you're just going to run Gunicorn blah, how do you patch that? You don't want to go having your own version of Gunicorn. So how can we solve that? And that's why I have a companion package for RAPT called Auto RAPT. So we can install Auto RAPT. And this time what we're going to do is we're going to bundle up our patch actually in a module itself and install that module. Now, if you've ever written modules yourself, one of the things you create is a setup.py file. You create a setup.py, which eventually declares what your module is, what the bits are in it, and all those sorts of things. And then you're going to do Python setup.py, well, Sdisk, for example, it gives you a table you can then install. So we're going to build a module, we're going to put our patch inside of that. Now, one of the things important here is that, this bit here, entry points. When you use setup tools, rather than distu-tils, it has this concept of entry points. And it's just a way of tagging information on a module really, which you can look up later. And what I've declared is I've declared this entry point called RAPT-Whiskey-Ref-Debugging. And in that is essentially just a list. And it just says, Whiskey-Ref-SimpleServer, and here's a function. So I'm going to declare a module called RAPT-Whiskey-Ref-Debugging, and in that side, that's going to be a function called apply patches. And Whiskey-Ref-SimpleServer is, I'm referencing this case, a module that's in the standard library, and I'll get to what I'm saying. So my code module then is, there's my apply patches function down the bottom. And this is the actual wrapper that I'm going to put around it. So the idea here is I want this apply patches to be called when this target module is imported. It's going to call that, apply my wrapper to put a timing function on this. So I can time that. How am I going to get that to actually work though? I've got a module. I've got an example bit of code, which is going to use Whiskey-Ref, for example. How do I get the two together? And this is where AutoRap comes in. Now, this is going to scare some people who know about this stuff, I'm sure. All I'm going to do is set an environment variable. And that environment variable, all it's going to have is RAPT-Whiskey-Ref-Debugging, which is the name of my entry point. I'm not going to run my Python app, and that's it. And there is my debugging that my wrapper did. I have not modified that code, yet it worked. This is scary. This man knows where I'm going. It's probably very unwise, and I think people probably were going to do it after a sec. This is really, really weird and dangerous feature in Python. When you install a package, in site packages, depending on how it's stored, there's this thing called .pth files. If you've ever gone in site packages directory and you've seen these .pth files, the main use that people will know about them is as essentially an alias, or a simlink type thing, where it says that, what it allows you to do is, if I install guinicorn, for example, and I've got version 19 or 20, whatever it is, it'll put guinicorn-20.0, whatever as a directory name in site packages, and then I'll have a guinicorn.pth file, which actually then just has a name in it, which is the name of the version directory. So it's a funny little thing like that. It means you can install things with under version names. But there's a weird behavior in it. If the line, start of the line in .pth file starts with import space, it will take that line and execute it when Python is starting up, okay, before anything else. It was originally done to do, to allow people to add in new codecs for char sets, or, I don't know the right terminology, something to do with unicode and all this sort of mess like this, where people wanted to actually put extra codecs in there. So this was to allow people to do that bit of execution to register their new thing with Python. So I'm using this. It's gross hack, but it works, it works really well. So the other thing about this is, yes, it works. Now, I mentioned how I've created a package out of my patch. Now for those who are using New Relic, you'll know that it was just installed New Relic and it just worked and did everything. Inside of that was this huge number of different sub-modules in there for patching all sorts of different things, Django, Flask, Request, and all these different things, but they all had to be in that one package. Now, from instrumentation perspective, what's quite cute with what I've done with Autorapt is that potentially you could, if we want to say, oh, New Relic's great, we can go and write our own one, but write one module which is the core of the instrumentation for performance monitoring system. But all of the different instrumentation packages and different frameworks could be all separately installed packages and you only install the one you want. And that way you wouldn't be dependent on the original author of that performance monitoring package to actually implement it because you could bundle one up yourself and put it up on PyPy and people could use it. I thought it was really interesting for that, but I haven't got around to doing about it yet. So reasons to use Autorapt, create better decorators, my awesome Fred's Synchronization decorator, but also a safe mechanism for monkey patching. So that's wrapped and I'm wrapped about it. I hope you are. All right, questions, yes? I'm just curious about, you had the synchronized decorator, do you have an unsynchronized one as well for the converse situation where you have a synchronized method and you want to drop out of the atomic operation for a moment? Why would you want to do that? You might want it generally synchronized and maybe you're doing some IO or something, I don't know, it's a general situation. Wouldn't you just not put it on there in the first place? Because the whole thing is it's a synchronized decorator which is applying it, so if you didn't need it, just don't put it on. I don't understand why you want to come in after the fact to try and undo it, but you're going to have at synchronized, at unsynchronized, or you mean in a testing context. Okay, couldn't that get dangerous? I'm having flashbacks of async programming here. Can you port PyContracts to use Wrepped, please? Can I? Port PyContracts to use Wrepped. PyContracts? PyContracts. You decorate your functions with a contract? I've never heard of it. Yeah, and the stuff was up, exactly as you said, stuff's up the signature. Talk to me later. The decorator's obviously used in lots of different places, and I keep pulling this, man, it's their fault. It's Django's fault. Django has one of these things when they have a problem where they have this decorator in there. They create a decorator in Django, and yes, it works on a normal function, and then you have to actually create a second version of the function using another function wrapper to turn it into an instance method one. But also even the Python Standard Library, there's things like LRU-Cache now. And, well, it's going to muck things up in the same way. Well, if we ever get this in the Python Standard Library, I'm looking at all these other things inside of the Python Standard Library. Would they let me change that to use this? Any other questions? So I'd have to look at the details, but I remember trying to do things that write something that was both a decorator and a syntax manager at the same time, and that's rather painful, and I'm just wondering whether RAPT helps you with that particular task? No. You can go and look at how I did it, but it's really convoluted. I never sat down and thought about, could I actually create something to make it easier? You probably could, but I never did because I was doing it in a one-off case. What are your thoughts on licensing and monkey patching? There are some licenses that allow you to use a module in the library, and then if you're... Are you modifying it if you monkey patch it? Oh, this is... That's... I don't know. I don't have a simple answer. It just reminds me of the whole argument about using GPL modules even in a Python program, because there were some people who say that, and the general opinion seems to be, but I'm still very nervous about it, because I'm using GPL Python module and using it in other things. It's not strictly linking, so it's okay. You're not poisoned everything else, but licensing is a scary area, and I... I don't know. So, assuming you get this in the standard library, when would you recommend people use this as opposed to Mock? Mock... This provides the building blocks which I think you could use to make a better Mock. It does not provide the equivalent functionality to Mock. This is a much more roll your own solution. So Mock tries to be this one do everything type thing, and I've used RAPT for doing testing as an alternative to Mock. But yeah, you have to roll things yourself as you go. You're only patching out the little thing you need, whereas Mock just replaces the whole object. It's not wrapping and then letting it change one little bit. So you can't just say, drop Mock and use this. It's a lot more work to cut over. This is a lot more... Okay, I won't say it's more flexible. I'll say at least we'll ensure that the integrity of things like inspection will work, because it depends on the use case. Any more questions? No? All right. Thank you. You can now... What do they call when they get rid of an illegal immigrant out of the company? Deport me? And I'll go back to Cindy. All right. Thank you, everyone. Thank you.