 Okay, so welcome back. Let's find out if monkey patching is a magic trick or a powerful tool. Welcome, Elizaveta. Hi, my name is Elizaveta Shashkova and today I want to tell you about monkey patching in Python. In this talk I'll try to give the answer to the question. Is monkey patching is just a magic trick or is it a really powerful tool? Just a few words about me. I'm a software developer in the JetBrains company. I work in the PyCharm IDE and I develop Debugger in PyCharm, which is also used in PyDev, the Python IDE for Eclipse. Monkey patching is a dynamic modification of a class or a module at a runtime. Here is a simple example which illustrates what monkey patching is. The standard module math has a square root function, which returns the square root of a number. But if the number is negative, this function raises an exception. We want to change the behavior of this function. So what can we do? We save the original square root function to the attribute original of the module math and replace the square root attribute with our new function, save square root function, which doesn't throw exception for negative numbers and returns that not a number constant. Which is defined in math module. So what happened here? We've got the standard module and changed its behavior inside our program. That means that monkey patching allows us to change third party code without changing the source code secretly. It explains where does the term monkey patching comes from. It believes that it has come from the term guerrilla patch. Guerrilla is a member of group of soldiers who don't belong to a regular army and who fight in a war secretly. This term is referred to changing code secretly and possibly incompatibly with other such patches. It's believed that later this term was transformed to guerrilla patch, which sounds similar, and later it was transformed to monkey patch. As you might already notice, all these terms has rather negative meaning. But despite of it, monkey patching exists in a number of other programming languages. For example in Ruby, you can redefine any method in any class, including the standard modules like string or array. In Ruby, it's named re-open the class. In this example, we replace the standard upcase string modeling with the reverse method. As we can see, the monkey patching is Ruby is almost a part of the language philosophy, because there is even a special simple syntax for doing this. So what about Python? I believe that everybody here remembers the Zen of Python and one of the most important statements in it. Explicit is better than implicit. Monkey patching doesn't satisfy this requirement, because instead of raising explicitness, monkey patching makes Python code less readable and more difficult for understanding. And even in Ruby, we can patch everything. In Python, we can patch almost everything, because on Python level, we can't patch built-ins defined in C. There are some solutions for that case too, patching on C level, but we won't discuss them today. So as you can see, monkey patching is a very interesting tool. It gives you a lot of opportunities and also impose the great demand on the person who uses it. Now let's have, like every object in our galaxy, monkey patching has its own light and dark sides. Now let's have a short survey. Please raise your hand if you're on the light side of monkey patching. Well, and now raise your hand if you're on the dark side of monkey patching. Okay, I would like to start from the light side of monkey patching and consider how we can use it in real life. So as I've already mentioned, we can change the third-party code without changing the source code. And for example, if you found a bug inside in some library and you know how to fix it, you can just you can just apply patch to this library and use the correct version of the library inside your program. But remember that nowadays almost every library is open source. So the best decision is to fix this bug and create pull request and share your fix with other users. The next important example of usage of monkey patching is code instrumentation. You can use monkey patching in order to add performance measurements to your code without changing the code of your project. Also, sometimes tests need to invoke some functionality. It depends on global settings or which invokes code, which cannot be easily tested. A lot of test libraries allows you to replace parts of your system under test with mock objects. And the next important example is changing the standard modules. This point looks similar to the first one, but it's not about bug fixing. Sometimes there is need to change the standard libraries for your own purposes and there are some libraries who do it. Later, we'll consider such an example. It's time for examples. The first example will be from the PyCharm internals. As I've already mentioned, I developed debugger inside PyCharm. And it's very important for us to catch all the new processes created in the user's program in order to add debugger's tracing to these new processes. And monkey patching allows us to do it. Here we catch new processes created with the fork function from the module OS. We define a very simple class, process manager, which takes the original fork function and calls it in the method doFork, the patched version of this function. It calls the original fork function and checks the result. If result is zero, that means that we are situated in the child process, we call our debugger's tracing function, start tracing, and after return the process ID, like the original fork function, does it. How can we use our process manager? It's very simple. We import module OS, create the instance of our project manager class and pass the original OS.fork function to it. And after that, they replace attribute fork of the module OS with our method doFork. And after that, every time when we call OS.fork function, we add our debugger tracing to every new process. The next example is about import hooks. Almost every script starts in with the import statement. And every time we're importing some module, we in fact call built-in import function. Let's try to monkey-patch it too. In the previous example, we patched the OS.fork function, but what if we want to patch some module as soon as possible? As soon as possible means right after importing. That's why we need to create import hook. This is import hook manager class, and it's look very similar to the process manager class. It also saves the original import function and calls it inside the patched version of this function. But also it checks the name of the module, and if the name of the module is the string OS, that means we want to patch the object of our module. We again get the original attribute of the module, create the instance of our process manager, and replace the attribute fork of the module. And like standard import function does it, we return the object of the module. How can we use our import hook manager? It's very simple. We import built-ins, the module whereas the import function situated, create the instance of import hook manager, and again replace the attribute import of this module with our new method. And now every time when we type import OS, it will be patched as soon as possible right after importing. And the third example on the right side is related to the G-event library. G-event is a framework for scalable asynchronous input and output operations based on the green light loop, a lightweight coroutine. It implements its own event loop, so even one-threaded programs are not blocked by the IO operations. That's why G-event library is very useful. But there are blocking system calls in the standard library, including, for example, socket, and when you use such modules in your program, you break the main idea of the G-event library, because your I-O operations becomes synchronous back. In this case, you can replace some statements like import socket with a modulus from the G-event library, which are very similar to the standard modulus, but which are compatible with it. So instead of import socket, you should just write from G-event import socket. But if you already have an application which uses such modules, there is no need to modify such import statements in many, many places, because there is a special module inside G-event, gvent.monkey. It clearly replaces functions and classes in the standard modulus, and after that they can be used together with that library. It allows pu-pies and applications and libraries to become synchronous with no modifications. MonkeyPatching has many interesting and useful applications in real life, like changing third-party code, testing, code instrumentation. MonkeyPatching is a very powerful tool, but MonkeyPatching is a very dangerous tool as well. As I've already mentioned in the beginning of my talk, MonkeyPatching violates the philosophy of Python language. Changes made by MonkeyPatching are not explicit. When you patch something, it leads to the unpredictable behavior of your program. Changes are not documented, and they may be very unexpected for people who use your code. Also, even if you documented your MonkeyPatching, sometimes you can meet people who decided to MonkeyPatch something too, and it will be very sad to realize that you decided to MonkeyPatch the same object. The examples on the dark side of the MonkeyPatching are also very important. And this example will be related to the G-event library again. In this example, I want to show you how MonkeyPatching can lead to creation rather complicated and tricky solutions. As I've already mentioned, the G-event can MonkeyPatch some standard modules in order to easily make Python applications asynchronous without a lot of changes of code. In the docs, the creators of the G-event library mentioned that patching should be done as early as possible in the life cycle of the program. But what happens if user runs the program with G-event patching inside PyCharm's debugger? The debugger has its own event loop based on the standard modules. And after G-event MonkeyPatching, user's program is based on the G-event event loop. But two of these event loops should be separated, because the debugger shouldn't affect user's event loop. It shouldn't break the logic of the user's program. So that means the debugger should use the original versions of modules, like threading, asofic, or socket, instead of patching versions. So that means that we have a problem. Somehow, we should save the original versions of some standard modules and continue to use them simultaneously with the patched versions. Let's try to solve this problem. First, let's consider how the importing in Python works. This is a very, very simplified version of the standard import function. First of all, import function is trying to find the name of the module in the sys.modules dictionary. This dictionary contains the objects of all imported modules, and the keys in this dictionary are the names of these modules, just strings. If the import function didn't find the name of the module in this dictionary, it executes the file again, creates the new object of the module, and put it to the sys.modules dictionary and returns it as a result. Let's create some module saved modules. Here, we import socket module. So that means that after that, it is situated in the sys.modules dictionary, and the key socket is inside keys of this dictionary. After that, we pop the object with key socket from this dictionary. What does it mean? This object is not inside the dictionary, but we still have a link to it. And after that, when we import the socket module again, the import function is trying to find it inside sys.modules dictionary. It can't find it, and it creates another object of this module and put it to the sys.modules dictionary. And after that, we have two objects of our socket module, and one of them is situated in sys.modules dictionary, and another of them is in our secret module. So, how can we use it in our program? We can import bus, we can get access to bus of these objects, but after that, when g-event is trying to patch module socket, it is patching the module which is situated in the sys.modules dictionary. And g-event doesn't know anything about another module, but we know about it. So, that means that we can use bus of these modules, the patched versions and the original one, together and simultaneously. So, as you can see, the solution of the problem was not very easy, because we tried to fix the bug after monkey patching, and we had to create some dirty hex with importing. So, monkey patching can lead to some complicated bugs and to complicated fixes of them. Today, we consider it both sides of monkey patching. Monkey patching has many different and interesting implications in real life, like changing third-party code, code instrumentation, testing, working with standard modules. But monkey patching raising the implicitness of the code. It can lead to some unpredictable behavior and surprising bugs in your program. Sometimes it may be incompatible with other such patches. I believe that sometimes you can use monkey patching in your program, but you should use it if and only if there are no other solutions for your problem. If and only if you are absolutely sure that it is necessary in your case. Monkey patching is a fascinating tool, but monkey patching is a very dangerous tool as well. As for me, I am on the light side of monkey patching. We use it inside patcher, because it is a really powerful tool. Be brave, but also be careful. Thank you. Okay, so now I believe that we have some time for some questions. Does anybody of you have some? I don't know if you are allowed to do this, but can you elaborate on some examples of how you use monkey patching in PyCharm? Sorry, could you repeat your question? Can you describe some instances where you use monkey patching in your work at PyCharm? Yes, some of these examples were from the PyCharm internals. So the example where we catch new processes, we use it inside PyCharm debugger, and the import hooks, we use them inside PyCharm too. And these solutions to protect from monkey patching is used inside PyCharm. So I'll look at these examples. Okay, any other questions? Thank you, thank you for your talk. You mentioned at the beginning using monkey patching for testing. I've seen mocking like the mock module in libraries. Do you use those or do you use monkey patching directly? Because my worry is that if you monkey patch in a test, then surely the next test, or maybe I'm misunderstanding, will have that monkey patched object. In some testing libraries, they use monkey patching. Internally in the library, but not in your own tests. Yes, where I mentioned testing in the light side of monkey patching, I mean that how they use monkey patching in testing libraries. There is no need, sometimes of course you can do it, but there is no need to do it on your own because there are a lot of powerful testing libraries, which already does it for you. Any other questions? This may be a dumb question, but can we not solve the problem of monkey patching by not doing monkey patching, by creating new objects or classes that do the same thing with the monkey patch that we want to do? So the question is, can we avoid monkey patching? Yes, by creating new objects, so we will only use those instead of changing the ones that we want to change. So for example in the OS example, you create like monkey patch OS, and you use that instead of the OS, instead of changing OS? Yes, in this example we monkey patched the module OS, which was already imported, and we monkey patched the function inside it. So my question is, can you not create a new OS? Yes, you can create OS, and in this case we should monkey patch it too. Okay. So there are some possibilities to avoid such monkey patching as we did, but this example doesn't protect us from the solution which was shown in the example of the dark side. Yes. Thank you. Do you have any other questions? Okay, if no, then enjoy the coffee break and let's thank again Elisabeth.