 We'll be starting with Python and inversion control by David Seddon and now I'd like to hand over the stage to David. David, good luck for your talk and happy learning everyone. Thank you. Hello everyone. So yeah, I'm David Seddon. Thank you for having me at PyCon India 2021. I'm going to talk to you today about Python and inversion of control. Bit about me. I work at Kraken Technologies, which is based in the UK, and our product is called Kraken. Kraken's a piece of software that is for energy suppliers to run their business. You're probably a customer of an energy supplier. It's the kind of company that gives you electricity and gas into your home and also sends you bills. And Kraken Technologies came out of an energy supplier called Octopus Energy based in the UK, but we've started expanding it so that it supports other energy suppliers around the world. So it's currently being used by 16 energy suppliers in seven different countries and there are 6 million households on supply using Kraken. There's quite a lot of us. There's about 170 developers at the moment and there are nearly 10,000 Python modules or in one Python application. So why am I telling you this? Well, it's because I'm used to working with big complex systems and I'm very interested in how to organize complex systems so that they're easier to work with. And inversion of control is a key tool in helping us to do that. So what is inversion of control? Well, this is the traditional flow of control. We've got some custom behavior and that's going to call into a library. So this custom behavior might be, say, a script in Python that you've written. And the control flow is with that script. We run that script and then it maybe imports a function that we use in our library. Inversion of control is a little bit different. The control flow is with the library. That means we run the library, but we customize it with some extra code. And we actually have a name for libraries like this, which is frameworks. Now, there's an interesting thing about this diagram, which is the arrows. In the traditional flow of control, the arrow can refer to an invocation of a function. But it can also refer to the dependency relationship between the custom behavior and the library. If you got rid of the library, your script would stop working. But in inversion of control, these arrows are actually going in different directions. The framework's going to invoke your custom behavior. But the dependency is going the other direction. And this is a bit of a strange thing. So we can notate this more simply using this notation. The way to read this notation is think of it like a ball and a socket. And what's happening is the framework is exposing this kind of plug socket that's got a particular shape. And the custom behavior is going to know what shape that is, and it's going to provide this ball to plug itself into. And whenever you see this particular ball and socket thing, just think in your head, oh, that's inversion of control going on there. Here's an example with Flask, which is a web development framework. We run Flask, so our control flows with Flask. And yet we have some custom code which configures Flask so that it knows where to root its web request to. So hopefully, most of us or certainly many of us are used to working with frameworks. But perhaps we're not used to writing inversion of control mechanisms ourselves. And so you might say, well, I'm not a framework developer. Why do I need to care about inversion of control? Well, this brings us back to our big complex system. How do we make these systems easier to work with? Well, the answer is we modularize them. We break them up into smaller bits that are easier to think about independently. The problem is, I think most of us sort of insure that this is kind of how you have to do things. We probably don't write even quite small projects in a single Python module, but we tend to break it up to make it easier to work with. But I don't know if you've found, often you'll break it into smaller bits, but then the different bits seem to need to ask questions of each other or issue commands to each other. And one module might ask a question of another module, which asks a question of another module, which then asks a question of the first module. And pretty soon you have something more like this, where there's an interdependency of different modules and everything's tangled together. And the modules are no longer simple, they're complex. You can't think about them in isolation to the whole system. And once this has happened, really you haven't modularized your system. The modules aren't doing the work of a module. You've got a big complex system again. So what does inversion of control have to offer? Well, here's a slightly simpler example of two modules that know about each other. And what's the problem with this? Well, if we were to try and understand this system, we might start by trying to fit A into our heads, trying to understand it. But the problem is it knows about B. So we'd also need to understand B in order to understand A. And that means we've got to stuff all of it into our heads at once. It's like eating a sandwich all in one go. It's indigestible. The same with B. You can't do either. It's just, really it's just one big system. And there's another problem with this, which is about making changes. Let's say we wanted to make a change to B. We'd have to think about how it might affect A because A depends on B. So what inversion of control allows you to do is to pick any dependency direction and just flip it. Anyone, any dependency in any software system, you can always do it. And so imagine if we pick this top arrow from A to B and we just flip it around using inversion of control. So now we've got this ball and socket thing where B is plugging into A, but A doesn't directly know about or depend on B. It's going to invoke something in B, but only indirectly. And suddenly this is much easier to deal with. We can look at A. We don't need to worry about B yet. We can just learn A on itself. Once we've understood A, now we can move on to B. So there's a nice learning journey. But also we can change B without having to worry about A. And so if we take a step back and think of a larger system, what this allows us to do is to layer our larger systems. So layering is a common pattern of architecture. And the idea is that you sort of break it into these smaller high-level components. In the case of a Python package, these might just be the immediate child sub-packages of the root package. And there's a rule about layers, which is that dependencies float downwards. So the lower layers don't know about any of the layers above them, but the ones above can know about the ones below. And this makes it a lot easier to understand and work with the system. But there is a problem in this diagram. You may have noticed, as we've seen, dependencies tend to creep in the wrong ways. And these two are bad dependencies. We don't want them here. We want the arrows to be traveling in another direction. So what we can do, we can invert control there. So what we've done is we've kind of made these mini frameworks within this application, just in a couple of places, and suddenly we can have layers. OK, so that's why you might use inversion of control. How do you do it? I'm going to zoom in a little bit. So this is our bird's eye view. In order to achieve this mechanism, you need, in a slightly conceptual sense still, two extra things. We're going to add an abstraction, and we're going to add a binding. A and B are going to know about the abstraction, and then the binding is going to know about A and B. So looking a bit more detail, A is going to interact with this abstraction via an interface, and B is going to implement the interface. That's why they've got two different arrows. This open-headed arrow is a UML notation, which typically is used to sort of say, this is a subclass of this. But you can think of it more conceptually as just that it's implementing an interface, or even just that it is an example of one of these. So that's kind of like the idea, but I think it's time that we think a bit about what interfaces mean in Python. Because Python doesn't have a native idea of an interface. In fact, it goes in the other direction and supports duck typing. The idea of duck typing is if it walks like a duck and talks like a duck, it's a duck. And so Python doesn't care about what types things are. What it cares about is whether or not you can interact with them in a certain way. So here's an example of two functions, notified by email and notified by text message. And because these both are functions which take a customer and an event, they both implement the same interface. That's just another way of saying you can interact with them in the same way. And so you can see here we're able to store, by looping over them, store either one in a loop, in this notify variable. And then this final line, we can call it with a customer and an event. But actually this code here doesn't know specifically which function it's interacting with. All it knows is that it supports this interface. It's got this idea of this abstraction of notification. It doesn't need to know specifically that it's email or text message. And you can do this in other styles. You can do this in object-oriented style where you have a base class which is abstract, which sort of represents the interface. Now, here I'm showing you how I would represent this in Python. Python again doesn't have a native idea of an abstract base class. You can represent it in different ways. In one way, you can communicate that this is conceptually what's going on is by raising a not implemented error in the speak method of animal. That implies that we don't want to instantiate it. We instead want to subclass it and then instantiate those subclasses. That's what makes it an abstract class. So here animal is kind of an interface and it's an abstraction. And then we're implementing this interface by actually putting some working code in the speak method of both cat and dog. And then this allows us again to loop through and call speak on the animal without knowing specifically what type it is. So these are the two steps. The first step is to separate B into an abstraction and an implementation. And the second step is to have this binding step at the top, which basically is just somewhere in the system you need to actually plumb B into A so that it's actually getting B rather than just its abstraction. Now, don't worry if you don't quite know what I'm talking about yet. Just have this picture in your head because we're now going to look at specific code to do it. So here are three styles of inversion of control that are going to be ultra minimal. I've tried to use as little code as I can to show how little you need to do in order to do inversion of control. And these are the three main styles I think of if I notice that I need to reverse the dependency. I'm thinking, oh, which of these shall I use? So the traditional flow of control, this is our example. We've got a function called hello world that's going to print hello world. And we've also got a main dot PY, which is going to be the entry point of our application, which imports and calls that function. So what are the dependency relationships? We've got our main. We've got hello world, and then that's going to depend on the print function, the built-in print function. So what we're going to do is we're going to pick that bottom arrow and we're going to invert control so that hello world no longer depends on print. So let's just sort of reorganize these so they fit that little diagram we saw before. We're going to remove that dependency. And we're going to put in our abstraction of print at the bottom. And then we're going to use main as the place where we do the binding. OK, so what is this abstraction of print? That's the first question to ask ourselves. Well, what is print? It's a function that takes a string and then it's going to output it. That's the abstraction I'm going to choose for print. So let's just call it output function. OK, we're ready now to look at our first example, which is called dependency injection. You may have heard of it. Sounds quite complicated. It's actually not. So this is what's happening in dependency injection in our hello world module. Look, actually, there's no extra lines of code at all, but rather than printing hello world, it's receiving output function as an argument and then it's calling it there. We also need to do our binding. So that's going to happen in main. So here's where we pass print in as an argument. That's dependency injection. That's all it is. It's hardly any extra code. And so if we sort of think about how this fits into our pattern, at the bottom we've got the output function. That's really just sort of floating in the conceptual world, really. It's represented in hello world function signature. Print is implementing that because print happens to support that interface. Hello world calls that and then main is where the binding is happening. So that's dependency injection. The next style is called, well, I call it configuration registry. And this works like this. So not much more code, but a little bit. We've got this config dictionary and notice hello world isn't receiving an argument this time. Instead it's looking in that dictionary and getting the output function from there and then it's calling it. That's the difference. So this config dictionary is what is known as a registry. And a registry is sort of something that is used when sometimes you have inversion of control to as a container that other parts of a system can put things in and then the thing that declares that container can then look in it to find things out. So that's going on there. What about the binding? Here's the binding. So it's just going to put the print in that registry before calling hello world. And I'm just going to give a real world example of a configuration registry from Django. So the Django settings framework is a configuration registry. And this allows you to customize behavior of Django by putting various different constants in a settings file in your application. And then Django can access that by doing from Django. And then accessing it as an attribute on settings. And this allows you to have bits in your application that customize or pass in dependencies to Django. But you can also do it within your application. If you've got something in a higher layer, you can put it in a setting and then a lower layer can then access that using Django's configuration registry. Okay. Let's just compare the two that we've seen so far. Dependency injection is great if you want to vary the dependencies. In this example, we're using print, one time we call hello world, but pretty print another time, this pprint file. They both use the same interface that we can just pass them in. You couldn't do this with a configuration registry because the idea is that that registry should be immutable and it shouldn't sort of change at runtime. However, configuration registry is good if you don't want it to change because you don't need to keep on passing in the print function. So if you're repeatedly calling hello world, it's much cleaner to use a configuration registry. So that's kind of like the main strengths and weaknesses of those two techniques. The final style is called subscriber registry. And this is a little bit different. It uses a bit of a different model. So if you've come across the observer pattern, the idea is that you have an event and then other things can subscribe to that event. And then when the event happens, those get notified. This is also known as pubs up. And this is using inversion of control. Subscribers know about the event, but the event doesn't directly know about subscribers, although it does invoke them indirectly. So I'm going to slightly change the example. We're not going to care about print. And instead we're going to look at logging. A logging step is going to happen after printing hello world. And we could invert control using dependency injection and just come up with an abstraction of a logger that we pass in and then call it there. Fine. But what if we want a different way of thinking about it? What if we instead want to think of hello world as an event and that logging and maybe some other things just happens as a kind of callback after hello world? That's what subscriber registry allows us to do. So here's how to do it. Subscribers here is just a list. It's still a registry, but it's not a dictionary anymore. It's giving us the ability to store an arbitrary amount of things in it. And then after printing hello world is going to loop through the subscribers and call each of them. And notice the interface here. The interface is just a callable that takes no arguments. So let's now implement that in a separate file called log. Right to log is our implementation of that interface. It doesn't take any arguments. It's going to do the logging. Finally, we do the binding. We append right to log to that subscribers list. And then we can call hello world. Now there's one little thing here. Wouldn't it be nice if you could subscribe to it at the same time is in the same place as you do that right to log a lot of things which use subscriber registry tend to support this. You kind of want to register it when you declare it. But and we could just move this here, but this would only work because we're importing log and then that's executing this code. You need to be a bit careful with subscriber registry to make sure that the code does get imported. Otherwise it won't work. Django has quite a nice way of achieving this. So in Django, you can have models which represent your database tables and Django needs to and you put them in these models files and Django needs to know about them. So the way it does it is it uses as well its configuration registry settings and you have this installed app setting which lists all these different apps and then Django will go through and check each one of those for a models.py and if it's in there, then it will import it. So that's how Django does it. You can make use of this if you're working within Django to get it to auto discover other modules as well. So to recap, we've got three different main styles of inversion of control. Dependency injection is good for variable dependencies whereas the registry approach is better for fixed dependencies and then the main difference between subscriber registry and the other ones is about an arbitrary number of dependencies. We've also got these different dynamics about where the binding is and whether or not it's scattered or not. But a lot of it, there's overlap and there's a lot of stylistic choices ultimately. I've written a blog post about this so if you want to revise this in a different way, you can do. A couple of closing remarks. First of all, my examples have been very implicit in the way the interface has been expressed. I think too implicit. So I would strongly recommend that you find a way of making your interface explicit. There's no one right way to do this but I like type annotation and that's why I prefer the object oriented style of having this abstract class that represents the interface that you're going to be using. It's very clear here that we're passing in an animal even though it will probably be a cat and a dog that actually it gets. But if you're using functions, say, you can use type annotations to say, well, this is a callable that takes a customer and an event and returns none. Or if you're not comfortable with that, just put it in a doc string. The point is, make sure you document it somewhere so people actually understand what's going on. And this leads into the final point which is you need to use inversion of control sparingly. It can make your code a lot more confusing if you're not careful. If you think about it, a direct invocation of a function. We're used to that and it's easy to, in your IDE, for example, click on it and it will take you through to the function. But in version of control, you've got this disconnect because it's indirect and it can be very confusing. You wouldn't want every function invocation to use inversion of control. So it's actually, it's the exception rather than the rule. But in the right places, it can help make everything much better. So in summary, controlling dependencies is key. It's crucial to controlling the complexity of your system. And if we don't control them, then we naturally get a complex system because our systems model the real world and the real world is inherently complex. We end up having these tangle of dependencies. And so what inversion of control allows you to do is say, I can choose what way the direction of the dependencies go. I don't like the way that arrow goes and I'm going to design it so it goes in a different direction. And once you start doing that, you can start feeling like an architect or software designer who can layer your systems, even large systems become less tangled and you're able to work with them more easily. You identify these points for mini frameworks within your application and then your systems can become simpler, easier to work with, more productive within them. Thank you. Thank you for such an amazing talk, David. We have some questions for you from our audience. Sure. The first one would be, is it a good idea to keep the config registry dynamic and have the interface do the cleanup post the call is successful? Okay, let me take a moment to try and understand this. So dynamic config registry and have the interface do the cleanup post the call is successful. I think I might need to ask, I don't understand the question. Is it possible to have a bit more detail? Yes, just give me a moment. I think Mayank asked this question. Mayank, can you clarify what you're trying to ask? Until then, I'll move ahead to the next question. I'll leave it for a comment by Mayank. The next question would be, is it good by test fixtures compared to DI? Good question. Well, you're right. Pytest is a testing framework and it does use inversion and control a lot. So Pytest fixtures for those that don't know, it's a mechanism, rather magic mechanism where you can just include an argument to one of your tests and then it will just know that it's Pytest will then know that it's a particular thing called a fixture, just based on the name of the argument and then it will set up some state and pass it in. I think that it is dependency injection, probably. Maybe. I'm not sure. I mean, you're not, you're not, it's a good question. You're not perhaps, you are certainly injecting something and your test isn't directly depending on it by doing an import. But is it decoupling things? I think that it's a matter of opinion, but I would say, if you were to force me to answer this right now, I think I would say that probably Pytest fixtures are more about creating a bit of syntactic sugar and making your code less for the both so that you can be more productive in your tests rather than about changing the direction of the dependency. Maybe. But I mean, if anyone's got any other other opinions, feel free to write in the chat. Thanks, David. People can share their opinion in the state chat. We can move on to the next question. The next question, would it be a good idea to use immutable dictionaries for config registry? Yeah, that sounds like a good idea. I mean, obviously at some point, you need to put things into the registry. So you do kind of need to mutate it somewhere. But I mean, obviously with Python, you don't really have true immutability anyway. But I mean, I think the Django settings framework is an example of something that's trying to make things immutable a bit. And I'd be very interested to see an example which uses a bit more machinery so that you have a dick that you can't change. So yeah, I think it's a good idea, potentially. Thank you. The next question would be, would it be fair to say that Django signals are a version of subscriber pattern? It would be exactly right. Yes, very much so. And you'll notice with Django signals, I don't know if anyone's ever run into this, but you can declare signals. Well, you have signals and receivers. So the receivers are like what are subscribing. But if you're not careful, you can have like a receivers module which connects your functions to these signals and then it never gets imported. So a classic pattern is to import receivers in like the app config ready thing. Don't want to get too into Django. I'll definitely run into this. I had to spend a weekend trying to fix my post-save signals. We have another question, I think. Is there a neat way to have runtime checks on the injected dependencies types? Type annotation only works with static type signals. Well, I guess you could assert that it's the instant instance. But of course, run... Yeah, I mean, you could assert that it's there. But maybe that's sort of working slightly against the principles of Python, which is duct typing. But yeah, I think if I wanted to be ultra-sure, then I'd just do assert is instance of this thing. That makes sense. Thank you, David, for your talk and answering all these questions. I think this was the last question we had from our audience.