 Hey, good afternoon everyone back from the break and I hope you all had good lunch Hi, Chris. So next talk is by Chris and the topic is building plugins using pluggy. So Chris Stacey's yours Thank you Before getting stabbed at is my screen visible And my voice audible Okay. Yeah Yeah, good evening. Welcome to my talk build plugins with pluggy. Who am I? I'm Chris Kumar from Bangalore I'm a software engineer and I like reading fiction and poetry How is this talk structured? The talk is structured into three parts in the first part. We will show you code which is Developing a normal style without plugins in second part. I will discuss about what is a plug-in system What is pluggy in third part? I will discuss the same code application which is developed using pluggy and we will discuss why is Writing code using pluggy or plug-in architecture is better suited for certain use cases Here is an example code for the course of the stock. I will use a simple example application Where the application? Collects the user input from the command line and makes a HTTP request to a service called Gutenberg and prints the output There it is a command line application to illustrate the Need for plug-in system and when you should use such kind of a concept. Let's get into the code Application class is responsible for workflow of the code The run method is orchestrates the entire workflow. The first is guess the use input from the user second the term Which user entered is processed using the processor second and sorry third is like you search the particular term in the In internet find whether the book or the author is available there depending on the Output choice of the user you format the output using the render output function as of now the render output function can output the Output in two particular formats one is JSON format other in a tabular format So in a normal architecture, how does the code look you have a core functionality and other functionality at built on top of it in our case the core functionality is getting the user input and Searching for the respective results is the core functionality other functionalities include like rendering the output exporting the results in a certain format What are some shortcomings of this particular way of writing code? Let's go back to the code once and then come back to the slide As you can see the code the entire render output function is something Which is hard to extend for example here. We only support only two formats tomorrow if the Programmer or the developer needs to support multiple formats a XML HTML and other formats It becomes hard for the maintainer to keep adding the code Next is it's hard to extend the program for all the user request that maintainability becomes harder The final disadvantage of the code is not extensible for an example There can be a new format like YAML format or it can be very specific format Which is related to the Microsoft or Windows or any kind of an operating system And then it requires the developers to know the ins and outs of the particular format to support it So without having a plug-in kind of an architecture, it is hard to maintain the code base and becomes non extensible Let's get into like what is plug-in system? So what is a plug-in? The plug-in is a software component which adds a specific feature to an existing computer program at a runtime and The concept of plug-in is not it has been in the software industry for very long time for an example All major web browsers approach plugins. It can be Google Chrome plugins or Firefox add-ons For example in Firefox if the output is in JSON format other Firefox automatically renders in nicely Formatted with indentation and everything but it's at the same time if the output or the response from the server is in a HTML format It knows how to render the Output as you can see you must have used any of the plugins already to say on hovering over an image It should enlarge it or you want to zoom in the images So plug-in concept is not new to the world, but it has its own advantages and disadvantages and if you see Linux Linux has been using Diagonal drivers which almost uses plug-in kind of a concept and while developing a custom software there are use cases where plug-in based architecture really Helps developers to maintain the code and extensive extend the functionality of the code Here is how a plug-in architecture looks as you can see there are two different colored components One in blue color other in orange color So the one in the blue color is the core system or a host program the host program is the driver of the program At particular runtime whenever there's a plug-in register to do some kind of a functionality the core system calls the plug-in to To render a functionality or to act on the particular data in a plug-in Architecture there are two important components one is core program or a core system or a host program And second is the plug-in or a hook So now let's come to the particular library which we will discuss throughout the talk the library is called plug-in Plug-in is a python library which provides a structured way to manage Undiscover plug-ins and enable hooks to change the behavior of the host program at runtime Plug-in is like any other python library. You can add it as a dependency and start writing your code with few modifications it helps you to develop your applications as a plug-in based architecture how pluggy works Before getting into how pluggy most let's look into visual representation. How a plug-in actually works As we have already seen the plug-in architecture has two important components one is a host program and second is the plug-in In this gray box represents the host program and all the other blocks represent the logical functionality of a program Rather than the exact flow the arrows represents the flow of the execution and with only limited understanding of the code So the program starts and first step in a plug-in based architecture is you call some kind of a plug-in manager So plug-in manager responsibility is to discover the plug-ins in this case. We have two plug-ins output and header So it discovers two plug-ins output and header And then it registers the plug-in. It does some kind of a bookkeeping And once at runtime the the host program decides to call the that particular registered plug-ins Those hooks gets called and the hook Gets the data now it's up to the hook to receive the data and it can act on the data and return the data Return data is optional not mandatory But receiving the data is mandatory and transforming the data depending on the conditions and the configuration it changes Once the hook has completed its functionality that The execution comes back to the host program and some sort of a cleanup happens and then the program makes it I'll take a sip of water As you can see this gray box represents a single process the biggest disadvantage or advantage of a plug-in based Architecture is everything runs in a single process space. Even the plugins are different code base They act in a single process namespace and everything works like a yet another library code So here's a sample code of a plug-in First thing is the host dot py is the host program So in this case the host program is responsible for getting user input and making http request and rendering the output The hook specification specifies the contract of the hook of plugin And the actual implementation of the hook lies in this program output dot py So this is a high level representation of the directory structure Next is the terminology. Now there are multiple terminologies associated with pluggy and plugin based architecture. Let's see one by one the first Important concept is a core system or a host program The where the program enters and the second is the implementation of a plugin for example in this case, we will be implementing a render or print output In for different formats each format will have one particular functionality to Implement that is if it's a json, it will print the output in json format. If it is in yaml, it will print the output in the yaml format So the plugin manager is the coordinator plugin manager is responsible for finding the plugins and registering the plugins and then giving access to the all the registered plugins to the host program Next is the hook specification. So hook specification is a blueprint or a contract where The plugin manager understand what is a hook name and what should be the signature of the function or a method in in pluggy all the specification is specified as a Python code that is for the kpython function or a method Now hook implementation is where the actual implementation of the hook happens Here is a high-level overview of the host.py file with all the code being removed and replaced by the comments the setup sets up the plugin manager and passes on the command line arguments which are passed during the execution of the program to the registered plugins And then this cli is responsible for getting the user input For developing this application. We are using a click Python library once the user enters a term the user input is validated then the search work clocking set and the search Class is responsible for taking care of that The run function is the one which coordinates the entire workflow First is to make the request then is to check the response and pass on the output to the The registered plugin. Let's get into some of the details Hook specification hook specification is nothing but a contract of the The first step in developing a hook specification is creating a hook specification market with a project name the project name is how the both hook specification and Hook implementation are tied together by the plugin manager. So first you create an instance of a hook spec And then you mark the python function or a method where as a hook spec is Decorated and here the contract is the Function arguments this function takes two arguments response and a conflict and as you're seen this body is empty For hook specification the body should be empty actual implementation goes into the hook implementation pattern And the name of the hook is going to be the name of this particular function The hook specification and the implementation should follow the same name and the same function name and the same function arguments Next is hook implementation The first step in creating a hook implementation is creating a implementation marker with the same project name and you use the The creator instance to mark the function as a implementation The plugin knows that print output that's a respective specification mentioned with the hook implementation marker It knows this specification has to be matched with this implementation The function signature is same and it can take extra arguments But it is not compulsory but minimum number of arguments is required And let's let's dive into the function body. What is the actual implementation? So this particular hook implementation outputs the response in a json format. Uh, so first it extracts the uh json data from the response and formats it and it's some sort of information from the response like name And what is the title and download count and then it goes about Getting the configuration details for the uh Output for example, there's if condition which says that the format is json then it prints the json in a colored format if the colored specification is passed And if for example if the format is something else a yaml Then what happens is this particular implementation will not do anything. It will not print any result. It will just move on So third is a plugin manager So plugin manager takes care of the responsibility of registering the plugin and all the plugin Creating an instance for the hooks to be called later So plugin manager, how do we create a plugin manager first to create instance of plugin manager with the same project name? And then we add the hook specification By now the plugin knows that for this particular project name. These are the hook specification and then you register a Implementation in our case output dot py file lies in the same repository, but sometimes what happens is like developer May create this hooks and third party developers may create a hook implementations So in that case what we do is we can you load using a setup with entry point That is any python package, which is being published as a setup dot py packaging method can Be used during the execution of the flow the important part is that setup tool sentry point should also be same name for both plugin manager and the registered Setup setup tool entry point both should share the same name once that is done. Then you have a plugin manager So final is the invocation of the hook So this is the same code what we saw earlier, but with actual code or no comments First user gets the input And then the plugin manager is an instantiator then the search route through and the run method which is responsible for Or coordinating the flow. So the important point to note is once we have a Plugin manager the plugin manager has an attribute called hook. So we create we pass the attribute pm.hook to the search Class its hook. So this particular attribute hook contains all the entries for the Hooks which are registered to the plugin manager As you can see the name of the hook which we are calling is print output Which is same as a specification and the implementation And when we call self.hook.print output the program flow goes into the plugin Develop plugin and then the code returns back. Let's see the example output of such a code Here is an output from a program with what we just saw The in first case the search term is my freedom and my bondage with our configuration indent as to in the second Execution we use same Title but with a different indent. So indent is 214 And the output is indented using the indent size as 214. You can see the difference in the printing And and the since by default colorize is equal to two The output is rendered in different colors Let's look into some of the internals of this plugin As you have already seen that it is quite possible to have more than one particular plugin. So for example, we only implement a json Printing let's say if we implement a yaml printing now what will happen? There will be a two implementation for a single specification So it is technically possible to have one specification and many number of implementation In that in that case what happens is like he calls each one of the registered plugins in last in first out order Let's say if you have three registered plugins one for xml one for json one for yaml It calls them in last out last in first out order And each of the registered Hook gets the same data And it gets passed on and it can act depending on the configuration. So it is up to the developer of the Plugins to figure out how to When to act when not to act so then for example if the configuration is passed in as a yaml It is up to the json plugin not to modify the data or not to change any of the data That is up to the developers As we have seen there can be more than one plugin for one specification So in that case sometimes it may be important to short-circuit the system. For example, if there are 20 registered plugins Uh, we may want to Short-circuit as soon as there is an output from the one of the respective or designated plugin Let's say we have a yaml plugin json plugin and uh xml plugin as soon as a yaml plugin completes We want to not call next to registered plugins in that case. What we can do is the configuration that is the hook specification while decorating the particular Function we can specify first result is equal to true in that case. What happens is when the first plugin returns the result to the host program the host program will not call the rest of the Uh plugins that is pluggy will short-circuit and it will Pass the collector result as a result and the program execution moves on So this helps in saving a lot of execution time and improves the performance And as I as you already know, uh, the result is a list of results from each plugin. It is not a single return value It is a list of return values So testing the plugin How do you test the plugin? the first thing is You create a unit test So first thing to remember is like uh, testing plugins in pi player for pluggy is simple For example, here is a print output. So we are passing some output to the Output from the request to the print output Function, uh, and we are testing it like any other function Of course, this you may have some kind of a version that is not actually unit test because we are making HTTP record That's correct. We can use a dummy response or a fake response That should not matter but testing any plugin function or plugin method is same as testing any other or python code So next is the uh integration tests How do we do an integration test as you remember from the earliest size the the applications are command line application So to have a integration test, you need to have the host program and the plugin come together and first thing in this Is to setup and we are using uh click as a command line application So click gives a nice cli runner to run integration test So we create on a runner instance and then we invoke the respective registered function and then we pass the arguments The important thing to note is the colorize is made false because checking the colors in the render terminal is a bit difficult But we are only going to test the indent size it is true That's why the expected output is entered and accordingly And you get the result and you strip the output and check both the return result and the expected outputs are same again in doing integration test is simple and how we do for any other command line application Some of the examples of real world projects The pluggy was created as a plugin system for pytest library so pytest is a python test runner which helps in Writing python test easier for example if you're used coverage plugin coverage plugin helps you to Find out what code has been tested what code has not tested So which is something most people tend to use And pytest coverage is implemented using pluggy. You can go see the source code and it is And the and pytest ecosystem has more than hundreds of plugins And the next is a tox so tox is another system which uses pluggy So tox's intention is standardize a python testing with multiple python versions and And make the testing easier and the final project is data set so data set helps you to publish your data set and make the users to Explore the data set for example if your data set in csv format, but you wanted to publish that The data set in a sequel format and then you have some interactive system where users can explore That is exactly where data set comes into play and helps you to do that and some useful links that The code which I have shared for the entire talk is available in my github repository You can go to craze kumar slash pluggy talk and access the code And in case you have any questions regarding the talk, please feel feel feel me to reach out during the talk or after the talk In twitter if you are on twitter And finally stay safe and wear mask. Thank you