 Right, so I think Ali is ready. Hello Ali. Hey, how's it going? Yeah, do I pronounce your name correctly? Like how do I pronounce your name? It's it's Ali. Ali. Yeah, that's that's great. That's amazing. So where are you connecting from? I'm calling in from Chicago. Oh, wow That's great. That's amazing. Okay, so yeah, I can see that your slides is there and take us away Awesome. Well, I want to start by thanking the organizers for giving me this opportunity to share my knowledge My name is Ali Sivje. You can find me on Twitter at kiasivjes I've tweeted a links to these slides. So if you want to find them, they will be available on my Twitter There'll also be links at the end. I'm one of the organizers of the Chicago Python users group We call ourselves Chippy. Chippy is one of the largest Python communities in the world We have around 6,000 members and every month when we were meeting in person. We held four to six events Normally at this time I'd invite you all to come out to Chicago and attend a Chippy meet-up But we've had to postpone all of our events until it's safe to meet in large groups But we have sort of live streaming our events to our YouTube channel Now there's no reason to visit Chicago. Although I still suggest you come when things clear up Subscribe to our channel click that bell icon and you'll get notified of upcoming events Hope to see you all on the live stream and we actually have Diana appearing on August 4th So if you want to hear her give a talk about Flask come on out. It's gonna be a good time Let's just get started Plugins are software components that extend or enhance existing an existing program Plugins are also known as extensions or add-ons Technically these definitions are a bit different for each of those words But for the purposes of this presentation, we can treat them interchangeably If we start looking around we can find plug-in systems everywhere we look Why browsers are the most used piece of software? Each browser has an ecosystem of third-party developers making plugins for things like adblock password management them keybindings and JavaScript has a bunch of developer tools as well IDEs and text editors also have plug-in systems in my VS code I have plugins to make it easier to debug JavaScript to improve my user experiences with ligatures to interact with third-party Libraries and tools like Docker like GitHub and it also adds support for query language With media players we can use plugins to add functionality from new file types and also change user interfaces They call those skins. Also. Does anybody remember all those fantastic visualization plugins from Winamp from back in the day? WordPress is a popular content management software It's written in PHP and it's estimated that WordPress powers 35% of the world's websites There are thousands of plugins written for WordPress that allow you to customize your website to do pretty much everything WooCommerce, that's a word plus a WordPress plugin that lets you add e-commerce to your website WooCommerce has its own plug-in system. So you can write custom plugins for a plug-in Your favorite web framework that probably has a way to use third-party libraries as well. Later on We're going to talk about adding plugins to Django and to Flask There are many benefits in implementing a plugin system The biggest one is that plugins they make it easier for us to develop new features Plugins also allow us to split up our program functionality into logical chunks This makes our program smaller and easier to understand for instance when we want to make a plugin We really don't have to understand how the host application works and Third-party developers they can leverage plug-in architecture to design solutions that fit their needs that fit their clients needs Could you imagine using a web browser without any extensions? We also have to discuss trade-offs for plugins I don't like to think of these as disadvantages. They're more along the lines of things we need to consider before we make our decision Like everything else in programming, especially in object-oriented programming We have to do a little bit of upfront design in order to implement plug-in architecture We have to figure out how our plug-in is going to interact with our core application logic Plugins also mean that our core application logic will require additional complexity to handle these plugins We'd have to decide if this additional complexity is actually worth it Before we start diving in any further, I just want to give over overview of my goals for this talk By the end of this session I hope to provide you enough knowledge that you can go out and write a plugin for any third party or a third-party plugin for Any application you want This doesn't have to be a Python plugin the concepts. We're going to talk about they can be applied pretty generally a Brief outline to what's to come we're going to start by deconstructing plugin systems into their base components Then we're going to use that knowledge to build a small application with a naive plug-in system I like doing things from first principles. So the best way to learn how to build plugins is to build a plug-in system Finally, we'll explore how we can write plugins for flask and jango While we do build a naive implementation of a plug-in system We don't have time to discuss all the things that make a plug-in system be something you can like push out to production But if you are looking for resources, I've provided links both of these talks are fantastic And I really want to thank Doug and Rose for sharing this knowledge with the community I use this material to create this talk. So let's start by exploring what makes a plug-in system a plug-in system When we have a plug-in by itself, it doesn't do us any good It needs to be part of something larger That is it needs to plug into something and we call that something the host application Plugins and hosts need a way to talk to each other The communication channel could be calling a function getting a value back Or we could be using a communication layer like WebSockets There also needs to be a way for plugins to register with the host application This could be done a bunch of different ways Some common ways are loading information or loading plugins into a folder And then when the program starts up, it initializes all those plugins inside that folder And if this sounds familiar, it's sort of the way how Python works When we pip install a package, all of that information goes into our site packages And then we're able to import it into our process Other programs that require us to specify plugins explicitly In Django, we can add middleware by explicitly listing them in the middlewares list in settings.py So when we first start the program, it has no idea what plugins it's going to load Once we've registered plugins, those plugins are going to be loaded at runtime dynamically And once the program is registered and loaded, it needs to be able to respond when it's called upon by the host application And if the plugin was registered for specific actions, we're only going to call that plugin if those specific actions were triggered Now that we know what goes into a plugin system, let's make one of our own This is the checklist we just talked about We're going to come back to this at the end of this section, but I just want you to keep it in the back of your mind as we go through this exercise So let's talk about this example project we're going to build One of the best parts of Python, in my opinion, is the Python package index or PyPI Chances are if you need to do something, there's a library or a tool that gets you most of the way there But what if you find multiple packages that seemingly do the same thing? How do you know which one to use? Some of the things I look for when I'm making this decision include looking at the number of open issues, the number of open pull requests, the number of stars a project has, when the last activity was All of these things, they help me understand how actively maintained this project is We can gather this information by going to GitHub, going to GitLab, or we can use an API So we're going to build a command line tool that takes in a URL and it's going to print out some repo statistics The majority of projects we're going to use are on GitHub, that's just what I've been finding, but there are some like Flake8 which are on GitLab We might have to support some additional providers later on, but since we're building with a plugin system, we don't have to worry about it at this time So we're also going to need a way to identify what provider belongs to a certain URL And we're going to need a way to go out to the API and get some statistics And as I mentioned, most importantly, we're building with a plugin system So we're going to start by defining our data objects that are inside of our program So we have a repo details type, and this is a name tool pool that's going to take in organization and repo name We're also going to have a repo statistics table, sorry, a name tuple, and this defines all the information we want to retrieve Next let's walk through how we're going to go out and use the GitHub API to get this information We're going to go out, make sure that the URL is actually for GitHub Next we want to use the request library to hit the API We're going to get the response back and then parse our JSON payload and then we'll return that information that we generated When we produce output, this is what it looks like after we format it a little bit Now that we have all those steps to solve our problem, let's generalize it a bit more So I'm going to create a class, I'm going to call it base plugin, I'm going to initialize it with a repo We're going to have a check method that takes in a domain and returns true or false And this function, it's going to help us identify what providers belong to what domains And finally we're going to have a method that abstracts going out to the API, getting some information back and then doing some parsing and sending it back to the caller Let's walk through how we would take that GitHub script we wrote and make a plugin out of it So we're going to start by creating a GitHub plugin class with base plugin as its parents We're going to override our check functionality and here we're going to return true if the domain is github.com And we're also override our repo statistics and like before we're just going to go out to the API, get a response, parse it and send some information back Next let's examine how we're going to write a plugin for GitLab We're going to create a GitLab plugin with base plugin as its parent, this is just like before Also just like before we're going to override our check method to return true if the domain is github.com And just like before we're going to go out to the API, get some information back, this time we're going to be parsing the GitLab response structure so it's a little bit different So now let's look at how our host application can be used to run our plugins We're going to start by adding these two plugins that we just created to a list, call it plugins Next we're going to create a class, we're going to initialize it with a URL which we're going to parse to get things like the domain out We're going to have a private function that's going to grab the domain as well as all this other information We're going to look through each of the plugins we just initialized If that plugin matches the or if that plugins domain matches the domain that's passed in, we're going to assign it to a variable If that plugin doesn't exist, we're just going to raise an exception And then our host application is going to go out and use that plugin we saved in that instance variable and use that to get some statistics for that particular repo Now when we go out and run this code, it has that same output before after a little bit of formatting So we've designed an application that has a plugin system and this is fantastic If we ever wanted to add support for another provider, say Bitbucket, we could do this pretty easily Here's how we go about implementing a Bitbucket plugin, there's a code right there And then in our host application, I'm just going to update that plugins list by adding that one particular plugin type So going back to that checklist, does our system have a host application? Looking at the diagram? Yeah, I think it does So that's a check Next item is, is there a communication channel between our host and our plugin The plugins and hosts communicate by sending messages by calling public functions and returning a value We're using object-oriented Python So yeah, that's also a check mark Does our plugin register with the app? From our host application, we register plugins at that top, so yeah, that's also a check Is our plugin loaded dynamically at runtime? Until our code reads that first line, it has no idea what plugins are going to be loaded So yeah, I think we can also knock this one off the list And for our last bullet, does this plugin respond when it's called by the host application? The plugin functions does return a value when they're called, so yeah, I think this is a check as well So it looks like everything's good to go, and we can give it a stamp of approval So now that we've implemented our own naive plugin system, let's explore how plugins are implemented into Python libraries It's going to help us understand how libraries in the ecosystem do things Django, that's a popular Python web framework, Django is very opinionated and the developers, they specified many places in the Django codebase Where we can go out and customize functionality A quick scan of the documentation sort of revealed all these ways that we can go out and customize Django And this doesn't even include third party applications that we can pip install and add to our installed apps list in settings.py So we're going to take a quick look at how we can go out and extend Django functionality by writing custom middleware And in Django, middleware is something we can do or something we can write that allows us to hook into Django's request response processing framework And so let's say this is our middleware list in our settings.py Before the request gets down to the view from the client, it gets passed through all of those layers of middleware And then we do what we need to in the view and then on the way out, it goes back through those layers of middleware again So what does this middleware look like in code? I pulled the sample from the Django docs So we're going to start by creating a class, notice this doesn't have a base class, we're going to be using duck typing here The middleware, it's initialized with an argument. It's called get responses. We can just save that to an instance variable just boilerplate code we're doing In our dunder call method, we're going to pass in a request And in this section, we can write code to modify a request before it moves down the chain into further middleware or into the view And then we're just going to call this function to specify that this particular middleware is done with the current request And in this following section, we can do things with a response on the way back out to the client And then finally we'll just return that response to signify that we are done and it can move higher up that list in layers or back out to the client So now that we know what to do, let's write up our own middleware that's going to add a unique request ID to every single request So we're going to create a class, call it request uuid middleware, add a doc string We're going to initialize this class sort of boilerplate code just like before In our dunder call method, we're going to just take it, we're going to like see if the request already has a header set If it does, we're going to use that as a request ID. If not, we'll just generate one from scratch And then we'll signify that we want to add something to our request and that our middleware is done and it can go further down the chain And since we're not going to be doing anything with the response on the way out, we can just return And now we have to update our settings and we're good to go We can just add some tests to confirm things work. It's pretty straightforward We can't really talk about Django without talking about how Flask also does things Flask is a Python web framework. It builds itself as a micro framework I don't know how micro I would consider Flask, but it's definitely a lot more lightweight than other options I've seen out there And since Flask is less opinionated, there aren't as many hooks available as what we had in Django And this is not to say that Flask is not customizable The design philosophy behind Flask is that we want to figure out what third-party libraries that we need to use And we'll have to wire them into our application ourselves I would say Flask would be more plug-and-play than Django But there are definitely, like it doesn't make choices for you, but there are definitely best practices and recommended libraries like Flask SQL Alchemy Sort of like a gold standard for ORM Well, let's take a look at how we can extend our Flask functionality We're going to write that same middleware to add a unique ID to each incoming request So in order to modify requests and responses inside of Flask, we're going to have to play around with the whiskey middleware And I believe this is a web service gateway interface. It's just the level below Flask that processes all that information So we're just going to, like before, create a class. It's also going to be using duck typing. We're not going to have a parent, a base class This class is going to be initialized with an app parameter. This is just boilerplate We're going to have a dunder call method that's going to take in some parameters And we're going to create a request using our environment using Flask's request helper or the library below Flask Works the way I believe it's called And if the request has a header set, we'll use that value, else we'll generate one from our own, sort of similar to before And then we're just going to update our environment with the request ID and then return the app with some parameters to pass it on to the next layer If we want to enable this middleware, we're going to have to update our whiskey app attribute of the Flask application instance And now we can add some tests and things are going to work just like they did before Before we do close out, I want to provide some advice on how you can go about writing your next plugin And I thought it'd be useful to talk about my thought process for when I'm going to go about writing a plugin for a host application I've not really dealt with too much before The first question is, does this app have a plugin system? We can't really extend something that doesn't have a plugin system To find out if our application supports plugin, the best way to do it is to go check out the documentation I would search for keywords like extend, customize, add-on The docs are probably going to have a section that defines where you can go out and find an example plugin But if you're looking for more example plugins, I recommend going out to maybe PyPI, Google search And an underrated search is GitHub search. I feel like you find a lot of code that you can pretty much readily use by changing a line here or there When I'm writing code, I like to have a minimal project setup. I call this a sandbox development environment And this is going to have my host application with a single plugin And this is going to enable me to quickly iterate on my solution And it's also going to make debugging easier since I have a very minimal interface And once I'm ready to go out and build my plugin, the code I've already built out in the sandbox, I can use it to copy over to my logic or maybe use it to create some tests Also, when you're playing with your sandbox development environment, I recommend adding breakpoints to code that you do not own In order to find out what messages your plugin needs to respond to, it's really helpful to get down in the application instance and mess around with things like PDB Sometimes you won't be able to get into the host application, so you're going to have to depend on logging And this could be done really depends on how this application has been written You can take advantage of things like pythons built in logging, or the host application might have a framework that you can use to log messages If you're deploying this out as say like a plugin on a computer, it's good to collect logs that users can send back to you so you can help diagnose if they have issues And now once we have our plugin already complete, it's time to start thinking about how this plugin fits into the application ecosystem So if you want to release a plugin for other people to use, I recommend looking at other plugins as a template to get a feel of how things should look And it also helps you get a sense of if you need to initialize your plugins with a set of variables or if you need to create exceptions or override exceptions that users already know about in that application There's a lot of material we can talk about when we're talking about testing, but the main idea I really want to get across is integration tests are the tests that we really care about We want to check that our host application and our plugin, they work well together and the best way to do that is to check the integration boundary And if we have a plugin that supports more than one version of software, we're going to have to run our plugin against each of the versions we care about If our plugin is connecting multiple different software, we're going to have to run against each of those component pairs And if we're doing something that's cross-platform, I recommend having tests run on Mac, on Linux and on Windows To recap, plugins are software components that extend or enhance an existing program While plugin systems can be designed in many different ways, they share a lot of common characteristics Taking the things we learn by writing our own plugin system and the things we learn by extending a plugin system for a well-known library We can see that it's not too difficult to extend the functionality of your favorite library without modifying existing code So the next time you find yourself working on something complex, stop, take a step back and breathe Break the problem you see in front of you into smaller logical chunks? You got this, it's not really that hard These are all the resources I found coming up with this talk Thank you so much for your time. Before I go, I do want to give a shout out to the Chicago Python users group I would not be where I am without that fantastic organization And I'm also going to be speaking at a few conferences in the coming weeks, so I hope to see you all there Thanks so much for your time. Again, my name is Ali Sevje and hope to see you at EuroPython in person next year Thank you so much. It's amazing. It's a very good talk and I love your energy. That's very, very good And we have actually, yeah, thank you. We have one question from Jill that Have you tried using any plugin library? Like, yep, see, like that's Y-A-P-S-Y. I don't know about that, so Or would you prefer to write your own? Yeah, so that's a great question. So the question about if I used a particular library, so there are libraries in the Python ecosystem That abstract a lot of the common boilerplate like if you're building out some things like you need to like set up like an entry points Like thing in your setup.py. So you can use tools like I'm not too familiar with this yet another plugin generator But I have played around with stevedor and the links I provided earlier, they were about stevedor I thought it was cool. And if I was building something, I might leverage something along those lines But at the same time I like controlling a lot of what I'm doing And the things I am really concerned about are not that complex, so I don't mind writing an interface to do that for me Right, so, yeah, I think that's it And if anybody want to, you know, continue this discussion or have more questions to ask afterwards, then I think Ali will be answering the questions in the channels specific for this talk And I will also post a message in the need track that people can find the channel So thank you so much. That's really great to have you this year at Europe Hibern and hope to see you next year as well So, um, thanks so much. Thank you