 Okay, well, it's we're at time. We'll get started. Hi, my name is Adam Miller. I am on the Ansible core engineering team at Red Hat I before that have been on the foot or engineering team, and I've been a foot or community member Since roughly been a contributing for our community members since roughly 2008 Been a long time user been doing stuff here and there But I want to talk today about advanced ansible extending ansible through plugins We're going to kind of talk about a quick outline why this matters to Fedora why this is you know being presented here Some ansible internals plug-in types and examples how to write a plug-in and how to contribute a plug-in if you find something That is very useful. You think they might be useful for other users out in the ansible community So why advanced ansible flock well quick is history lesson. So ansible and for the shared history So for starters ansible is created by Michael Dahon who has history in Fedora and then actually as well a red hat emerging technologies Adrian Likens Greg de Konisburg and Seth Vidal Greg Deck being a Fedora project leader and Seth Vidal Being the original author of young long-time Fedora can't remember before he tragically passed And then so funk also does anybody remember funk from thorium structure team like Okay, two three So funk was a tool that was kind of created once upon a time to do this concept of orchestration within the environment ad hoc tasks those kind of things But it was based on a message bus and these various different architectural decisions and Ansible's born kind of from some of those concepts mixed with other concepts learned and things But as far as a history of technology and as far as the concepts that came into the design and implementation of ansible Many of those things actually stemmed from experience gained in the Fedora community as well as directly from for a community members input The other thing is the first floor infrastructure team actually started using ansible since it was created because Seth time was on the floor infrastructure team and he was part of the influence and the creation of the Project they initially started off very quickly using it for ad hoc tasks like that over time it eventually repaid a replaced puppet for Full end-end configuration management automation and things in the Fedora infrastructure for our CI and QA so things that Seth Walters and those folks work on Tim flink and those those groups are working with ansible as well Also our friends over not directly related Fedora, but our friends over an open stack If anybody's familiar with Zool the CI system Zool is completely implemented using ansible as well as all their tests being written in ansible Does anybody remember Jenkins job builder? Yeah, so Jenkins job builder the open stack team Actually implemented a translation layer that turned Jenkins job templates into ansible And that's when they moved Zool to Zool v3 and it's all ansible powered So just an example another example of CI efforts for releasing during does some workflow Automation that was some work that I was involved in a while back where we started actually doing a workflow automation Because there was these you know decade old shell scripts that have been sitting around that we're just kind of have to limp Along and it just kind of made sense to try and move that stuff along to something that was a little bit more and more Maintainable a little more general use Can be more easily contributed to other aspects of the Fedora project Last I checked we're kind of evaluating where it would work I don't know who's actually using it widespread if everybody saw Matt Miller's talk this morning with the the web graph of you know the 37 odd different sub projects in Fedora. I didn't survey everybody. Sorry And the advanced use of ansible through plugins and things doing Using ansible plugins to extend the functionality to better integrate with different systems You might be using so for example The QA team they use fabricator, you know, there's potentially some integration points We could actually interact directly with fabricator based on actions during runtime of a playbook based on Some input or some, you know side-effect of of an automation component executing we could you know update their tickets We could update different aspects of their their planning whatever Similarly with you know github or if you're you know doing IRC bots those kind of things we can actually interact with those as well We have some plugins built in today. It'll do those sorts of things But if you know there's a component that you need or that's gonna be custom tailored to a specific workflow somewhere in the Fedora project Plugins and those kind of things will be useful there. So We're gonna go through into all internals one-on-one just kind of given a overview of how some of the stuff comes together How it might be useful to you in things you're doing in Fedora So a quick refresher or a quick refresher so I can talk I promise Ansible is an agentless item potent task automation tool Task functionality is provided by modules and modules are technically a type of plug-in, but they're special We'll talk about that a little bit tasks are grouped together via plays a play operates on a set of hosts hosts are indexed by an inventory Playbooks can contain one or more plays and a role is a bundled reusable set of files templates variables and tasks is Any of this news to anybody everybody hopefully familiar with Ansible using it? Yes, cool. Cool. Not a bunch of head nods wonderful Went over jump. Okay. Now How answer works under the hood when you run Ansible or ansible playbook more specifically It's gonna parse and load an inventory. It will parse and load The playbook or playbooks in the event of includes and things or a Role for each play in the playbooks It will then for each task in the play and then it will for each host Filtered on your your host group filters things like that It will then run the task on the host and read the results during each of these steps one or more plug-in types Are used so the plug-ins are actually taking action During the execution on each host depending on what's in your playbook So conditionally based on what's actually happening during the automation is gonna pen which plug-ins or play some of them are always in Play and we'll kind of we'll kind of walk through some of that stuff So the plug-in system there's a top top top class loader called the plug-in loader so this is responsible for Keep kicking this thing and it's probably not great because it's kind of jiggling display I'm sorry It's responsible for loading all of the other types for of plug-ins and also Like I noted before modules are technically plug-ins. They're just kind of special in a lot of ways It does actually load modules, but it only finds them. It doesn't actually load them because we don't load Plug-ins on or we don't load modules on the controller Since the amplitude up for plug-ins can declare their own requirements So if you're a plug-in author, you can actually declare your requirements And those can be exposed to be configurable to an end-end system administrator and as of 2.5 on the control host you can actually Explicitly enable or disable specific Plug-ins on your control host as a system administrator if you would like to So modules shipped and ansible are in lib ansible plugins by default That's going to be universal if you installed from an rpm or pip or you extract a tar ball and you're running from source The main plug-in class lives in the init inside of that directory And then you can do custom defined plug-ins through The type of plug-in underscore plug-ins so we have these types of plug-ins and these this is just an LS dash one of Live ansible plug-ins so action cache callback CLI comp connection filter HP API inventory Lookup net comp shell strategy terminal test and vars any of those and then underscore plug-ins You can have a directory name that and then ansible will search for Plug-ins in that directory of that type or you can actually do custom Definitions in your configuration file if you want something to note is some of these aren't actually going to be general use They're going to be specific for ansible networking. So for those who are not familiar ansible has networking extensions to configure network gear physical network gear juna persisco Palo Alto, I'm not a networking person. I'm leaving out a whole whole bunch of them But so some of those aren't going to be directly useful for necessarily people in the flora Ecosystem who are Probably generally going to be doing things with systems and doing system administration and automation tasks things like that So just kind of take note some of these are going to not directly be applicable and as such we're not going to go through them today also in the interest of time so Special cases modules are loaded. Okay. I mentioned this earlier. I forgot which slide it was on Modules are loaded through the plug-in loader, but we don't actually execute them or into the into the Python Runtime at the time documentation fragments. So There are special cases in which Files are basically documentation fragments that can then get merged with other sets of documentation Instead of with their direct code place and we'll talk about this a little bit later But so doc strings used in common code module common those kind of things those are there I forgot to mention the beginning of the slide. I just realized I didn't James Kamarata that I stole most of the slide back from him I had a little attribution down here because he wrote it how those who aren't familiar it goes by Jimmy see on GitHub or the Jimmy see on Twitter. He is Bdfl since my clap Anyway, so the first time plugging we're talking about an action plug-in and the action plug-in is very interesting from the aspect of If you are working with a module or you written a module Actually really quick notes show a hands who's written a module Okay, four or five So an example of a module be the DNF module so you have a task DNF name Bash state installed An action plug-in is effectively a Module that runs controller side you you can Invoke an action plug-in just like you and invoke a module But it takes action on the local system and you can actually overload that and have both an action Plug-in and a module Name the same thing and you can have them interact together and an example of that would be the Oh my gosh, I'm gonna blink on this Sync pretty sure it's the synchronized Yeah, so synchronized so for example it has to actually do some things on the local host that you that you're doing controlling from so It needs to you know find some files probably find our sink do some things and then it needs to contact the remote system so You would do an action plug-in If for some reason you need to on your local host perform a task like tariff a local directory before Sending it sending it across the wire to the remote system And if you want to do all of that without having to require your user or your team To create a set of tasks to do those individual things one by one and then do that You just kind of want to have it handled in the roundtrip single task definition your playbook an action plug-in is there There to enable that so if an action plug-in by the same name the action plug-in is used and then Otherwise, you know the normal remote execution of the module happen. So an example is Which one is this Okay, so basically is you everything starts with the base so there's a base class for each plug-in type so for action plug-in there's action base for You know shell plug-in there's shell base those kind of things and you so you subclass it and then everything's gonna have a definition of run that's gonna be kind of our invocation point and Something to note is that's kind of special to the action module as you have an execute module so for some reason you are actually defining an Ansible action module or I'm sorry an actual an action plug-in and you need to invoke the module the same name I think a really good example. This is the package We're gonna really quickly We're gonna do this just lost internet and I can't connect my demo box anymore. That's cool. Okay. Well, anyways, we'll just walk through this Instead So taskbars is going to be The dictionary that we get by by doing the super module or I'm sorry the super invocation of our parent class And that's how we're going to get actually the variables that are passed. So again, if we were using The example of DNF as our task of task name equals bash State equals install those will be sent into that dictionary. You get those as the arguments that get sent in you can actually Use those for making decisions in your action plug-in programmatically through Python and then if something that you need to do actually will Execute a module you can then pass those into That module because executing the module Needs those arguments passed into it. So this would be exact for example the implementation of the task And how that gets executed through the answer internals And then you you would Need to actually report those results in the same way that you report your module results Which is the return results and that's going to be the JSON structure the answer where it expects to return because the module itself Executes and returns JSON. So for an action plug-in because an action plug-in is effectively Implemented in a way that's meant to be used as a module locally on the on the control host we have to handle that Yes very quickly, why are you removing the answer will notify from the results? that is a Historic thing and we have documentation in the Devel docs that just says you need to do this and I don't remember why I Think the reason to do that predates me as as somebody who mocks around the code But I'm not actually sure there's there's a Reason somewhere Do you remember? Yeah? I don't remember Yeah, in the in the develop ox it says you just you need to do this because I think it was part of the answer 1.9 to 2.0 Transition like somewhere in there, which like we don't need this anymore. So always delete it. So we're not because every time just The question sorry to repeat for the sake of the recording question was why do we have to delete results invocation module arcs? And the reason is I don't I don't remember but I know that we need to do this And I believe it was part of the answer 1.9 to 2.0 transition because every time the results gets copied around There are various points in the answer on time that do a deep copy So that entire dictionary structure would be continuously copied over and over again And it won't add up to a whole lot of data unless you get into the world of like a large inventory or something where you're doing a lot of dynamic Revaluation of variables because then we just keep deep hot deep copying it recursively and then you can run the memory problems So I believe it's just kind of those like get it out here. You don't need it Okay, so callback plugins. I think callback plugins are probably the most common for system administrators And they're used to act on events which occurred during the execution of playbooks. So you can for example at the start of a playbook run or at the success of a task or Anytime there's output you can take action So this is this allows us to do like custom logging events This allows us to do chat ops because when something starts you could have a kickoff some message to Your IRC bot or your slack bot or whatever you're using So you have multiple callbacks and and they're all configured in the Ansible config file you can set multiple of them to run and Yeah, the the callbacks that do output to this to the screen to your local terminal You can't overload those Looks like it's weird. The one out nine X note I didn't I thought about deleting this but I thought just on the off chance. Somebody's hopefully not but maybe still Doing a one one nine to two O transition There was a configuration Component added So to know we decided to just add the different events I did leave this in there because if you're copying around all back plugins that have been around in the ether for a long time You're potentially going to see an on playbook star versus a v2 on playbook star. That's all compatibility stuff for the most part Anything you're writing today when you write a callback plug and should be doing the v2 underscore But the the old the old implementations still works if you do final callback plug and we maintain compatibility moving forward for those so So you have your callback module, which is again going to do the callback base And then your callback version should be to oh in your callback type. There's different kind of types Callback name. So the callback name is going to be how you identify in the configuration file and then your plugin Needs a whitelist. So this is going to be about whether or not enabled by default or if you have to add it explicitly to the configuration and I really wish right, so Our callback module here Has the definition of version the name looks anything. So this is the profile tasks So it's going to kind of go through and do some Some statistics, but we have a handful of definitions here. So playbook on on On task start on Handler task start so again you so you have your tasks in a playbook and you have your handlers And if we're trying to profile all of those things we want to make sure to take action based on based on that So inside of here we have a couple of internal methods that will record the task So that we can display the the different metrics and then we're going to display Information on setup and then on stats. So you have different action points or different basically schedulable moments throughout a playbook execution where you can actually run some action and So for the full like run through you have So here we go. Here's the v2. So you have runner on Async poll you have on async. Okay, they failed playbook start Start notify host match host remaining Task start clean up handler vars prompt So there's a lot of defined points throughout the Execution where you can actually have an inflection point to take action based on something that's occurring and you have a considerable amount of Access to to the information about the context in which the execution is going so the goal here is to allow You to get introspection into your your execution of a playbook through through the callbacks Yeah, this this got chopped off. There's supposed to be another slide with like some of the Different examples of Function calls. Okay, so a connection plugins so connection plugins are used to provide the transport layer between ansible and Whatever you're connecting to So the relatively simple you have a connect execute command put file get file disconnect And the goal there is to make it easy to write easy implement easy extend ansible to do things other than just an ssh connection the most often used connection types for Sys admins are gonna be local and ssh, but there's there's quite a few of them So we actually have a we have a charu transport So you can actually also define pipelining so for those who are not familiar the pipelining function I allows you to pipeline some of the commands together and for ssh that means you're able to use multiplexing and those kind of things so This again just drops off there. Yeah coupes coupes cuddle. Why not? So we actually have a Kubernetes with a coupes cuddle Connection plugin for anybody talking to Kubernetes. You can actually talk Directly to Kubernetes over this connection plug-in So it's got its init function Then we go through and then we define Exact command so that's how to exact the execute command over the the connection and We have our put file we have our fetch file and are closed So the idea is everything else all the other code in there is simply at service of the defining these required Methods and then at the point in time that the execution runs. It simply calls the appropriate plug-in and so you can kind of dream of just based on Everything that we have right now You can kind of dream up whatever you want or whatever might be necessary or useful So we've got a build a charu docker phone D hbfi Those kinds of things so network CLI so again like that's going to be in service of you know network routers and things Switches, so that might not actually be very useful to you But we have a considerable Number of different connection plugins and if you have some type of system You're trying to connect directly to without trying to do like an ssh jump through and run weird commands or something You can actually do those to those kinds of things natively with connection plug-ins So strategy for again plug-ins. These are probably the hardest to write. They're the most laborious to do But by default when you run ansible the way that actually executes is in order task execution And what we mean by in order is every hosts in that matches the host group that you're running against for your play Each task will wait to remove to the next task for every host in the inventory Matched group to finish. So if you have that DNF You know name state state installed and you have a hundred hosts that task will not move on to the next Next task in the play until all hundred hosts have checked back in That's gonna be linear then there's free free is Free basically says every task in the play Every host just go out and do it as fast as you can and then at the end of the play book Or at the end of the play will wait for you all to check in we're not worried about each task individually And then there's a debug which just subclasses linear and gives you all all kinds of debug functionality if it hits a problem It'll actually drop to a debug console But you can kind of do whatever you want you can you can get really really creative But do note that from a development standpoint. It's gonna it's gonna get pretty It's gonna pretty off the weeds and you kind of are left with a little bit of If you break it you buy it from from an execution standpoint the eye I Would be amazed that many people need to extend or do something different here But the option is there and so this was new and 2.0 And if you really want to get creative and if you just have a wild idea for the actual execution strategy of How your playbooks run you have that that flexibility if you want it There we go. So look at plugins. I think look at plugins are also very common aside from callback plugins so for example Just really quick. We'll just go so look at plugins are used in two ways traditionally So the template for templating variables and then also for loops on tasks Jimmy's not a fan of of the loops on task But if you use the with items it actually loads the the item.py uses the lookup plugin to find variables and that kind of thing But this is gonna be a very common pattern Who who is familiar that you could actually template variables in line in your in your play in your playbooks to do Lookups of various things Okay, about half. All right, cool. So Just since we're why not We'll just kind of for kicks pull an audible if DNS maybe maybe Nope just kidding I was gonna show you some docs and some examples, but I'm gonna lie to you instead Okay, so the main thing with lookup plugins is it's always gonna run on the controller This does not ever run remotely And that's important From the perspective of if you think for some reason you're looking up Information contextually about the host you're remotely executing a task on don't think that because you will not get expected results Yeah magic, okay wonderful Yeah, all right, so We have Plug-in lookup plugins for all sorts of things so you can get a database account attributes You can do Cartesian products of lists chef things So you can really read environment variables You can get the file the contents of files So for some reason you want to read a file Contents into a variable at runtime without having to store that information in playbooks I'll look at the lookup plugin for that is very useful Another one that I think is is very useful that people aren't very familiar with is pipe Reading the output from a command so the example of that Is you can literally just say look up pipe and then run the command like date or Add things there So that's basically the ability to just run some arbitrary command It's store its output as as your as a variable at runtime in line for it's a templated variable so Not not directly plug-in writing a related but just There wasn't nearly as many people who knew about lookup plugins. I thought so I want to at least mention those Okay, so look up plug-in example So again, we have our run. I think this is actually Yeah, this is the pipe. So this is the pipe lookup plug-in So basically you it just it just runs a sub process It goes ahead and runs the command that you asked for it gets the information and then it returns It returns that and then when the plug-in loader runs it at the templated variable It just inputs it in line where that where that templated variable value would be So again, so we have a lookup module It subclasses lookup base and then we have to provide a run method and The code for this is relatively, you know, simple So you pass in some terms for term and terms you want to make sure it's you know stream validated run it as a sub process Do a communication to the sub process? Make sure that return code is zero so that we're not Yeah, it's fine make sure return code is zero so we're not returning failed things or else, you know go ahead and Offer a meaningful error message Otherwise we would get the error messages the templated variable value and that's probably not what users want So filter and test plug-ins so filter and tests extend the JIN to two templating system For ansible so filters are used to transform data. So everybody familiar with the pipe syntax of doing filters and JINja and then tests are used to validate data Filter and tests so An example so again Oh that filter modules Examples outdated Yeah, I'm pretty sure that's wrong. No, okay. It should not come back if it comes back. We'll get to that minute so cash plugins so for fact caching cash plugins are used to store gathered fact data outside of local memory and this can be useful for The systems that aren't going to change very frequently if you don't want to actually have to regather that fact information You don't actually have to do anything beyond configuring a fact cache Plug-in, you don't have to go through and modify your playbooks or anything Ansible itself like should go ahead and handle this but if there is a a Fact caching or if there's a data system that you want to store your fact cache in that Ansible doesn't currently provide a plugin to offer that functionality you can extend Ansible through a fact a cache plugin to Store the information in that in that back-end data store whatever that may be Trying to go one that we don't have Cassandra react So Again base catch so base cache module for a cache module You need to have your get set keys contains and delete and that's basically As long as you could provide provide that basic functionality the rest of it We can handle because we can do Comparison to make sure that the back-end data store has values and then we can get set add and delete based on those shell plugins These are sorry, it's very warm in here Shell plugins are used to properly format commands for remote execution So an example of that's gonna be like POSIX SH or fish shell anybody fish shell show hands Alright three out three. Okay So yeah, so for those who didn't know Ansible will support fish cell So it was originally written to make handling SH versus WinRM Less painful, but it's been extended for all sorts of interesting things. I think we directly support Debian's ASH Which I think what Alchemist? Shell ASH Fish a handful of others things CSH Yeah, so we have various different implementations to it to offer them So this is a little bit more laborious in terms of doing common common shells the shell family Embedded end of life the null you kind of have to define the group of what? What's different or what you is unique about that shell environment? And then these different variables get used throughout the construction of commands and those kinds of things inside of Ansible itself that when When a command has to be run that we can't just implement directly or or makes less sense to implement directly in Python These are used and then You just define the M prefix And they may have any wild suggestions for a shell that we might not have All right I don't know how common this is necessarily in real-world use from a system administrator standpoint, but just know that we have that there So how to write plugins? So anytime Ansible or I'm sorry first off should I write a plug-in so If Ansible doesn't do what you need it to and you need to connect to a system Fetch data external to Ansible store retrieve facts from a centralized system, which would be our fat caching Do custom actions, which is going to be a callback plug-in Or or make Ansible execute tasks in a pattern that we've not thought of which can be a strategy plug-in If if you find yourself in any of these situations, that's when you are a prime candidate for writing Ansible plugins Otherwise, there's probably a module or a plug-in or something Built-in or available on Galaxy. You don't actually have to always search Galaxy first But anyway, so the power of Ansible is its simplicity and its sensibility. So we have the opportunity to do these things So always use the provided base classes number one. There's a lot of kind of standard generic structural Methods defined in there that are meant to help you implement these things so you're not having to reinvent those wheels Beyond that It won't get loaded if you don't load off the base class Why does this keep doing this? Testing debugging plugins so unit tests since it's relatively easy to load The the classes please provide unit tests if you're going to contribute your test up to Ansible unit tests are great integration tests are also Appreciated from a standpoint of our testing we try to favor integration tests mostly because as long as the Functionality of the end result of Ansible's execution of a playbook Mirrors what a user would expect it doesn't really matter what you did the internal structure or the internal code But unit tests are great. Please if you know if you're doing really wild things like a strategy plug-in you protest or two so So we will this is an example of a lookup module Why did this? Chronically, yeah, okay, so we have CSH fish shell SH and I lost it again. That's awesome. Okay. Well, that's been useless all right Well, I had a I had some code. I wanted to walk through there, but Doesn't seem to be my day. All right So a handful of guidelines for contributing plugins back to Ansible so coding guidelines is for GPL 3 is required With an asterisk on it module utility has been kind of a grab bag of licenses in recent years But plugins themselves should be the GPL 3 Docs are are basically required not all the plugin types of documentation sections, but you need at least have Some something useful in terms of comments or doc strings those kind of things for your functions Just because of what's going on familiarize yourself with the modules available And some of the stuff in the base classes because there's a reasonable chance that you will implement something that's already been done And then if you submit a code review, you'll be asked to not do that So we have Google groups as our our development forum Ansible develop on free note if anybody has any questions and then there's two core team meetings a week So every Tuesday and Thursday. We actually have a core community team meeting on IRC that anyone can add things to the agenda to on on github and Please do that because anytime there's a decision that needs to be made anytime There's questions about a review that people have if there's any kind of Debate about a decision in which something should go or people need advisement that they are not getting just from the general Develop discussions that kind of thing the core contributors are there and something to note is that as far as core core contributors go I believe it's about half and half now I think about half the core contributors work for red hat about half the core contributors are community based But it's not just red hatters who have direct get access to to the code base these days I think it's somewhere in the ballpark of like 30 people and about 15 of them work for red hat So yeah, we're doing doing better there So questions No, all right cool. Well Seven spare. Yes. Oh, I'm sorry. Go ahead. Yes Okay, so module so modules are kind of a special thing because they Are are loaded like a plug-in such that They have certain attributes like they're indexing and their configuration those kind of things But they aren't actually implemented like a plug-in because plug-ins are loaded directly as Python classes or subclasses whereas modules aren't inherently modules are actually Loaded as Python sub modules and subclasses when they're written in Python, but they're not required to be written in Python you can write a module in PowerShell or Ruby or Java or C if you feel like it a module just has to accept And return JSON as input and output So you can write in any language you want and that's why they're special because we have this This implementation thing called anti-ball Z which is a tip of the hat to Dragon Ball Z for those of our anime Which will actually allow you to basically package up and bundle a module to be sent over the wire written in any language and the idea there is to allow module development to be flexible because what Majority of people are doing the majority people are writing is going to be module Let me back up. Majority people are writing playbooks or roles Slightly less people will be writing modules for custom functionality Considerably less people are going to be writing a strategy plug-in or you know cash cash plug-in that kind of thing so the you know the flexibility in the range kind of in terms of implementation for for Proper plug-ins, which is you know kind of what we went through today Those are written in Python. Whereas modules themselves have a lot of flexibility to them In terms of what you can do So the the only stipulation that was ever placed on a module was that it has to accept and return JSON the big Kicker beyond that though is for a module to be accepted into upstream ansible. It has to be implemented in Python or PowerShell The window the windows modules are written in PowerShell. I always I sorry windows people. I always forget. I Don't mean to I just forget that you exist Yeah, so the so the question was how do you write plugins for windows the answer is you don't the plug-ins Because in the reason I say that is because the plug-ins run controller side and ansible is only supported controller side on Linux So plug-ins themselves are always going to run controller side on the Linux system Modules, however modules are written in PowerShell for windows. Yeah in PowerShell. Yeah, and we have native So from an ansible perspective, we have native support for PowerShell for module authoring and the mod in the modules are what? carry what Provide functionality for the task execution in a playbook So that's yeah, so modules themselves can be written in PowerShell to try to do the native stuff with With the local with the remote windows host I Have a follow-up question So since you can write custom Modules on our own, so where do we put it in the directory? So the question was where do we put modules that we've written our own So you can put it in your module path you which you can define your configuration or at the command line You can put in the default module path, which is Etsy ansible Modules or in your local directory under the library's Directory name, I'm sorry library Singular not plural so library in your local directory or within the directory of a role So if you've if you've defined a role and you have playbooks and stuff in there You can put library in there and put your custom modules in there and that will provide that functionality Why did it get called library? I don't know and it's historic reasons. I don't know maybe I mean that makes sense. Yeah, that yeah, I like that I should probably ask Jimmy or Michael one day Just always accept that as truth and never really thought to ask why Yeah, well, so it's funny actually because there's a handful of modules that like in their metadata It's like version added just as historic like because nobody knows like it's been here since the dawn of time We did sometime in the past yum is one of those if you go in the version added For yum is is historic. I think yum Existed in like zero dot two or something zero dot one. It was forever ago Yeah, so I had every intention to show a couple examples in like actual code But my mosh connection. I probably should just tethered my phone make that better Anyways, if you have a questions, I'm around I'm happy to like sit down and do some hackfest type things when we have like some time to break out if anybody wants to actually implement a plug-in But hopefully, you know, so like one of the things that I always talked about is Performing automation tasks based on Fed messages and that's something like an idea and I wrote a tool called loopable And that's been deployed in Fedora infrastructure. I have no idea if it's actually being used these days But the idea was is to actually be able to marry a Fed message event to a responsive Fedora playbook and then use plugins to kind of exciting to be able to integrate more more tightly coupled Or more natively with various services within with the within the Fedora infrastructure to to kind of do those things so Hopefully, you know throughout the Fedora project in different areas in different sub projects that I have zero visibility into just because There's only so many hours in the day Hopefully this can be useful to kind of extend the functionality of Ansible to take your automation Kind of in places you might not have thought it could cool. Thank you