 Yeah, so welcome to the session on Introduction to Ansible Collections. Ansible collection is supposed to be the future of content delivery, Ansible content delivery. So I'm Ganesh and he is Abhijit and let's get started. So the agenda for this session is what is collection? Why we need collections in first place in Ansible? Then we'll do a deep dive with collections. Then Abhijit will take over and he will show us a demo on collections. And then if time permits, we'll go through Ansible Galaxy and how we can distribute collections to Ansible Galaxy and couple of references. So about Abhijit. So my name is Abhijit Kasurade. I'm a free and open source evangelist. I work for Redact. I'm a part of Ansible core team. I primarily work on VMware Ansible models and cloud related models. And if I'm not doing that, I'm a co-host for Ansible Pune Meetup group. So where we talk about Ansible and related technologies. Okay, so I work as principal software engineer at Ansible and I work mainly on upstream Ansible project and my focus is towards Ansible networking. In my past job, I have worked extensively on the network management plane and I have worked on developing multiple on-slash-off box automation libraries for networking use case. I also co-organize Ansible Pune Meetup along with Abhijit. And this is our first time to first them. So what's the collection? So Ansible collection is basically a bundle of Ansible content. So when we say Ansible content, it means all kinds of plugins. If folks have used lookup plugins, filter plugins, then connection plugins, all can be part of Ansible collection. Then all kinds of modules and module utils can be part of Ansible collection. Then roles can also be part of Ansible collection. Roles sit at the top level of the plugins. We'll see that in the later slides. And collections can also include playbooks, but it is still work to be done. And collection can be referenced using a namespace and a name. So the current problems with collections. So if you have written Ansible roles, you might be aware of some of the problems that are there with Ansible content delivery with roles. So first one is it is difficult distribution for non-role content. So typically roles are being used to distribute tasks, user tasks, the playbooks, a bunch of playbooks. But if you want to distribute plugins and modules, it is not ideally suited for those kind of use cases. So I'll go through that in bit more details in coming slides. Then there is plugging slash role name collision. So Ansible uses a flat directive structure. And if more than one role have same module name, then there are chances of collision. And there are certain rules that are not very clear about from where Ansible would pick up those modules and plugins. Also with roles, it is difficult to share code among plugins. And with collections, when the content is being moved into collection, the user is owner of that content who writes the collection. And it can be released and managed independent of the Ansible release cycle. So this is a reference structure between role and collection. So the top level is my or my or it is a name space for both roles and collections. Then for role, inside the role name, there are various plugins. So if you see action underscore plugins, the naming system is not similar to that of what is used in Ansible code distribution. And it is basically not the first class citizens within Ansible. And also the code that is present in this individual action plugins or connection plugins, it cannot be shared across each other. And suppose library contains the library folder that contains each of the user modules that are written. And modules is a common code that is shared by modules itself. So the common code cannot be shared across the plugin code. But if you come to the collection side, all the plugins name, plugin name is the top level name. And it forms the first Ansible considers it as first class citizens. So now whatever code is there in module it is so a test word by it can be reused in one of the plugins. And all the modules set obviously modules set in the modules directory. And the modules can also be you also can share the code that is present in module articles. So content distribution, the first problem that is solved with collection is the content distribution. So if you use roles, if you want to use multiple roles, each roles has its own dependency. Then again each role might depend on various version of Ansible base version. And if you bring all those together, if you put all these roles down from Ansible Galaxy, it is not guaranteed that it will end all these roles in together. It will end working properly end to end. So ensuring that it works properly is quite a task when using roles. So roles as we have seen in earlier said they distribute plugins. But suppose if roles are only distributing plugins and they don't have any entry level tasks, in that case they need to be included into the playbook, user playbook even if that is not doing anything. So that execution is required because Ansible needs to seed and understand where this plugin in which role this plugin resides. And again as I mentioned earlier it is not the first class citizens compared to what has been shipped in the Ansible distribution. And again version maintenance issues that I described earlier. So with collections it is solved in a way that it is one-to-one mapping. So a single collection can consist of multiple roles and it can consist of modules that have been used within these roles and plugins. And playbooks are to be done feature. So since it is a single unit it greatly simplifies the versioning system. So you have to just download a single collection package and it will ensure that it works with the Ansible base version. And it can be installed globally in the standard locations then per user base and then content adjacent. So where your playbook resides in that directory itself you can have all your collections. So the default path in which the Ansible engine goes about searching the collection is current playbook and under collections directory. If there are any collections it will first load that directory then it will go into the user folder tilde dot ansible slash collections. And then the standard collections path. So Ansible Galaxy command for folks who are used roles they might be aware of Ansible Galaxy command to create installed and distribute collections. So it is extended Ansible Galaxy has been extended further to also support collections. So Ansible Galaxy collection is a new keyword that is added in 2.9 Ansible 2.9 onwards. And basically the init my name space my collection it creates a directory hierarchy that I had shown in the earlier slide. Then Ansible collection build it basically compiles the collection and generates the tar wall into the same directory into under the collections folder. And then if you want to publish this distribute the content through Galaxy you can publish it the tar wall in the Galaxy under your name space. So this is about distribution of content with Ansible then the second problem that collection solves is a name collision. So as I mentioned earlier it is a flat name space. So when I mention a task in Ansible playbook it is just the task name. I don't mention from where that task or module is been loaded. So with roles the problem is if I have my module module in 2 of the roles. Then which one will be getting used so that is not very clear from user perspective. So Ansible has set of rules but that is not quite evident and there are chances of tripping over those rules and dynamic behavior basically. So if this my module if I want to use it both the modules in the same one is that possible. So that is not possible because Ansible loads the my module at the time of discovery when the engine starts execution it goes and first goes to the set of paths. And when it figures out my module it will load that module and it will continue to use for rest of the execution playbook execution. And then suppose if you have a role with the my module and later on Ansible core comes in and it starts shipping the my module in its own distribution. Then again that is a problem. So the next problem is under your playbook and under the project where you have your playbook if your library directory and under that if your module. So Ansible also looks into that folder to search whether my module is present or not. So if you have that again there is a problem so you are not guaranteed that it will be picked from either role 1 or role 2. It might be picked from the library directory. Then again if the standard path has this my module again that is a problem. So this issue exists with all the modules and plugins that I just referred. So how it can be solved with collections is so collections has this namespace it has the top level namespace and the name of the collection. So if I have to refer my module namespace then I am using this dot notation. So it is the fully qualified collection name along with the module name. So my namespace my collection my module. So Ansible engine will go into this particular path and that will load that particular module itself. So user is pretty certain that which module will be used by Ansible. And since collection will have a set of roles I can also define in my task the exact complete fully qualified collection name along with the role name. So FI dot BGP FI here is the org name BGP IP is here the collection name and the provisioning role is a role name. So on top of that there are some built-in collections so Ansible dot built-in when this is upended before the task. It is almost always certain that the module of the task will be loaded from the ship version of Ansible distribution. And then Ansible legacy is if you are not repending anything by default Ansible engine will prepend the task name with Ansible dot legacy. And it will ensure that the old behavior is as it is. So there is nothing breaking when users move to the Ansible 2.9 or 2.10 version. And they can work as it was done before along with addition of collections. So this is a sample playbook that uses collection. Host here defines on which all hosts I need to execute my task. Then under task I am giving this fully qualified collection name along with the module. So my name space my collection I think I think of the module and I will be always always certain that I think will be loaded from my collection name space. Then Ansible dot built-in dot ping it will use a ping that is packaged with core. If you are picking under library that will be ignored and Ansible is Ansible engine is pretty clear certain that it needs to be loaded from the core module. So again you can also prepend it with Ansible dot legacy. So that will again load from the core directory. Suppose if you have library there it will be it will follow the legacy rule set to look for the ping module. Then the same applies for the filter plugin. So I can have a filter in my collection name my collection collection the filter plugin. I can use that along with the legacy task. And then ping still works by default Ansible engine will prepend Ansible dot legacy to the ping task. So if you see this is awfully verbose as compared to what is currently supported. So to solve that problem Ansible 2.9 has added this collections keyword. And collections keyword basically contains a list of Ansible plugins. Sorry a list of collections from which Ansible engine will look into one by one starting from top to bottom. So if I have my module in my task Ansible engine will first look into Ansible dot built-in. If it is shipped with core distribution or not. If it is not then it will move on and it will check into my collections. Then it will go and look into other collection and if it is not there it will throw an error. So it is first found basis it will load and use that. And suppose if I don't want to do that I can just mention a fully qualified collection name. And Ansible will just directly go there and pick up that load that my module collection even that is fine. So the collection keyword is a ordered list of collections. And if the fully qualified collection name is not mentioned then Ansible legal sees auto appended. So it basically uses the collection search for existing place. So in existing place you can just add collection keywords and list of collections you want to refer to. And it can be used that is I mean the change to collection is ensured to be minimal when you use the collections keyword. And this can be set either on play or on block or on a task basis. So this was about the namespace collision how collections solve that problem. And then the last problem that collections all is code sharing. So with with roles the problem is as I mentioned earlier whatever code is there in module utils it can't be used by plugins and other modules. So the general tendency is for folks to contribute to Ansible is to have a ship those modules as part of Ansible code distribution. And there are around 1000, 2000 plus Ansible modules right now shipped in core. And it is being difficult to maintain and there are a lot of cool request out there. So that is kind of a bottleneck. So the in future Ansible core would be a stripped down version. And most of the content will be moved out of Ansible and into collections. So this code sharing problem will be solved by collections. So in inventory plugin if I have to use something that is defined in module utils I can't do that. And unless the code is being shipped by ship with an Ansible core distribution I cannot use that in my roles. So code sharing is possible in collections because collection is an entire Python name space. When the collection is installed with Ansible the Ansible collections directory is generated and under that all the collections are being installed. And Ansible collection is a root Python name space. So if I have to use something in my code I have to just do import Ansible collection my name space my collections module utils. I have to just refer the entire path and this can be used across collections. So Abhijith will take over and he will give a demo of what I just explained. So I'll show you the how various problems can be solved using collections. So this is the demo directory over here. You will notice that it's a plain Ansible project directory. You will have Ansible CFG then collection directory a playbook that will run the collection inventory file. So if I open this demo dot pl you'll notice that it's a plain Ansible playbook. It mentions that we are using a rainbow dot test collection from the current directory itself. And I am including a role called as demo role from it. So I'll show you the how collection looks inside the directory. So over here you will see that Ansible underscore collection is the root directory of the collection. Inside this directory if I do three you'll notice that there is a collection called as rainbow. It is the org name that we are referring like in the case we have a five or anything like VMware or other organizations can have their own name. So rainbow is the name of the organization. They can have multiple collection inside it that can be a test. So inside test directory test is like it's a name of the namespace. It's not to be collaborated with like test directory. So it's a test collection. It contains different directories like readme, docs, galaxy.yaml, plugins. Plugin directory contains all the directory structure which is required like lookup plugins, module utils, modules, rules. And when playbooks are going to be introduced in the collection itself they will be residing next to rule directory. Okay so here so we are trying to solve a problem where let's assume that you have a shared code that is being used by a lookup plugin plus Ansible module which goes in module utils directory. So with the collection if I open this file so there is a shared function called as shared thing. It's just returning something. Okay right now I wanted to share this code with both lookup plugin plus Ansible module. So how do I do that with collections? So I'll show you how it looks in module direct modules. So this is my custom ping module which is using that shared code. So this is a bare bone Ansible module which is using that shared library. So if you notice over here this whole line it is importing that function from Ansible collection which is the root of the directory then rainbow the collection name, the org name, the test collection, plugins, module utils and the shared code. So that's how we are trying to solve the sharing of code within the lookup any plugin directory and plus Ansible module. Same case goes with the filter things. So I have a lookup plugin called as demo lookup. You will notice that I am using the same shared code from here and I am using it in the lookup plugin. So this is a bare bone lookup plugin and I am using the same shared code over here. So that solves the first problem of sharing the code between plugins and the module itself. In order to run this thing I simply call I specify stating that collection is the rainbow dot test which tells the Ansible core engine that you should use rainbow dot test rather than using the custom other legacy or built-in things. So if I run this like this Ansible playbook demo dot pl. So over here you will notice that over here the first task is being used from this collection. So this is that called shared thing. This is my custom ping module which is being called from the collection and not from the built-in Ansible module directory. This is another thing that if I open this playbook this is being called from this role and you will notice that first I am calling just the ping directory like Ganesh told like it is running from the collection itself and then here I am calling it from the built-in directory ansible dot build-in dot ping and the third is that if I want to just specify that I want to use it from the current collection itself then I will specify that rainbow dot test dot ping and that's how it is working and the third thing is like we are using it in the lookup plugin as well that rainbow dot test dot demo lookup. Conclusion? Yeah, I will just show you the help thing. Yeah, any questions? No. Thank you.