 our reminder, we'll be doing ATCASC, Incible Tower Configurationist Code. It's a subject that is near and near to my heart, and Kadar Kulkarni is going to do a great presentation for us. So we'll be starting again at 12.20 Eastern Daylight Time. It'll be again in three minutes. So Kadar has corrected me. It's ATCASC. I believe that is correct. ATCASC, Ansible Tower Configurationist Code. For anyone who's not familiar with the concept of configurationist code, it's the idea of building your systems based on a configuration repository, i.e. GitHub, Subversion, whatever, committing all your system configuration there, and treating it like software. It's a great idea. It's a great way of making sure that your systems are built consistently every time. Okay, let's get started. Let me go ahead and share the video with the audio. Well, hello everyone. This is Kadar Kulkarni, and you're watching this presentation recorded for Defconn.us 2020. Now, if you're watching this presentation, I'm assuming that you're probably interested in the automation as a topic. You have heard of Ansible, or maybe also used Ansible and Ansible Tower. But if not, that's fine as well. We are going to talk about both of them today. But we are also going to talk about infrastructure as code, configuration as code, kTops, and so on. So let me tell you a story. A few months ago, my team was trying to decide what would be our de facto automation tool to fulfill some of our use cases internally. And being Red Hat, the first thing we do is we look at what tools do we have available within Red Hat that we can use. And that's what we call dog fooding, eating your own food. And in that we decided to use Ansible Tower. Now, if you are using Ansible Tower, that means that you have to deploy a fresh instance of Ansible Tower, and then configure that Ansible Tower with job templates, workflows, and all the metadata it needs to do function properly in order to provide the functionality for all of your use cases. And when you're setting these up, it would mean that you will use these things using Ansible Tower UI or CLI or API. And maybe you will integrate these tools into your CI CD system, and so on. Now, the problem that we faced as soon as we started thinking about Ansible Tower was how do we manage the change? We were a team of three sys admins or DevOpsish engineers who were responsible for turning up this Ansible Tower for our team. And this was going to be our production Ansible Tower instance, which in case goes down, stops working, crashes and burns, it means going to take away our good night's sleep until we stand this backup. And just before we did this, I had an Ansible Tower instance, which had the exact same problem. I had one tiny playground Ansible Tower instance, which I was using for some functionality, just to make my life easier. It wasn't really critical. Nobody on the team was using it. But one day it stopped working. And I had to figure out, how did I set that up in the first place? Beginning that out based on the documentation or your in-memory notes is not really easy. It's a difficult task. And that's the time when I was realizing that maybe this is not really a clever way to do it. And since we were asked to set up a production Ansible Tower where maybe 20, 30, 40, 50 people were going to rely on it, plus all of our automation and CICD pipelines, it's going to be much more difficult if this production Ansible Tower, which has complex workflows and jobs is going to go down. And it's going to be difficult for any of us out of three of three people on our team to set this back up. So that's the time when I started thinking about and looking around the solution to use infrastructure as code, because we are in 2020. And I had recently been also introduced to something called JCASC, which is Jenkins configuration as code. And I found that concept pretty much relevant to what I was trying to do. And hence the name at CASC Ansible Tower configuration as code. So imagine if you have this production Ansible Tower in your production environments and it goes down, it crashes and burns into ashes. How do you stand that up back from the scratch within lowest, shortest amount of possible time to respond? So we developed this solution, which can do this for you in less than 30 minutes, out of which maybe 20 minutes can be set up. And then 10 minutes can be the configuration. So you'll spend about 30 minutes standing up a new Ansible Tower instance and running configuration as code against that Ansible Tower to get it fully functioning. If that sounds all interesting, let's dive into the slide deck. So a little bit about myself. I have been at Red Hat for more than four years now. And I'm a senior software engineer in management business unit, where I joined as a quality engineer who was responsible for Python and automation testing. But eventually I basically got interested into infrastructure, automation, DevOps, et cetera related practices. And that's where my interest lies right now. And that's what I've been working on in the past two years. I have managed multiple Red Hat virtualization environments, OpenStack environments, automated a lot of deployments and D2 operations using Ansible and Ansible Tower. I've worked with Satellite. I have worked with CloudForms. I have worked a little bit with some of the newer Red Hat offerings, which are SAS offerings, like Insights, Automation Analytics, et cetera. So now let's go into the topic. We are going to talk about GitOps. If you have not heard about GitOps, GitOps is basically a special way of saying that everything in your infrastructure has to be written down as code, has to be added into a Git repository or some kind of version control repository. And every change to your production infrastructure is driven by this code. So what that basically means is, since the day you stand up a new Ansible Tower or any kind of system, you are going to have all of its setup, installation, and configuration, as well as D2 operations written down in the form of some kind of code that you can execute and recreate the environment, update the environment, make changes to this code, and then use that code execution to make changes to your production environments. So you are not actually logging into any of your production systems. You are actually writing your code and then that code takes some form of automation, execution, or CI CD to go from that code to the production level changes in your infrastructure. And that's what the GitOps is. So GitOps basically enables you to do infrastructure as code or configuration as code. Now, a little bit more into the main topic, the introduction is here that saying that we have three important things to cover in here. Infrastructure as code, Ansible, and Ansible Tower. As you might have understood thus far, that infrastructure as code is very important for us. And for a lot of you, if you are trying to make your lives easier, because infrastructure as code has its own advantages, it basically helps you boost your confidence as DevOps engineer. And Ansible as well as Ansible Tower, these are the two automation tools that will help you automate all of your infrastructure activities, and they have their own advantages. So let's look at the Ansible Tower first. So Ansible Tower, I'm going to talk in relation with Ansible differs in slide things. So basically, Ansible is a command line tool. It's a basically a package that you install on your system, on your laptop, which you can do it right now using PIP install Ansible, or maybe your own preferred package manager. And installing Ansible allows you to run commands using Ansible, Ansible Playbook, and so on. These commands will help you execute ad hoc Ansible tasks or Ansible playbooks. And these playbooks are usually what is the main unit of execution in the world of Ansible. This playbook can contain a lot of tasks. So each task can do a certain thing against the host that you are running this automation for. It can install a certain package, it can remove a certain package, create a new directory, copy some files, download some files, configure something, and so on. Now, if there are some repetitive tasks that you are wanting to use against different hosts in different contexts and using different playbooks, then you can probably actually abstract these tasks into what's called a role. So in a playbook, you can have one or more roles. And these roles are basically a package that you can share across different playbooks. So if you are setting up an HTTP server, if you're setting up an email server, if you're setting up an Nginx server, Apache server, or whatever, in all of these, you might want to connect to your AWX, Google Cloud, or GC, sorry, Azure, to actually create an instance or a container or something on which you can actually do the further deployment and configuration. So that part of initial deployment is same no matter what kind of server you're setting up. So you can create a rollout of that and abstract it away. So you can have three different playbooks to set up Nginx, Apache, and email servers. And then all of these playbooks can have that one common role that they are calling to set up the initial virtual machine or an instance or some kind of container, whatever is your choice. That way, roles allow you to abstract tasks and be reusable across multiple playbooks. Now, in terms of Ansible Tower, it's an enterprise level tool that enables you to use Ansible at enterprise level. So thus far, talking about playbooks and roles can probably seen as an individual thing. You can do it on your laptop. I can do it on my laptop. But if you are trying to do Ansible in your production environment, you might want to have some more visibility control over who's running what against which hosts or which servers. And you need to have some historical logs to make sure that everything has a change log or audit trail that you want to check if something goes down and so on. So these enterprise level features of centralized Ansible execution, role-based access control, multiple user environment, having these debug logs present across all these environments, everything is possible if you use Ansible Tower because Ansible Tower is basically Ansible plus all these enterprise level features. So I'm going to show you a screenshot of Ansible Tower here. The screenshot basically shows you that this is the login page or landing page right after your login to Ansible Tower. And in this case, this is the snapshot or screenshot of our production Ansible Tower. This Ansible Tower has 111 hosts that it is managing because Ansible Tower can do inventory management. It has, excuse me, number of job templates that were executed on it. There are a number of recent job runs that are listed. Then there is a graph showing how many jobs are being executed on any given day. And this is the main page that gives you a lot of information about overall Ansible usage in your organization. Then you can look at the left-hand side navigation menu and navigate to some of the more important and individual types of objects like you can look at the templates, you can look at the credentials, projects, inventories, users, teams, organization where you do all role-based access control. You can have notification setup. You have instance groups if you have like an Ansible Tower cluster settings and so on. So now that we have looked into Ansible Tower and a little bit of Ansible, let's dive into the next important question. So how to set up Ansible Tower infrastructure with a single playbook execute? Now imagine you are in the situation. You just learned about Ansible Tower. Now you want to basically start using Ansible Tower. The first thing you do is you go out, you find out what information you need to download and install the Ansible Tower, you download it, you install it on certain system, maybe a virtual machine somewhere, maybe in a container you want to spin up a container that has Ansible Tower and so on. So you basically take care of all these things by hand and even after that your job is not done. You still have to set up the job templates, workflows, projects, credentials, users, organizations, LDAP are back, everything and all of this can take you a day or maybe take you a week and it's not done there. You have to constantly add new things if your environment is rapidly changing or with any new Ansible Tower instance, it's going to be difficult for you to get done in first pass. You're going to have to go back, change things as things work out for you and you're going to have to keep doing that. Now imagine if you are doing that by hand every time and if there are like three people who are responsible for doing this in three different geographical locations, all of them are trying to update this single Ansible Tower instance and now this instance basically has a lot of mess because nobody knows who did what. Doesn't sound very good, right? So with AdCask, with infrastructure as core, with configuration as core, you basically use all of this automation to set up and configure your Ansible Tower using a single Ansible execution. So what that means is basically you can have an Ansible Playbook that can execute against Ansible Tower and set up that Ansible Tower for you run that against like a new instance or maybe just run that against your AWS environment, create a new instance, configure it, install Ansible Tower, configure it to use all with a single Ansible Playbook. So basically Ansible Tower, which is a super set of Ansible, which is basically the tool that lets you interface with Ansible at an enterprise scale can be set up using the Ansible itself. You heard that right, Ansible Tower collections here on this slide is the answer that lets you do that. It's a little help from AdCask. AdCask is the project that I built based on top of Ansible Tower modules within Ansible Core as well as Ansible AWX collections, which is the newer thing that came out to supersede the Ansible Tower modules within Ansible Core. So basically AdCask is going to take you from a fresh installed Ansible Tower to a fully functional, ready to use Ansible Tower instance within five minutes. It's super fast, it's super convenient, and it basically takes a lot of guesswork, a lot of manual effort, a lot of toil if you're an SRE. You'll understand this concept. It will go all out of your day to day job because Ansible Tower configuration has scored is taking care of all of this manual effort for you, right out of the box. So enough about Ansible Tower and the configuration has scored. I hope you understand a little bit about what we are trying to tackle here. The problem that you're really trying to tackle were trying to make your Ansible Tower easy to reproduce, easy to stand up, easy to reconfigure. Everything tracked into the code repository. Everything has a commit history assigned to it. Every change is made through the code. So all of that, anything that you have heard for infrastructure as code applies to this. So AdCask, what is it providing? It is providing basically an as code solution to configure your tower. And who does it benefit? It benefits anybody who is a DevOps engineer, who is an admin, like a sysadmin, who's using Ansible Tower or whatever you might call these people or maybe Ansible developers who wants to work on Ansible Tower. Imagine a situation where you have this production Ansible Tower. This is a real story. We have a production Ansible Tower in our team. And we have three people at the beginning, all of them working on developing new playbooks, new roles and everything that goes into our production Ansible Tower. So how does AdCask help them? We are going to look at that in the slides coming up. But for now, let's keep that in mind and move on. So what all is supported? AdCask basically supports all kinds of objects creation, modification and deletion, which are listed here on the slide. So I'm not going to read through all of them. But basically, if some of the important names for Ansible Tower objects would be an Ansible Tower job template or workflow, credentials, hosts, inventories, notifications, and so on. So you are able to manage all this through your code. And at any given point, if you ask me in my team, like, hey, do we have a notification for that? Do we have a job template for that? What job templates do you have? Or what permissions this user has? I don't log into the Ansible Tower. I log into our GitHub or GitLab repo and look at all of our code base, which is basically using AdCask to configure our tower in the production. So I'm able to look at the code and figure out what is in the production Ansible Tower. Instead of going into the production Ansible Tower and looking at things manually flicking buttons and knobs and looking at forms and so on. That is all possible. But for an admin who's trying to be efficient with the job, looking at his code and telling quickly the answer to any of these such kind of questions, is really easy and really useful. So are you ready for some kind of YAML hotness, some nice YAMLs that are coming your way? These are some examples that I have taken from AdCask that we use and sharing some screenshots here so you can understand how it is written. So there we go. This one is showing you a tower credentials dictionary, which is a list of dictionaries. And each item in this list is a new Ansible Tower credential. So we have one, two, three and four credentials. Each of them are of different type and take different kind of parameters. And using this kind of YAML, I'm able to add like maybe another 100, 200 or however many credentials I want in this file. And when I run AdCask, when I run configure tower.yaml playbook, reads this file and creates all these objects on the Ansible Tower. Now imagine if you had to actually sit, look at your Ansible Tower and write down all these credentials or make all these credential objects in the UI using that form, clicky clicky clicky. It's not a great thing to do. It's not great use of your time. With this playbook, with this AdCask, with this YAML file, you're saving tons of time for everyone, for you, for your team. You're able to deliver your targets on time. You're able to quickly reconfigure or add new things to your tower. And you're able to have that assurance, that good night's sleep, that rest that will tell you that in case something happens to your Ansible Tower, all of the code is written. You just execute this again and you reproduce this and your Ansible Tower is ready for the production. So how does that sound? That sounds interesting. Let's look at another example. This one is from the towerprojects.yaml file. Here we have a top level dictionary key which says tower projects and it has a list of dictionaries. So we have added two different projects in the tower using these two items in this dictionary. All of the projects are configured with all the parameters it needs right out of the box and you don't have to remember any of this because it is written down. Now job templates. Job templates is the object which is executed within Ansible Tower alongside probably sometimes in a workflow template. And each of these things can be created using YAMLs that look something similar to this. And then this job template is responsible for executing your playbook which is highlighted here in the playbook section, playbook utils, remove VM. So it is able to remove a virtual machine for me, for my team members using this job template. And I have put together this job template and hundreds of others using the code in that file. And I never have to remember a single thing out of it because who can remember all this? And you might argue that I can document this, but what document is better than the code itself? You can read the document and try to do the UI things. You can try to click in the UI and fill up different forms with the information from docs and create these objects. One job template can take you maybe two to three minutes to fill up. But if you have the code and if you are running the playbook that actually creates these templates for you, it will create the entire functioning Ansible Tower in five minutes or less. Of course, depending on the size, the time might vary. But for us, for the tower that we are using in our team, it takes five to 10 minutes to set up after the initial setup of the Ansible Tower, a fresh Ansible Tower instance is made available to the adcast. So now, what we are going to do is we are going to quickly switch to the demo and we are going to look at how all of this works in action. Demo time, everyone. So here I have a prerecorded demo where I'm showing you an Ansible Tower that is freshly installed. I'm just logging into this fresh installation of Ansible Tower right now. As you can see, I'm highlighting some of the top cards. So first card here was hosts, then failed hosts, inventories, projects, everything. And as you can see, it is either one or zero. So if it is one, it is usually the demo host, like the local host, demo inventory, demo project, demo job template, because everything demo comes up as soon as the Ansible Tower is installed because it comes up pre-installed or pre-configured for you in the Ansible Tower on any fresh installation. But it doesn't really do much. It's just demo purposes only. So from here, we went on to add hundreds of hosts, tens of inventories, tens of job templates, another few workflows that comprise all of multiple job templates, adding users and lab everything using AdCast in our team. And every time we make a change, we make change to AdCast. We have a review process. We review the changes, we merge them, and we push these changes through CI CD to production Ansible Tower. So next up here in demo is that I'm going to show you templates. As I said, there is demo template, demo credential, demo project, demo inventory. And in that inventory, we have local host, which is the only host we have. Now you look at the terminal, you can see that we have tower configurations directory under AdCast. In this directory, I have a bunch of files, and each file has different tower object that you can specify. Now if you look at some of the files that I'm going to cat out here on the screen, you're going to see the same or similar ML formatted text that actually comprises of the configurations that I want to make or apply to my Ansible Tower in order to make it a useful instance. So what I mean by useful instance is that if you log into Ansible Tower as soon as it is installed, it is really not useful because there's nothing in there. And now you add all the stuff in there, all the job templates and workflows and all the metadata that is required to make that happen. Then that tower is useful for you or anybody on your team to start using it to run automation against your systems and get your job done. So let's look at tower job templates file. Now if you look at the tower job templates, I have two job templates and few other things that are commented out as an example. I have left those there because these help you understand how these things are created and how you can create more of these. But right now we just have two job templates, test template one and demo job template. And they have all the respective fields of information filled up. And next is tower inventories.yaml. So here we are creating inventory local host, site lab admin inventory. We are creating the RevM01, RevM02. The four inventories are being created. Everything else is left there in comments as examples for your reference. So now what I'm doing is I am, I'm running this Ansible playbook. Sorry about that. I'm running this Ansible playbook that is actually responsible for reading all the files under tower configs and create all the instances of all the objects that are required within Ansible tower, all the metadata for you. So now it has started execution. It is going through all the steps. So it makes sure all the packages it needs are installed. It is applying some settings to that Ansible tower. Then it is going to go into the next thing where it is setting up the organization, setting up the user. It is then adding the tower teams. So Ansible tower has ability to create teams of users and assign them certain permissions because it can provide the role-based access control. Now we are adding the credential. Obviously, if you set this no log equal to true, it will not actually share all the credentials on the screen, but these are some dummy credentials for you. Then you have some inventories that we saw earlier. So it created all four inventories we needed. Now it's adding the tower project. And in the tower project, we are adding two different projects. Both of the projects are dummy projects pointing to one of my GitHub repo. And once the project is added, what Ansible tower does is it connects to the GitHub repo or any repo where you're fetching that project from and makes sure that it is accessible. It clones that locally within Ansible tower instance so that all playbooks are available for you to execute. So that's what it is doing right now. It is sleeping or it is basically in a sleep timer where it is trying to make sure that all the projects that we added can be updated or are updated. If the projects don't get updated or don't get synced, that means that projects sync failed and Ansible tower would not be able to access the playbooks from that project. Once that's done, we are creating tower inventory sources. We are adding hosts. We added a local host here manually. Then we are adding the job templates, which is what we are going to be able to execute once it is ready. Then we are adding some notifications. So we have ability to do emails, IRC, Slack, and so on. Notifications from your Ansible tower on each job execution success or failure depending on how you might want to set that up. So now once everything is done, once we created job templates and workflows, the last step in the playbook is to make sure apply a role that actually makes sure all the Ansible tower objects that we created have correct permissions for the correct users that are going to use the Ansible tower so that prevents anybody from your team logging into Ansible tower and doing something inadvertently, which they are not supposed to do or they are not supposed to see. Now I'm going to refresh this page. I'm going to show you. Let's see. The inventories that we listed earlier are all created. We have inventory source created. Everything is pre-filled for you. Projects are created. Projects are showing the green dot in front of them. So they are synced. All credentials are ready and the job templates are also ready. So now this tower is much more useful than it was less than four minutes ago. As you can see Ansible tower is able to set up itself using AdCast project or you are able to set up the Ansible tower using AdCast project such that it becomes very easy process to set up and configure the Ansible tower. Now I'm going to switch back to the presentation and we are going to look at the next slides. So here on this slide, what we have is the workflow that we follow in our Ansible tower setup. So within our team, we are using GitLab to store all of our AdCast code and we have a pipeline configured in GitLab CI that actually does a lot of things that I'm going to describe now. So suppose you are an engineer who is responsible to write an Ansible role and a playbook to use that role and you want to make that role available or that playbook available to your entire team using your production Ansible tower. So anybody on your team can log into production Ansible tower and use your job template which basically is going to execute this playbook or role that you are writing. Now to make that happen, there are two routes. One, you ask your sysadmin or you go in yourself into your production Ansible tower and do those changes manually. Is that clever or is that easy? I'd say it is easy probably but not clever because AdCast helps you have all of that recorded and we have developed a pipeline that does all of the workflow behind going from testing staging to production kind of scenario with your code. So you write some code and you submit a merge request in our GitLab repo. As soon as you submit a GitLab merge request, our GitLab runners will pick up that merge request and connect to our Rev provider, Red Hat Virtualization provider which is also shown as Overt on this slide. Overt is the upstream name for those who don't know for the Red Hat Virtualization project and upstream means it is open source and it is available on GitHub or some kind of code repository on the internet free of cost. Now it connects to that Overt provider, creates a new virtual machine which is based off of Red Hat and Freslinux and installs a fresh instance of Ansible tower on that machine, configures it using AdCast and makes a test Ansible tower ready for you. Using this test Ansible tower, then you as an engineer shown here in the diagram is able to go into this test Ansible tower and run some tests based on what kind of changes you have proposed. So if you are proposing a new playbook or a role, you might just want to make sure the job template associated with that is available and is able to execute successfully. If you are changing an existing code anywhere in the repository, you might want to do more, you might want to check, test multiple job templates, workflows and whatnot to make sure that whatever you are proposing is not breaking anything else in the production Ansible tower. So you are able to use AdCast and set up this test Ansible tower for that sole reason where you make sure that everything is working and me as a reviewer is also able to look at this test Ansible tower, run anything multiple times and make sure that it works before I give you a thumbs up on your review and merge your merge request. Now the test step here or the step where you determine how your code is affecting production Ansible tower or what all tests you need to execute is going to be a manual effort on your side on the side of the reviewer but we are working on a project internally in my team which is called Ansible genealogist which can look at all of your AdCast repository and trace the changes between your proposed code and the production Ansible tower code that is running in the repo. So basically using that relationship derivation, we are able to tell you what job templates and what workflows you are going to affect based on what changes you have proposed in your merge request. And taking that output you are able to actually reduce or remove all the guesswork of what exactly you as a contributor or I as a reviewer need to make sure work is working before we merge your merge request or pull request your code into the production because all of that is derived for us using Ansible genealogist and this project would be open sourced at appropriate time. Now this project is going to help you get that output which then you can consume into an Ansible playbook that can read this output and call all the appropriate jobs and workflow templates in automated fashion so your entire testing phase is automated and if the Ansible tests pass if all the tower jobs and workflows that you have executed in this automated or manual testing depending on how you are doing that are passing in that case your merge request your proposed changes are really good and we are able to merge them without any significant harm to the production environment. So once we merge them we run the final step where we have another stage in our GitLab pipeline which acts on any merge request which is merged into the master or the devil branch and takes that and push those changes or reruns the Ansible tower configure dot yaml playbook reconfigure Ansible tower prod such that the new changes that you proposed are injected into the production Ansible tower and are made available to all of the users on the team. This way we have a proper workflow for anybody who's trying to contribute to Ansible tower production and this helps us all of the admins and all of our team to function more effectively than it would otherwise because having changes done manually is difficult and not easy to track versus if you have all these processes and all these steps which are all recorded and automated you can check which change introduced any failure if any failure happens even after all this testing and you are able to revert the changes and you are able to rerun the configuration so that those changes are taken out or maybe you can just destroy the Ansible tower and just rerun from a previous commit and create a new one which is identical to the previous production because it has everything that you need that's that's how easy it is so how do you get started if you're really excited about this if you have installed an Ansible tower and go to this repository this repository has a really nice read me and if you don't understand any of this stuff you can contact me I'll have my contact information at the end of the slides but basically you clone the repo you go into tower configs directory you add all this stuff that is required to be added there and you run the playbook which is configure tower.yml under the playbooks directory in this project and that takes care of setting up all your Ansible tower based on all the tower configs that you have in that directory of tower-configs a quick introduction or quick overview to the milestones that we have had in this project the initial idea was conceived in January of 2020 when I had some incidents with my old Ansible tower and we were also tasked to set up the new production environment then we developed AdCask based on the Ansible tower collections then we open sourced that AdCask so it was open sourced on GitHub in March 2020 in April I presented that to a larger Ansible community within Red Hat where a lot of different people from different teams and different functions and different geographic locations participated and a lot of them liked the idea or were working on a similar idea so we decided to work together and in June we actually released our new project called tower configurations on Ansible Galaxy which is under Red Hat COP namespace or organization it is based off of AdCask and some other ideas that enhance the stuff that we had in the AdCask so tower configuration is the next step and if you were to try out AdCask and if you really like it I would also recommend that you move on to the next step and look at the tower configuration which is built on top of AdCask in a certain way so right now at the present we don't have any of this supported by Red Hat but we are actively attempting to get this into Red Hat Automation Hub make it available to all of you to use all of the customers by and this is going to be supported solution if it makes into the Automation Hub right now it is unsupported and you have to use it at your own discretion here are some testimonials I'm not going to read those out but this is what people said about the solution once it was presented to them and lastly thank you so much for attending I hope you learned something new today I hope you learned about GitOps infrastructure as code configuration as code ansible and ansible tower and I hope you really enjoy enjoyed this and I look forward to hearing any questions from you and this is all my contact information and I'll be happy to answer any questions offline after the conference on any of these communication mediums thank you so much and you have a great day thank you Kedar I it was a very interesting presentation and configuration as code as I said before and deploying ansible tower and configuring it that way is extremely important do you want to join us for a question and answer okay everybody so let's think of your good questions and comments for Kedar he should be online in just a second can you hear me yes indeed although no video but I can hear you yeah my video is not working for some reason today that's fine yeah I mean for us this concept has really been very helpful and right now our pipeline has been executed 21,000 times in last eight months which keeps updating our production ansible tower using all of the adcast stuff so anybody who's trying to set up ansible tower it's the best thing to do in the beginning itself that's really neat yeah I've had a bunch of folks that I've talked to about ansible tower who've asked how to get started and how to set up the initial configuration as code pipeline and I will definitely refer to them this this is really good thank you yeah thanks everybody for attending the talk and definitely check out the tower configuration because that's where we are putting all of the latest code so adcast was just the place where it got started hence that repo has been also kept in place but I have updated the read me there to point to the new repo as well for all of you excellent so I'll ask you to talk about savings in time yeah definitely so the way we have been using this is basically for every so if we are quantify the savings every single time we propose a new merge request we have had more than hundreds of merge requests that we posted in last few months every merge request has this temporary ansible tower that's coming in and that ansible tower is used for all the testing so the ansible tower comes up gets configured everything in 20 to 25 minutes which you would have to do otherwise for your every merge request and you cannot really share the same ansible tower across different merge requests because you might just it's like testing you cannot test on the same system over and over again because you don't know what's the previous state is right so you need a fresh clean state to start with so 20,000 times running the pipeline we have ran so many tower deployments everything let's say multiplied by 10 or 20 minutes of automation that running and if you are to do it manually it would be like maybe one hour two hour depending on however the big changes so it's it's really difficult to give you a number but that that will give you an idea of how beneficial it is for everybody on your team to have something like this in place and it doesn't have to be something you need to start with like for example you just went ahead and did create an ansible tower manually there is a tower export module that is being integrated in the tower configuration which will let you export from your existing tower and use that to stand up a new tower to see all of your changes are properly working as configuration as code and then you can just use the new tower there on or just keep that as backup so there are many ways to do these things just the configuration as code as you said Alexander is a great concept and everybody should be trying to leverage this it just gives you a lot of peace of mind if nothing else yeah very much agreed that I've been on many projects over the past 25 years or so where some piece of infrastructure gets built and then no one knows how it's done and configuration as code is so critical to making sure that doesn't happen so it's really important in addition to the time savings it also helps with security and the fact that if you document how everything's built when you find a security issue it's very easy to go back in and go in the configurations playbooks or scripts or whatever it happens to be and make an update and you know you're going to get it right yeah and as our points about yeah traceability and attribute and attribute ability as well yep all those abilities special abilities I would say come in very handy when you're in a bad situation because it's easy to overlook the benefit of such a system like when I was setting this up initially we I was also trying to explain that to to my teammates to get buy-in from them that okay you have to follow this process this pipeline where you have to make sure your code is working you have to test it on our test and see what all that comes up for your specific merge request and show me that your stuff is working before we merge that that process itself can basically take a little bit of mind shift change uh I'm sorry mindset shift to to actually get that buy-in but once you have that that really works well in the long term for you and for your team yeah absolutely true so anyone have any questions or any comments for Kadar seems like not too much oh it looks like Alain says any suggestions about managing vaults excuse me um so vault is an interesting thing that we came across when we were moving from Ansible to Ansible tower and what happened is Ansible tower does not so if you let's say if you have an Ansible repo with Ansible vault file that has all your important stuff in there it's not easy for you to go from there to Ansible tower and when you when you talk to Ansible tower team or look at their documents they I think seem to recommend Hashi corp vault integration with Ansible tower to get better security and access levels related modifications or those controls so I would definitely suggest that if you're managing a more production Ansible tower probably you'll have to look at Hashi corp vault integration with Ansible tower for us we just use the Ansible vault directly using loading that as a inventory source and we would have to worry too much about access because we have the inventory and inventory sources permission so that nobody who's not supposed to see it can't see it but it's definitely not not probably going to be production grade for something that's more customer facing and critical yeah that's excellent that's that's a very good point Kadar and I'm I'm familiar with Hashi corp vault as well and Alan says thank you thanks for attending yeah thank you very much for the presentation this is very interesting yep thanks Alexander for hosting me I think there are no other questions I leave this so that the next presentation can get started excellent well thank you so much