 Okay, so first things first it has been pointed out to me that it's an all-choice of a tire for mostly about 3psd I didn't really do it on purpose Yeah, I didn't bring these No, it's it was just what happened it was just the t-shirt that happened to be So be it I mean a lot of it Will also be Should also be Appliable to open BSD. I'm just focusing on free BSD because I'm Much more familiar with it. It also applies to Linux by the way Or I mean if you're a mesochistic it might also apply to Windows Don't know about the preferences Some people have strange kinks. I don't know so First things first so a bit of scope. What I do I mean with reproducible I mean functionally reproducible. I Don't bother to do the things like the people from reproducible builds Where they really want to have every checksum and every timestamp identical Which is a great project and that really doesn't have its uses. I'm However, not sure if Ansible is the right tool to create something like that if you need it Personally, I don't really have the need for it. So and also I tend I mean I could probably but With Ansible it could be a bit tiresome if you really want to control every each and every file on the system So that's also a bit out of scope So one of the one of the things that I ran into previously was that people said yeah Well, we're only using configuration management to set up the system. So they should be it should be identical done Well, let me practice It's a bit different and the one thing is that Ansible defines itself actually more The definition is from the website It defines itself more as an automation tool than actual configuration management Can't pick the differences But in the end it's quite useful because it's more or less procedural and it's in a way quite How you would do it if you would do it yourself? I mean there are specialized modules for a lot of stuff and I Would in most cases it would otherwise using them than just trying to manually edit files if you don't have to But even there it's a step of yes, I want to edit that configuration file to be that way That doesn't mean that the rest of the system you don't really control the rest of the system which might interfere with your With whatever you're currently Um Do and also I mean so you always apply the same steps But if the original system is different for whatever reason for example, you have a newer version of Free PSD installed or for example That the outside environment changes the even doing the same step might not give Exactly the same result if you don't take some precautions and or you decide that yeah in this case I'll just ignore that because I'm fine with it Yeah As I said it doesn't you don't really define the target, but you basically define the steps on how to get there Which actually is a big advantage for for a lot of sysadments because The It's a relatively low barrier of entry Because it's in a way. I mean yes, you have to write YAML which Some people love some people hate But it's it's you define the steps Somewhat similar to what you would do otherwise so Since you just apply steps to it to a system if you really want to have identical systems, it's can be advisable to Basically start as early as possible or as early as needed. So what do I mean with that? Physical machines Do you manage your bias settings your IPM I said things your hardware watchdog settings For VMs the VM configuration Because it might some things behave differently for example you want to clone a system and Whoever is responsible for if you don't if you're in an organization where you don't do it yourself for Whatever reason Who is whoever is responsible for creating the VMs has now given you? And the way if you put in VM instead of the bias putting them or the other way around That depending on what you do it might lead to different results and Keep keeping that in mind Is usually good sometimes you just decide okay. I don't care For my use case. It's irrelevant. It's fine There are some nice there there are some already predefined modules for Dealing with stuff like that for example for the for the tell I've done that before with the iTrack management it's Yeah, it's a bit inconvenient in so far as at least the last time I had to write it from scratch The only functions that really worked were the ones that basically imported The configuration settings from an XML file which Yeah, but I mean if you if you just want to have identical settings It's still relatively easy because you can create them in one system Export it and then tell Ansible. Yes, please both systems. Thank you There are for example for Proxmox that I use for virtualization. There are predefined modules to manage VMs You can always and I've done that before you can always with the URL module It's not that hard to script rest interfaces Which a lot of stuff has nowadays. I'm not sure how the situation is with VM where I have Probably 15 years But I think most of the most of the stuff that got gets used in some kind of enterprise environment will probably have some kind of interface that you can Define and also for example with the with the tel machines If for example, you for whatever reason have to use a hardware rate controller You can also configure that wire the Wire Ansible and the iTrack interface for example Then yeah What one one thing that can fast that is quite important If we are talking about getting the same results or every time Are your steps the highest steps depending on external resources and External in this case means external to your playbook and Order repository that you have your playbook in that might still be some internal hitlap repository where You might get a different result because somebody changed something there If you pull stuff from git lab and just execute it yeah If you pull it from from the public from public github, yeah Well, I mean the old the old the DevOps installer with Grab that raw file from From github from github put it there and execute it Yeah, well you can you can still check the you can still check the checksum to Make sure that it hasn't changed but in the end it's something completely external But still there are a lot of there but also I've had had that happened to me before when you have the situation where You depend on on stuff with with arts with the arts module for example You depend on stuff that's on your development machine at exactly that path and then maybe your colleague runs it and Gets either a different result or an error depending on what happens Also, I mean that the the last thing doesn't necessarily Get you a different result, but it will still produce errors if you didn't manage your known host's keys And if you just if you just run it and don't check the out Don't check the result and yes I've seen that where people just ran ansible from grown and didn't check the result and thought ah everything is fine I Small things Well some some some solutions for for how to deal with external things for example You have a with you deal with packages If you do quarterly you will have a much easier I mean yes, they change as well, but they change it at the at the define at a relatively defined point Because if you just use if you just use latest and then you have situations where you say yeah I'll just I'll test it on the development machine and If everything works a week later, I'll apply the same playbook to the to the production machine Well, if you use packages from latest just might not get the same result Which yeah, I mean you can either fix package I mean you you have the option of just saying Install exactly that version of the package and if that's not available then fail Then you have at least a defined state It's however, I mean, I mean it's a defined state. It might not Still produce work, but it's at least it's a defined state It this however can be quite a lot of work because you have to keep track of a lot of things Obviously you can have your own mirror would your system if you say well, I really want to control What's going to be installed on my systems? It's an option and for yeah, I mean for the For external files, you can do everything from yeah Having your own mirror that gets updated in a defined manner You can at least use checksums You can I mean you can also just download them to your local machine and always copy it out from the local machine I'm doing that for some stuff The only downside is that if it's big files And you're behind an asymmetric internet connection that can be annoying at times Depending on where you sit if you're in an environment where you have symmetrical gigabit link anyway, then yeah So beat Yeah, and also, I mean Just do a degree with especially with internal resources. It's a bit of a question of procedures Do do I want to have so? can I Change the procedure so that I can track the resources That I'm copying around or using to install the service. Do can I can attract them in the same git repository then that I track the Playbook in If I can't do that, I mean I can still say okay, we have for example if we use CI CD system We have a defined system where it's in some way guaranteed that if I reference a release That will not change. It's not not always a technical solution Depending on your environment Yeah, well with editing files, so One of the one of the things that you do in configuration management a lot about from apart from installing stuff. It's Modifying files in some way or another So, I mean if you just want to have small changes there is It's it's relatively convenient to just say well lining file Try to match whatever and change it to whatever That works fine however One of the big problems that you run into is so you're depend for the whole for the whole for the content of the whole file you're dependent on what you start with and So if you install a system at some at a specific point You will start with a specific version of the configuration file that was shipped with version 765 and if you try to set up a new system You will probably start with a newer version of the config file shipped If you do the same modification it might create the same result then The old configuration file with the newer package version that you have on your old system It might not so what what can you do you can I mean use template and template and or completely replacing it and copy over the whole configuration file every time Which means a bit more tracking of stuff in your repository on the other hand You can be sure that it will be exactly as you intended it to be it might still break Obviously over time if your configuration file is no longer compatible with the with the old version But you can you can solve a lot of the problems There is also I mean a lot of the a lot of the time you have You have you have a lot of version things to do one of the big things where I ran into problems with different with different Interactions between a newly set up system and the system that has grown over time is I mean there it's quite convenient. There is a lot of things nowadays that have include include directories as part of their configuration file Nginx jails whatever and a lot of people including myself Like to just say okay, I'll just create a template for whatever if I want to create a jail I'll just have a template that drops a file into the Into the jails D directory And it will be picked up so far as a good and If I add another jail to my configuration there will be another jail However, have you checked that you actually remove the file when you? When you removed it when you remove the definition from your data in your playbook Depending on how exactly the include works Some of the stuff defined in the file included might actually be global and then you might pay accident references somewhere else That has happened to me with Nginx Where basically Since the the upstream definitions as some of the upstream definitions were global basically It only worked because of the old definition somewhere else that should not have been active anymore so it's convenient but One should be aware of the potential problems I mean, of course, you can especially things like Nginx You can create a template that will just create one big configuration file for everything Which works perfectly the only problem is that the template might get a bit Convoluted over time and complicated and hard to debug but Yeah, I mean that's the Some some of the things so for a small note I think probably most of you already know it. I mean the Templating is done with Ginger too, which is really powerful It can be a bit hard to debug in the if you get errors In-enable it can be a bit of a big to debug I'm going over it a bit faster in some areas because I Want to show some things in the end as well So Also for commands, there is always the question if you have something where there is no Direct way to do it. Of course, you can rock run commands in different versions You have command which has the advantage of say where you can at least say, okay This command by the way Creates that file and if that file already exists don't execute it again Raw is relatively use is useful for things like installing Python because it has no dependencies at all But it also is a bit hard to deal with to try not to execute it if it would not do any changes And yeah, I mean expect is in the area from yeah I have to script some program that was meant to be interactive. I Try to avoid it personally actually but Yeah, and that was what the what I first point pointed out, I mean the splitting out part is Quite useful because it makes for simpler. It makes for simpler smaller Structures It can be solved. I've seen it being solved by explicitly Including the files and then you don't have them files might still lay around but they're no longer used It's it's always I think with a lot of the things It's more or less always a thing with how much work do you want to put in? What do I really need? How much time do I have for for stuff and I mean The other thing is also Yes, I know it's it's seldomly true But sometimes one knows that you will only use it once or twice and the system will go away soon Unfortunately systems have a way of staying around Have I have been in the in the business for some years now Related What can also be quite nice to Have because another problem where you run into interesting problems With with things is if you Define your date if you define your data multiple times in your playbook because you need the data in multiple formats That also didn't tends to have Interesting problems at times Because you then forget to adapt something somewhere so for things like that filters are quite useful and really powerful actually because big basically Nearly every transform you can find The lookup filters are actually quite nice. It's also I can show a small small example Later, it's really not hard to write a filter. It's basically a Python module with two functions That somehow Reads the input and then Creates whatever you wanted to create with it in the output On The same thing where you can say okay Get get the data from somewhere else where it's already defined on the same note to not Define data multiple times where you then run into the problem that you have inconsistencies that One is can't can be a bit tricky. I just put it in for more or less for for completeness sake It's also a quite interesting feature Connection plugins can be quite useful if you for example want to have jails that don't have an SSH team and running For example with the Ansible SSH jail unfortunately also ran into some interesting problems because then the become doesn't work correctly Yeah, and that can be a bit annoying to deal with so it always depends on what you want to do and some yeah, it's But what is really quite useful especially for for for the whole thing for starting from scratch is The local transport when you just say Yes, please run these modules on my local machine if you for example want to Script some rest interface for as we put it creating a virtual machine Adapting the adapt adapting the automatically adapting the monitoring so that the monitoring for the new host behaves correctly That at times you need to you need you actually need to not only you can run it locally, but you need to run it locally because your machine might be in a DMC somewhere and And you have the problem that it probably shouldn't be able to access your monitoring configuration interface Also with with delegations You have the same you have you have the same thing and they're you we're back. We are back on Trying to control everything. So for example, I tend to If I create if I create the host, I also automatically Configure the backup for it so that I don't have any unpleasant Experiences that somebody yeah, we created this new host. I will do the backup later Nothing will happen on it until then it's fine In my experience you find out that you didn't do it when you actually need the backup and I don't know about you, but I Usually I'm in a in a environment where I deal with humans and humans make errors including myself It's not like I it's picking on users It's much easier for an admin to create a big problem that needs a backup than for a user Because the user might not be able to delete everything. That means actually Or mangle everything I've also heard people who forgot just forgot a Wear-glaze in a delete statement in a database And if you also if you're for some reason also have an auto commit and we enable then yeah You're back at do we have a backup? um Yeah, also, which is actually I got got quite interesting especially once you also start to create Once you really enable everything really try to control everything my experience your Inventory tends to grow and while Main the most the most basic them the most of probably all of you have seen the most basic Version of an Ansible inventory, which is basically an e-file with a list of hosts With some group definitions If it gets bigger at some point You will lose track of stuff, especially if you maybe need the same inventory in multiple projects um Well The dynamic inventories are quite useful there and come to the rescue And it's actually quite simple because as with a lot of other stuff Ansible has this interesting notion that well yeah, this file is actually is executable So I'll really love read it, but I will execute it and see what it gives me back and Well, I mean the dynamic inventory basically just Needs needs you and it needs you to print out the inventory data in the chasen format and So for example get getting getting the list of virtual machines from whatever virtualization Environment you're running Might not be that hard to get I've also done it before when you when I had Had to deal with a few hundred embedded systems that I didn't want to manually enter it and keep track of in an inventory because well There is already there somewhere and I've also I've also be very previously used You used basically the The option of if you have any kind of management system for your environment where you're already keep track of What systems do we have? What should they do? stuff like that Yeah, even that's it's also quite nice to just pull it out because then you just define it once When you when you when you document your environment and you already had to have the information Okay, this this host is in the group somewhere and why not reuse the information in deploying stuff Which again helps with having a consistent state and not having any Having not that many Surprises because you forgot you forgot. Ah, yeah, but that came from there and that came from there You only have to look at if if you say, okay, I want to duplicate that host You look it up in your documentation system and say, okay That's the definition of the host. Let's duplicate that and Ideally you have a you have 100% of it of a application Obviously you're back to Obviously you're back to depending on where you get your data The problem is you're back to some external dependency that you need to trust Yes, on the other hand, if you don't trust your own documentation It might be a good thing to also think about what to do about that because that can also be quite annoying if your system Don't really correspond for what you do with your documentation That's another that's another thing so I Apply the state I applied something and then Maybe I need to do some cleanup. Maybe I need to do some checks It's always helpful to note it so for example if you If you if you edit the configuration file the simple that the sim obviously the first choice would be okay I have a step where I added the configuration file next step reload the configuration file Problem there is that it's gets a bit hard to judge the outcome in the end because in the end Ansible normally will show you a nice summary of what's actually what has actually happened and Ideally you want to have it in the bottom so that if nothing changed The step will not show it show up in the change stuff the simple thing is the handler where It can just say okay if that changed do whatever most Commonly used to just say okay. I just edited my web server config. So I might want to reload it to actually apply it Problem with that is that that only works with a single with single things So for example, I ran into the situation when creating containers that I basically have a loop creating the container But the problem is I can't use a handler there because I could only have I only have one one handler But if I create or if I modify like five jails, I Don't want to reach a five chains out of ten. I don't want to restart all ten jails And I don't you want to restart one that I actually touched And for that it's it's quite useful So you can Register you can say register something and then it will put the output of whatever happened in the stop Into an array and you have an array that you can can deal with That's knows that's not so much about that's not so much about Having making sure that it it always does the same but in this case it's more about Having a chance to actually check What happened? What what what was the state? So for example, you can just run it again afterwards and then then it will most likely if you if it was If you were thorough with it It will just say okay, nothing changed then, you know, okay Everything worked the first time and I don't have something that's changing all the time because I have some interaction somewhere That's actually one of the things that I've run into quite a bit or have helped other people debug Which leads back to saying okay, I already have I already have a playbook to that defines this host So we'll just run it will copy the host wires and maybe adapt some Minimal things because some ports or IP addresses and if something are different and then apply it to the next host Problem is if you As it happens developed it over time You might have you might have circular dependencies that you that are not automatically Solvable so what I do I mean with that Relatively common problem that I ran into and I've seen in other people's playbooks is Simple web server configuration so you start up say okay This is the basic configuration of a web server. That's the path where the ACME challenges lie Fine, then you run it you request your certificate wire Whatever ACME client you want to use ACME does sh dehydrated third bot personal preferences Um Then you know oh now I have the certificates. I'll add to the I'll add to my configuration data for my web server I'll just add the HDBS bindings Everything works Well now I just I duplicated the configuration for the host and I'll try to apply to a new host It will most likely fail because your web server won't configure Because it says well, yeah, but you said I should use this certificate, but it's not there and There are I mean there Yeah, I just wanted to get so I mean there are there are multiple ways to handle it so You can either you can either Make sure that you create your web server configuration in steps But then you have to be a bit careful if you run it against the already running system that it doesn't for 30 seconds Tear down your web server And interrupt all your connections One other thing that I with nowadays that I actually do with let's engroup specifically in this case is I just use the NS01 challenge Which once you have the infrastructure set up is not that much more Work and I need it for some cases anyway It's it's it's it's not it's not impossible to deal with it's just that something if you really want to have the option and depending on your environment the person writing and developing the Playbooks might not be necessarily be the same person deploying a new server You might have some junior admin or I Mean in turn Trying to create a new server the intern hopefully not the production environment, but Could be for test system as well It's just that if you think of it early enough It's easy enough to deal with as you said you just put there put the dummy certificate there, and you're fine It's just I just wanted to point out it's things like that that you have to think about And I've had that with I mean SSL certificates are relatively obvious in that area I also had it in situations where a System runs multiple so multiple services that do a degree have a dependency on each other We also have the problem that yeah, it might not be up yet for example, I've I've had to deploy Oracle databases and Then you have the problem. Yes when it starts up for a new database it will take depending on your hardware it might take a few minutes and If you immediately try to do something with it afterwards You have a problem. I mean there are ways to say okay wait Until for example, there is the option where you could say okay wait for a file to be created If you have to for I mean it's not really convenient, but sometimes it happens that you have some asynchronous stuff that you need to wait for Because somebody thought it's more convenient to have it asynchronously done But the problem is you need the result of it So you have to wait for it anyway That that that's the that that's the small things that I have to say that I just wanted to point out so Pre-prepare if you want to have the safety prepare her to deal with it and Yeah, I mean I either with adapting the system The other the other thing is actually also your environment I ran into the problem. It's I think it's probably not so much of a problem if you're if you're in in in a company with a somewhat identical setup for workstations But I also for example I'm responsible at least partly for running the infrastructure of the free software foundation Europe There we basically have a mix of some people who are employed some people and a lot of volunteers Basically, all of them are Responsible for the old for their own systems which means that you have a varied mix of Some some might run PSD In one flavor or the other multiple Linux distributions windows is probably not common in that environment, but so and and and multiple versions of Distributions they might do upgrades at different times Which means different Python versions different Ansible versions and stuff like that that does create problems Also, when you install also when you install Python packages To get some stuff running a lot of the time you have the problem that it's still some of the stuff is still system packages that are different so You can't deal with that with things like virtual environments the slightly more complex or more Bigger bigger thing would be a wx Or Ansible tower if you want to go to the full redhead solution My experiences with a wx are a bit mixed because it currently requires Kubernetes to get it running which Is it what then you have to decide is it worth it or not? Yeah Just wanted to point out it is a solution for the problem It might not be the solution that you want to run and as I said it might not be a problem because if you if you're in a company where you have Either a central a central system where you execute your playbooks because I've also seen Quite a few environments where you execute your playbooks out of a CI CD system Where you can also make sure that it will always be executed in the same environment That you can control and say okay now we want to update or Yeah, this playbook run that in this old environment, please we haven't done we haven't touched that so long It needs to run there Yeah Also Synchronizing playbooks I've had it in the past where a Colleague destroyed some work Not saying where but When they basically Edited the host configuration file and apply the playbook Not checking that there have been some serious changes to the configuration of this hosting between Which will lead to anything from It being on an older level of on an older configuration to a broken configuration whatever happens We're back I think to a degree in the procedural thing Where you say okay, please make sure before you start editing the stuff, please pull your get repositories make sure You might even have stronger rules, I mean I've we have Customer sites where you have to really make change requests and have a really complicated complicated defined procedure For any small change that you want to apply Then you don't have the problem But on the other hand for a small and by a first small company the overhead might not be worth it It might still be worth it if because I mean a small company can still deal with Things where it gets really costly if you make small mistakes Just looking at the time I'm nearly running out So Should I show some small examples or do you have any questions? Or a mix of Okay That might I might not have kept up with some Naming in the redhead space. I'm I'm I'm not a redhead customer. So they might also have renamed stuff Ansible tower is the old name For what so for the free core of which is AWX It might also be only part of that. I'm I can't completely say that it depends so This is for example, this is one of the examples I talked about when I wanted when I needed to to adapt the hardware controller Because I said well, I have a cluster of free machines I really want to have identical rate controller configuration because I don't want to deal with some strange performance differences because I Somehow said the block size wrong somewhere So it's a relatively simple example where just configure the SSDs and HDDs and Yes, I'm configuring single disk rate zeros here Reasons why the hardware was yeah, okay I've just been shown that I have run out of time so If you want to have any more examples, you will have to come to me I will be available outside