 My name is Mike DiPaolo. So I'm best known within the open source community for my roles on the X2Go project and the Fedora project. X2Go is a remote desktop solution, and I've been contributing multiple components there such as desktop environment integration. I've also been the primary maintainer of the Windows client. What? Louder. I've also an X2Go product which is a remote desktop solution. The two biggest things I've done is desktop environment integration, and maintaining the Windows client. Because although X2Go is a server, X2Go remote desktop solution is only available for Linux, the client is available for Windows, Linux and Mac. On the Fedora project, I've been a packageer and an ambassador. This talk is based on my experience as a system administrator, which I've been doing for years, and recently have started using Ansible for. There will be a demonstration of Ansible running about one third of the way through the presentation with some of the code that's on screen. The example code here is, some of it's not specific to any Linux distro, much of it's specific to Fedora or specific to CentOS 7 slash VEL7. I'll begin with the explanation of why this presentation is needed. So Ansible is typically used to configure servers or an entire data center for servers that's in all terms is configuration management. But typically in supported servers, you're very different paradigms like users aren't going to physically interact with the computer by pressing the power button for example, and you're running a single application on the server, and the entire configuration of the server support that one application. Often there's developer desktops to configure via Ansible, but there are typically just in support of the application. Whereas you're managing a fleet of Linux desktops, you have to worry about setting things like screen locks, a common security policy that only really applies to desktops, not the servers. You have to worry about things like recommended settings for users, you want to set them to help them, versus mandatory settings for users, you have to set for them. You'll be configuring very different parts of the OS2 for the desktops, and these desktops often run like a hundred different applications. If you just look at the start menu, you may see all the applications there, and as a system, you're often configuring settings for probably dozens of those applications in the start menu. So, I'll begin with an overview of Ansible for configuration management. So, whether you're configuring desktops or servers, like 20 years ago, and more like 10 years ago, if you were configuring parts of the OS, you're setting settings config files, and maybe restarting services based on those settings you send config files, and the way you would do this is bash scripts. You would not just set in your bash scripts, well it's a general purpose programming language, and you would not just set, how do I set these settings, but also, so you're not to set what settings I want to set and when to set them, but also how to set them. So, if you run your bash script against the machine two times in a row, it might put the same value twice, and that could be a syntax error. So, Ansible was developed as a better solution for system administrators where instead of writing in bash which is close to a full-featured programming language, you write in a simple domain-specific language. My current job, we actually have technicians starting to write Ansible code because it's somewhat simpler and it's almost syntax. Instead of writing logic like, instead of the setting is set, if it's set to the wrong value modified, if it's not set at all at it, instead you simply declare the settings that you want in those config files, like a name and a value, and ensures that the setting is already set. That's called declarative code. As you're writing Ansible, essentially you're calling it called modules. These modules are written in Python. They take care of how to set the settings you want to set for you. I have examples here like a specific one would be sysctl for kernel values, and a generic one would be line in file, and a happy medium between those two would be something like iini file. You call these modules as tasks, and the tasks go into playbooks, and for the proper structure of your overall code base, those playbooks goes into roles. You can write your own roles or you can use roles to write other people and just pass variables to them. With Ansible, you have a management node, which could be your own laptop or desktop, or it could be Ansible Tower or AWX, or there's also a third-party solution too like that. Then the managed systems are reached via ssh is no agent on the system. Here's an example of a task. The task is calling the module called iini file, and this demonstrates the first topic that lots of OS components, especially desktop OS components, can be configured to be the iini file module. Does anybody want to explain what this task does exactly? Okay, so one thing we discovered at work is that people they want to reboot the computer, they want to reboot it instantly. They're not willing to wait even for disk to sync, and if NFS is hanging, well, the disks aren't going to sync quickly enough, and they're just going to hold down the power button. SystemD has a feature called the control-alt-delete-burst-action, which is if you press control-alt-delete seven times in two seconds, it does a special action. By default, the value is reboot-desk-force, which still tries to flush to disks. So a setting we often set is reboot-immediates, which it's almost like an alternative to holding down the power button and pressing the power button again. So as we're calling the iini file, it's saying this is the section of the file, this is the name called the option, this is the value reboot-immediates, and I often specify new extra spaces because most config files on the system don't put extra spaces between the name and the value, although it's typically superficial and they're ignored. Another thing too is typically in the Ansible tasks, I put the task name would be like what I'm doing, but to make the example interesting, because this is a really annoying problem at work, I just listed why I'm doing it in the name, and typically why I would be in a comment. So lots of desktop OS components are configured via the iini file, can be configured by the iini file module. GDM's config file, which in a corporate environment, you often have lots of requirements for what the login screen should look like, well that's best configured by the iini file module. KD's config files are also iini file modules, and Nome uses the DECOM framework to store settings, and typically in your home directory, it's stored in a binary database, but the system-wide settings that the distro can provide, and that use the system it can set are set as plain text files which are iini files. So I recommend using iini file module for these DECOM settings. Now there is an Ansible DECOM module, but that only modifies the currently running session. So if Ansible is logged into a system as root, it's not going to see a running session, the DECOM module will fail. The DECOM module is only useful if you're modifying and usually use the session. But it's a system and you're typically saying, multiple people could be using this desktop, I want to configure the setting for everybody on that desktop. So that's why I recommend using the iini file module and we'll be showing that in a second. So this is a common example with a setting that you want to set an enterprise environment, which is to set the background of a computer. This is especially commonly desired in an educational environment and I'll explain why in a second. So what it's doing here, it's saying there's an iini file, I'm specifying the path to it on disk of the managed system and the section name, which is sort of like a folder, is a GNOME desktop background. The exact option is picture URI and the value is a file on disk. Now, you may notice at the end there that I have a handler to generate the DECOM database. You do need this when you're modifying the system-wide DECOM settings. Because although you're supposed to write all your settings as the iini files modules initially, they need to be joined to a database so that all the user accession will actually read them. All you have to do is just run the command DECOM update and technically the command module would be better than running the shell module, although the shell module I've been doing in my current job for this. I have a simple check to see if it failed to generate the DECOM database. Now, here's the funny part is in an educational environment that the background and the screensaver is often very important to set mandatory. Otherwise, people will set their background or screensaver to the text that's not appropriate especially like the principle sucks. This actually was a problem at my first internship I did at my own high school where they love the fact that Microsoft committed group policy when it was 2000 and therefore they could set mandatory background to mandatory screensavers. Before they had the ability to set mandatory settings via Windows mechanism, they had a problem all the time. So what this does here is it says here's a DECOM plain text file and this says that the sign we set in the other file is mandatory. It cannot modify it and it calls the same handler. So at this point, I'm going to demonstrate that task. There's two tasks running. Oh, we've sent you to do this on this. Sorry, that's- here's a simple ansible playbook. It has some skeleton lines like saying elevate to boot become and there's the two tasks that have showed you along with the handler. It's an ansible playbook. I'm sorry? Oh, yes, certainly. Certainly. Yep. So there's the skeleton code up there and then there's the two tasks in the handler to show to you. So ansible, I'm going to tell you on the playbook and say to prompt for my password and I'm going to run against my own laptop right now. So for demonstration purposes, I'm going to maximize this and there you go. Because DECOM background is set and it's the ansible logo. Because DECOM runs as a daemon that runs under the user's session and that daemons can figure to look at both the system database for mandatory settings as well as in default settings and listen to your own settings in your own home directory. The fact that we updated the system database means DECOM was immediately aware of it and we applied the change to the running session. Can I slide back up? Okay. Now, I'm going to- this third example is setting some similar settings under KD. Instead of doing the entire- the background would similarly be a single setting, but one setting you often have to set hopefully of desktops is the screensaver settings. So many corporations have requirements like, oh, you must lock the screen after 10 minutes or after 20 minutes. This example locks the screen after 20 minutes. It brings up the screen say after 10 minutes and then applies the locked 10 minutes later. Just like before, I'm doing the i9 file module. KD does have these exact same config files under your home directory and you're welcome to generate them under your home directory and copy them to the system-wide location. And do you know what I'm doing here is I'm saying, you know, this is a file and these are the names and value pairs and that's- but in the name, I have the bracket $i. That is says immutable. It's basically where you're saying this is a mandatory setting. Not simply setting the default setting for your users. You're setting it to be a mandatory setting. I would point out an interesting piece of experience that often users hate the screen savers in the screen lock, well, not the screen, but the screen locks so much and they'll often look for ways to defeat it. They'll often look for third-party apps to jiggle the mouse, for example, and simulate mouse inputs for the X-11. I'm hoping Weyland will address this, but in the meantime, this is one you will often have to- You may find that like 5% users are finding ways to defeat this and you'll have to look for organizational ways of telling them to stop trying to defeat the screen lock policy or look into ways of killing those processes that would defeat it. So, another desktops of differences. So on servers, you know, if you're defining who can call these elevated, who can run privilege commands you're gonna do it via sudo. And sudo has its own well-known syntax. But graphical apps try not to use sudo or anything like, because sudo runs the entire program, the entire binary in GUI as root. So what many graph graphs do nowadays is they have the main graph, the GUI binary, the main binary running as the user account and they call special helper binaries to do the specific actions as root. The best example of this is GNOME software. And the way you configure policies for who can install software is, or who can update software that's already installed is done via policy kit. The best way to configure a policy kit is just to copy the config files from, you know, that you wanna install the config files as static or generate the config files according to templates. There's no specific like module for configuring them right now, like i9 file or deconf. In this specific example, I'm copying some policy kit files to handle the fact that if your users are actually in their desktop remotely, like over Tiger, VMC or X2Go, that they'll receive prompts on login. These prompts, the policy config is, oh, if you're logged in, if you're logged in locally, you can modify the caller schema. If you're logged in remotely, you have to need administrator credentials to modify the caller schema. And if the users don't go to modify the caller schema, they still receive the prompt at login to I need privileges to modify the caller schema. So here I'm just, as an example, I'm just copying some policy kit files, rules files to replace the system right rules and I'm gonna use a system default when there's less USRs, so that users can modify the caller schema and they don't receive the prompt to do that. So another big difference between desktops and servers is users will often run like a hundred of applications. If you have a large enough set of users and therefore a large enough set of applications to use the desired, it could basically be about a hundred of them. So as opposed to servers, where the server is supposed to run only one application, so the section on my talk is about tips and tricks for speeding up how you install all those packages and maintainably installing those large packages also. If you're on REL7 or CentOS 7, I strongly suggest that you install YUM4, which is the compatibility name basically for DNF. If you can then install a package with a YUM module and it'll call the YUM4 DNF under the hood room present. If you really wanted to use the legacy YUM, you could still just, that's just an option to the YUM module now in Ansible. This is as of like the latest version of Ansible that they made this change of YUM actually calling YUM4 DNF. And the reason why DNF is so much faster, lots of people know that DNF is faster than YUM that installing packages begin with like evaluating dependencies for example. This is true, but there's a more important reason why you wanna use DNF with Ansible, which is that Ansible doesn't trust YUM to check where the package is installed and only install it if it's not installed. So instead, the YUM module consults the RPM database, I think it calls librpm or something to see if the package is installed and then tell YUM to install the ones that are not installed. This results, this severely slows down how long it takes to install packages. It basically makes it take about twice as long. When you're using DNF, it's calling the DNF stable API and that's, and therefore you have basically almost no overhead. It's Ansible DNF module works about as fast as the Ansible, as DNF on the command line does. So another trick for installing a large number of packages is to use the package groups. These are, you know, the distruder developers have decided that, you know, these are the packages that should be, if somebody wants to know them, they have this complete set of packages and that's the workstation product environment. And if they want KDE, they want this complete set of packages and that's the KDE desktop environment. If you want a specific group of like, commonly press applications like text editors, well, install the editors group or if you want Ansible, install the Ansible node group because they might install things like Ansible doc, for example, or some Ansible like test tools. This will speed up, this is one reason why this is simply to speed up the presentation time and simplify the amount of code that you specify Ansible playbooks. The other reason too is that, you know, like it said in software engineering that premature optimization is the duty of all evil. Well, if you have a tightly controlled desktop environment with countless systems like a small number of apps, you can slim down the number of packages you install aggressively. But if you try to slim down packages without making sure it's safe to slim down the number of packages you install, you will notice you will run an issue of under clear dependencies. A simple example of under clear dependencies is fonts. Like this application looks best with this one font, but it's not installed, so we sort of use another one. And in general, like, you know, Fedora and Rallin sent to us can get better handling under clear dependencies because there's no weak dependencies, but you will see lots of weird behaviors if certain common desktop dependencies are not installed. An example, one of our remote system running another Linux distro, like certain graph gaps are running, but then they're like the content like the public dialogs were not rendered because some under clear dependency was not installed. It was very frustrating to have to debug that and get the organization to install that one under clear dependency. But if they had just installed the entire KD or known package group that under clear dependency probably would have been installed. And just to clarify, so you have to use special syntax in the Ansible YAM or DNF module for package groups versus environment groups versus regular package groups. This is the same as if you had installed on the command line with like DNF install rather than DNF group install. Another tip that I have too is, say you have like a list of about 100 packages you want to install. There's Ansible, you could keep that list of, or maybe it's been a thousand packages if you, depending on an organization. So you may have that those packages, you could put them all in a giant list in your YAML file. What many people want to keep in an external source like a CSV file or a text file with comments. So what I've been doing is I've been putting the list of packages to install in a text file with comments. And I just, I'm using Ansible lookup plugin to call a command. And what the command does is it just strips out all those comments. So this YAML, this task will just install the list of files from that text file one line for line. Each line is different package name. This other very good lookup plugins too, like see it like, I think this one's to use spread sheets and like actual sources of information over the network. Okay, so six topics of package could offline updates. So package could offline updates are a mechanism for installing packages at the time of system as we booted rather than while the system is running. This was developed based on some painful experiences that like for the world developers experience. I think the example they had was that if you update the Mozilla Firefox or Flash plugin package while those are running, Firefox or Flash like refused to work with each other until they restarted. So there's other examples of packages that don't update it while the system is running too. Like they proprietary invigorator driver, which is commonly distributed as RPMs for third parties. Well, the invigorator kernel module says, oh, user space has been updated, but you haven't been booting the new kernel yet. I'm gonna refuse to do any 3D acceleration. I refuse to do 3D acceleration. It won't even launch NomeShell. So therefore what I've been doing is I've been updating any packages that are disruptive to update while the system is running via package could offline updates. And package could offline updates is in well seven now. So first you would do is I would just young update all the packages that are safe to update while the system is running. And I would just apply a filter by package name or by the package that I want to update or I just disable those third party repos that have those packages. And then I call, first I call package could offline update, package could offline update command to just download the package, put some under slash bar. And I have to wrap around the return codes of the command I'm calling. And it's basically, you know, you know, like zero means successfully downloaded them, five means I have nothing more to do because it's already downloaded. And I put a comment there explaining what that five code return code exactly means. And then you have to, after this, you would call another Ansible task to trigger the offline updates so that it actually will apply the offline updates at reboot and that minimal system reboot environment for installing updates. And the idea is that you basically retell users, you must, you know, we've applied the updates to your system to finish applying the updates unless we boot by this date, like in the month or two weeks from now. And so if they don't reboot, we still reboot the workstations from anyway, but. At least this way, we don't have to worry about their, oh, many apps refusing to launch because 3D acceleration isn't working though. And this is, as you can see, the logic here is pretty lengthy and it's kind of messy to wrap around all of these. This is, I started working on an Ansible module for package could offline updates, but I have not completed it yet. I'm just dealing with nuances of the D-Bus API for package could offline updates. Another difference between desktop and servers is how you handle firm updates. With servers, typically, the server vendor has a BMC, a baseboard management controller, like a Dell iDRAC or HP ILO integrated lights out. And these are special, like single board computers or chips on the motherboard that enable you this, they enable system into a lot of things, like we boot the computer if the OS is not responding and monitor its fan and temperature speeds, and they often have their firmware update features also. And firmware updates often necessary because of the security updates on them too. So typically we'd call the BMC command or APIs or whatever to update the firmware on a server. But on desktops for years, there's never a good way of applying the firmware updates to them because they didn't have the BMCs. And so fortunately, people of all links community have been working hard on the mechanism to update firmware from the links OS and it's basically the same UEFI feature that Windows uses called UEFI Firmware Capsule Updates. That's the commands for this or the tools for this are frapped FWUPD and the websites that hosts all the firmware updates uploaded by the system vendors themselves is called LVFS, Linux Firmware Services. So the two biggest system vendors that are uploading their firmware to LVFS that can be applied this way are Dell and Lenovo. It's basically any three or four year old laptop or desktop or newer will support LVFS. So this is something I also like to develop a module for but calling the commands works pretty well. It's not, it would be a bunch of logic like if you want to run, I'm just going to show you the commands you would wrap with Ansible right now. It's, you just call it firmware update manager install. Well if you want to, if you want to just get the latest version and get it from LVFS directly is to call that simple command on the bottom, so latest version. If you want to control the version that you want installing the workstations like say, Dell just released a new update but you want to test it yet because you always have to test the updates before you update a thousand users. Well then you would, I recommend you specify it as Ansible defaults which are like variables saying this is the first one I want. Here's the URL for it and do that for this model. If the model, if the desktop model is such and such as an Ansible fact of the computer model then install this version from this file. That's the end of my presentation. Any questions? Yes? Yeah, I know. Let's start configuration management on the server side. It's changing very rapidly and we support about a 24 hour window. So if the server goes offline for 24 hours it may not be able to be converged to a state so we're always assuming that we catch it. Laptops will offline for a week you can do all this sort of stuff and they go offline. So we run in problems where IT is trying to use all the production and configuration but our production and configuration makes assumptions that people will be making changes they'll be massaging it for a day changing it four or five times making sure that installations go properly and then laptops are ready to do it and they like to trip all the way down. So what do you have in terms of experience or tips for people that already have a an existing server fleet that's managed and trying to bootstrap that onto the laptop side? Okay, so I mean... Yeah, so... It's also more obvious. Yeah. So he said at his organization they have a fleet of servers and they have an Ansible code to find a configuration. They want to manage laptops also. However, if the servers are powered off for more than 24 hours it's assumed that they may be two four out of date for the new current Ansible logic successfully applied to it. So servers 24 hours is pretty reasonable cut off but for laptops, well, they might be offline for weeks or even a month because people take them home when it's powered them off. So, am I current workplace? We're implying a high standard of like, the Ansible code you write must... You know, first of all, we have basically patching appointments with users every single month and they have to be there for the patching appointment. That's true for Windows and NAC as well as Linux. We don't have Linux laptops yet. That's something I want to work on. But the idea is we would have something like, the state must be... We typically support replacing the old system configuration or updating the old logic on the system and for about like two or three months like, you know, say we add it, we no longer install this one config file on the system. We want it to be the system of the fourth version that config file. We'd leave that Ansible code base for two or three months. That way, we give us the chances of some system being too far behind that we have to re-provision it with the current Ansible logic is very much minimized. Does that answer your question? And there are tips and tricks. Oh, so there are lots of tricks on how to like, use the same Ansible task for both configuration updates as well as new provisions with that configuration. There's a talk at Ansible fast about how to do that and like, you know, you can use, you can use like, you can differentiate update versus provision with tags. And if you write proper like a different end code and like clean up tasks and that's, it should be able to handle that well enough. It's just more work. So does that answer your question well enough? Any other questions? In the example, number six, so there are a fair amount of bitters here. There you have a lot of bits and a whole bit of say, it's not equal to five. And right after that, it's not equal to zero. So there are these tasks for less. So yeah, I'm saying that any return code other than zero or five is an error of return code. Not just, now, there is a better Ansible syntax that we basically say if download, the RC is not in the list of zero or five. I should have just done that, but I just, I'm sorry, let's wait for the door to close and I'll ask you to repeat that. I'm sorry, I'm still not understanding this. If the return code is zero. Yes. Then the first check will fail, it's not five. So it will fail in any instance, zero, one, two, three, four, five, it will fail in all cases. So this is an, it might be a bug in my code, that's true, but I thought this is an and. It only fails when both of those conditions. Yeah, sorry. All right, the problem is it probably is a bug in my code. I was not, I didn't copy my Jack code that I'm using in my current job until I tried to recreate it at home and I was doing it pretty quickly. But the point is the two acceptable return and typically I would do like a list like when the RC is, it's, it fails when the RC is non-list of zero or five and there is a syntax for doing the unanswerable. Other questions? I think that's it. Thank you. Thank you. Thank you.