 All right, our next session is a love story about how FormanMet Ansible, Adam, take it away. Thank you, Walter, for introducing me and the talk. As Walter mentioned, it's a talk about what happened when FormanMet Ansible and where we are now. I'm Adam Ruzicica. I'm a software engineer at Red Hat. I joined Red Hat something a little bit over five years ago and since the start, I was working on Forman or related projects as plugins and so on. I'm Adam Ruzicica on GitHub, in case you want to check out my stuff. I'm Ruzicica on FreeNote, if you want to reach out to me or underscore aruz underscore on Twitter. And that was enough about me. You're not here because of me, you're here for this. So this talk is about the Forman Ansible plugin. If you're here because you thought it will be about Forman Ansible modules, I'll have to let you down. It's not about that. It's about this plugin and we'll start right away. So I assume that you have a Forman deployment without Ansible. So just a couple of steps how to get it up and running. Ansible is in fact a specialization of the remote execution plugin. So it needs it to work. You can install remote execution with this simple command. You run the Forman installer, feed it some options and it will do its magic. Then the similar thing for Ansible, just the names of the options are different. And in the end, this is what I personally like to do. It's probably not such a good idea in production environments. And what this option does, it makes the installer take the public key that Forman is using to connect to the remote machines and put it into a root's authorized keys. So you can use Forman to run remote execution or Ansible against the Forman machine itself. Again, you may or may not want to do that. Of course, you can run it all at once. You can use a slightly condensed form. It then looks like this. It takes, I don't know, five to 10 minutes to get it up and running and you're there. Okay, so you had an empty Forman. You're in the installer. It installed the Ansible plugin, but you have a blank Forman or you have one host, the Forman itself. Oh, sorry, where is it? Different tab. It may look like this. Sorry, the UI kinda breaks when you zoom in too much. But in here, I have a single host. It's the Forman instance itself. And well, you could do that. You could deploy the Forman and then use it to run Ansible against the Forman itself. But that's not the point why you have either Forman or Ansible, right? So the best option would be to let Forman do what it does best, and that is provisioning. If you provision machines for Forman, you would get them in the inventory in Forman. You can then run Ansible. When you're provisioning, it deploys the SSH keys Forman knows about to the machines so you can connect to them. And even if during the creation, if you assign roles to that host, it applies them to that host at the end of the provisioning. So you can get the machines configured fairly quickly. But every now and then, someone comes and asks, hey, Forman is cool and all, but I already have machines and I don't want to reprovision them just to get them into inventory to be able to run Ansible against them. What can I do? Luckily, there is a way. When you're in the installer, it deploys an Ansible callback, a plugin basically to Ansible, which at the end of the run, reports the facts about the machines to Forman. And when that happens, you get your hosts created in Forman. So let's try that right now. Let's see if I can remember all the right magic incantations we need to do. So first, decalback is configured only for the Forman proxy user. If you did this as route or any other user, it wouldn't work. I'm talking out of the box. You could make it work, but out of the box, you have to do it as the Forman proxy user. So we'll get another bash shell as that user. And then I have an inventory, handy, ready in here. And I can do something like Ansible playbook dash inventory, the path, and then we need to provide a playbook. I don't want to write it as a file, so maybe we can do something like this. And then we write the playbook. So we do all hosts, we give it some tasks, and we run the setup. This is important. It doesn't work with anything else. Setup is the one that triggers the callback. And that is enough. We end the playbook, we end the sub-shell or whatever it is and it runs. Permission denied. Okay, who knows why? What did I miss? Any wild ideas? Actually, two things, keys and the user. So let's jump right back and connect as a root and use the right private key. By default, it lives in user share, form and proxy, .ssh, idrsa, form and proxy. That's the private key we use. Hopefully it should be better now. We spin the wheel. We wait for a bit. But it's taking a bit of time. That's a good sign. It means it's actually doing something. Okay, here's the first one. Second one, go on. Come on. The fact is don't get reported until this all finishes. So it makes no sense checking in the UI right now with, first one. Question in the tree? Sure. If you have a proxy. Okay, so the question was, what happens if you have a form and proxy and with the keys on there? So if you're provisioning through form and, form and collects the keys from all the proxies you have and deploys them on the machines. So you can then use any of those proxies to connect them there. If you're doing something like this, then you have to pick the proxy where you know the keys are, right? Or use a password. It works too. Okay, this finished. So let's check into the UI. Let's refresh that and boom, there they are. We can now select them and we can do things with them. Back to the slides. This is what I did. And even in here, I missed the user. Sorry about that. Okay, now we have the hosts. What can we do with them? As I mentioned, the Ansible plugin is a special variant of remote execution plugin. So you can use Ansible the same way. You can just run commands or scripts against the target machines. But let's be honest, it's not what Ansible is good for. Ansible is not as good as SSH of pure SSHs, right? We have time. Funny story, I was doing some performance testing. I just wanted to know how things behave under load. And I wanted to run a simple date command on a bunch of hosts. It was this laptop. It has two cores in the hyper threading, so it's not really super beefy. And if the load goes up above 10, the machine locks up for like five minutes until the load goes down. So I went into Forman UI and why I wanted to use SSH. But I misclicked and selected Ansible instead and ran the date command on 10,000 hosts from this machine. It was back then when we had only the Ansible Playbook implementation, which spawned a single Ansible Playbook process for each and every single host. So I just clicked the button in the UI and suddenly spawned 10,000 Ansible Playbooks. The last thing I could solve was that the load went up to about 200. And the machine didn't recover. I had to kill it and start over. So if you just want to run shell scripts, why not use just Remote Execution? Maybe a better use case is actually running Ansible Playbooks. You can do that, you can go into the UI. You can either create Playbooks as job templates because in Remote Execution and Forman Ansible, what you run are actually templates which get rendered and you can provide inputs to them. So you can either write or Playbooks as templates which have no inputs or you can do what I do and that is have a single template which only thing it does is renders its single input. So basically what you give to it as an input is the thing that gets run. And then you can just go to the UI, select Custom Playbook, write a Playbook in there and click Submit and it runs. You can do that. However, there's another way how you can use Ansible. And it is more kind of configuration management type approach. Anyone is familiar with Puppet or to the other way, who doesn't know Puppet? Okay, we have several people, good. So in the Puppet world, there's the Puppet agent which runs on the remote machines and every once in a while it wakes up, calls to the server, asks, hey, what stage should I be in? Then it tries to get into that state and it reports back to the server. And this happens periodically, every, I think, 30 minutes by default. And you can mimic this behavior with Forman Ansible. You can describe the state, you want the machines be in, either by roles or just a single huge Playbook, whatever, and then you can have it run periodically. So instead of the machine, waking up every once in a while, Forman will wake every once in a while and run that against the machine. You will get fresh facts in Forman, you will be sure that the machines are in the state, you want them to be. And that's a perfectly viable approach, if you ask me. Several times I mentioned that you assign Ansible roles to the hosts. But with the roles, it's the same problem as with hosts. In order to have Forman to be able to do something with them, it needs to know about them. How do we do that? Luckily, it's simple, it's much simpler than with the hosts. You can just import them. They have to be by default in slashetc slash ansible slash roles. You just put them in there and then you go into configure Ansible roles and there will be a button that says import from and the host name. And you click that, you will get a list, you can select which ones you want to import and you will get them to Forman. Once you have them in Forman, you can assign them to hosts. And then they will be around either when the host is provisioned or when you kick off, it's called run all Ansible roles, I think. And that's what will happen. It will use those roles as the source of truth, basically, and will apply them to the hosts. What we do in addition is that we scan those roles for variables. So we look at the defaults and I think even vars from those roles and we try to guess what type they should be. So if the role provides some defaults, Forman can take a look, see okay, it's a string right now and the value is something. And then in Forman, you can see that this role has these variables which can change how the role behaves. And you can see that it's a string and you can override it there. You can either override globally or you can create matchers to say if the host has operating system, I don't know, CentOS, then this value should be XYZ. Alternatively, you can set those as parameters, directly on the hosts that works as well in host groups and even globally. So there are several things how you can do the same thing. Okay, I was talking for way too long. Let's see what I talked about. So we already have the hosts. Let's import some roles so we can sign them. So it's configure, the Ansible section and roles and see there's a big button. Click it and it will list us which roles exist on the machine. Most of these come from the Forklift project. It's something we have for development of Forman. It allows you to spin up Forman instances using Vagrant. But it uses Ansible roles to do that. So we can import everything, why not? Let's update and now we have them in Forman. It's a two-step process. Right now we only have the roles, so we may want to import the variables as well. And it's again, just click the button, select the ones you want. I want all the new ones. Too many of them and click update and it imports them. Now what I will show you, I will deploy CJDNS. It's a project for mesh networking. It can either act as an overlay network or it can act on the level two or later two. And in that case, it doesn't need any configuration which is ideal and I can just switch a toggle in here and it will be fine. So this is one of the variable. It's set as a default in the Ansible CJDNS role and we can see that it's a default and by default it's false. I don't want that, I want it to be true. So I just override it here and that's it. Now if I look at the variable, I can see a flag that it has some override configured. So it gives a visual indication that there is something non-default. Then I go to the hosts and I assign the roles to them. We go to edit, then there's the Ansible roles tab. Yeah, and we have to find them in this list. There are somewhere at the end, small screen size. Okay, it's this one, I hope. Ansible CJ, yo. Okay, so we submit it and it's assigned to the host. I could run this on the single one but let's do the CJDNS one as well just so we can see that it actually works. There, and we submit it. Then we can go to hosts, select the first two and have them run all Ansible roles. It will take the roles we assigned and run them against the hosts. This is current nightly setup. I must admit we had some issues with the JavaScript stack recently. So there should be chart right here which isn't currently there. And everything failed, okay, that's new. Failed to connect, could not resolve host name. Okay, we'll use poorman's DNS and it is ETC hosts. I'll just, and I only need to do the first two so it won't take that long. Okay, so this was a fun little exercise. I can now kick off the job again. Send settings, and have it run. Hopefully this time it will be green. While it runs, before I start about it, do we have any current form in the host? Any current form in Ansible users in here? Okay, one, two, someone there, awesome. I have some slides on what changed recently. And that is, we dropped support for the original Ansible playbook thing. So now it's only using Ansible runner. It also lifted the one process per one host requirement. So that should be better. You should now be able to use Ansible as an orchestrator, basically. And we reworked the way how we generate inventory for external services. So if you use Tower or AWX to pull the inventory from Foreman, it should be faster. And we started deploying Dimeflow. That's the internal orchestration engine or task engine we use to use Sidekick under the hood. So it should perform better, be more scalable and more reliable. That's for the news. Let's take a look back. And we can see that it is still running. All the role does is it enables EPL. It installs a package. It generates configuration and it enables the service. So it shouldn't take this long, but it's still running. We can use this time to do a little questions and answers session. Is there anything you would like me to talk about? So the question was whether we can change the variables for different hosts or host groups. Is that correct? Okay, you can. Either you have to specify the matchers. So you go to Ansible variables and then at the bottom of the page, you can specify matchers which match on different parts of the host. So you can do it either this way or you can go to the host group or to the host and edit the parameters of the host directly. Then that value will be used too. Okay, it's moving, it's moving. It performed the automatic configuration. It restarted the service. Awesome, so this is done. We can see that all the hosts completed successfully. CJDNS creates a new interface on the host. But if we go here into Nix, we can't see it. And that's because the facts about the host are collected at the beginning of the run. So the facts got collected and then the interface appeared. So we can't see it. However, if we run it again, it should run much faster now and it should give us the right facts. So this is something just to keep in mind. We see that now it's blazing fast, almost. And it is done, oops. And if we go back and to those details and into Nix, we can see that there is ton zero interface with MTU of 1300 and something and it has an IPv6 address. Go on. I think the plan is to keep Cthulhu as a plugin and not really merge them to one. Forman is fairly modular. You have the core which does provisioning and there are plugins for almost everything under the sun. So yeah, I think it will stay this way. All right, we are almost out of time. So one or two last questions, please. Can you select which smart proxy is used to do the NSPOR on if you're having separate networks? Okay, the question was if you can select the smart proxy which is used for running the job against the host. You can select it directly by going into the form and saying, hey, use this one. But you can configure your organizations, locations, subnets and things like that to perform the selection for you the way you want. So it's doable in a kind of roundabout way. Yes, please. So the question was if the configuration of which hosts, which roles are assigned to which host is something that could be version controlable, right? Yes. Not currently, I'm afraid. But it sounds like a reasonable thing. So please open an RFE for us and we can see what we can do about it. Please, if we can use it together with Puppet, well, you can. There's nothing stopping you from doing that. They live completely side by side and it's up to you what you use. Okay, it looks like there are no more questions. So thank you very much for listening. Please stop by the Forman booth and we can talk more. Thank you.