 Thank you for attending. My presentation will be about configuration management for your configuration management. Slightly going through the contents of this talk. And yeah, welcome FASTDEM 2019. This is the first time for me attending FASTDEM and also the first time for me hosting a talk. So slightly nervous, but that should be fine. Thanks man, that's cool. So I'll start with who am I? Yeah, I'm born in Enschede. I mentioned Counter-Strike 1.5 because that's when I kind of started with internet and found ISE. Worked for Shark Media, MSP for a while. Then I went to work for Takeaway or pizza.ba in Belgium, I believe. Obviously I listen to a lot of hip-hop music, collect music, vinyl cassettes, everything. Like true crime, architecture, interior design. I like to visit concerts if I can. And obviously IT. Well now I get paid for IT. It's not really a hobby any longer, but that's another story. Via OpenVAT I kind of got into Linux. I had a shitty router at home. We had optic fiber and strict nut with a console. It's always a little bit shitty. And I found OpenVAT and that was pretty amazing. It's kind of a rabbit hole that led me to me being here right now. Thanks to OpenVAT, I guess. Last year I moved to The Hague for a new challenge. Now I'm working for Olin Data. And the guys here at the door and some on that side are also volunteers for FASTDEM. So, yeah, slightly of a slight intro. Yeah, what's Olin Data? It's a consulting agency with a heavy focus on free open source software, DevOps. Yeah, sorry for the term. We use AWS a lot. Did some puppet training in the past. Like to work with Terraform when we can. And yeah, like I mentioned, we host trainings every now and then. So now the contents for my talk. The contents are, yeah, the subject is obviously configuration management for your configuration management. What that means is a lot of people are familiar with tools like Puppet, Solp or Ansible. And in the case of Puppet, you obviously need a Puppet server to work with that. And what I would like to show you is how I'm using Puppet or Ansible to deploy my core services to work with these stuff. And a little bit about the scope. So far this has been a hobby project of mine. I've been using configuration management for quite some time. And the scope is really limited to I won't go into security, for instance, about SA Linux. And I won't go into all of the details about the tools that I'll mention because it just takes a lot to cover. Then I'll talk a little bit about the setup and how we can verify that everything is working. And finally, with all this in place, what are the next options we can look at? So I want to talk about the problem first. Yeah, a couple of topics here. And the first problem is Puppet. Well, Puppet is pretty awesome. You have a lot of modules available to work with various systems and various applications. Yeah, Puppet is actually used to manage applications and using a Puppet agent which runs every so often to make sure that your configuration is still working fine. Yeah, the next quote also kind of applies to a server of conflict management or servers in general. Like, often servers are not completely managed or just parts of it are managed through Puppet. And in my example, the Puppet servers are often set up manually. Well, Puppet servers often grow with business. If you have a new location you want to deploy infrastructure or you need to set up a new branch office for your company, you often need another Puppet server. Puppet servers scale horizontally and vertically. You can add more resources to your Puppet masters to make sure that they are able to still compile the catalogs for your agents. But that has a limit, obviously, and then you can scale horizontally by deploying more Puppet masters, which then turns into like one at the top for your CA, for the root CA. And then additional Puppet servers which handle compiling of the catalog. So you can deploy, yeah, compile masters on different parts of your infrastructure. And Puppet server does not support HA. It actually mentions that the high availability or the failover of a Puppet server should be managed by the furtherization layer. For instance, if you run your Puppet server on over it, you are able to configure that it automatically fails over to another machine. And this is the prime example of Puppet, I guess. You can easily manage an application like NTP on your servers by simply applying this. And this way, yeah, your NTP servers are pointing to whatever FQDNs you specify. We can also manage quite complex applications using Puppet, for instance, Kubernetes. There's a Kubernetes module from Puppet itself. And it has a lot of same default values and simply applying these manifests to either your control nodes or your worker nodes, you are pretty much able to deploy a complete Kubernetes cluster. Yeah, then I want to talk about Forman. Forman is a life cycle management tool. It allows you to manage virtual and physical hosts, but also containers, for instance. You are able to deploy, a forman itself is pretty much your centralized web interface or your management console. And it is able to connect to smart proxies, which you can deploy either on your own side or on another environment. And using a smart proxy and forman, you are able to manage your DNS, DHCP reservation releases, your TFTP and PXE environment, and etc. There are a number of add-ons which I won't cover. Through Forman, you are able to specify a so-called external node classifier, which is kind of like a script that should supply you with a YAML file, and this is able to merge with your Puppet modules to simply apply configuration to your environment. It's also very useful to manage host statuses, is a node on and off or has it checked in recently to apply its latest Puppet changes. And Forman is also able to do a lot of reporting for basically the patches that are available to your environment if you use Enterprise Linux. Like I said, the statuses and you are also able to create a lot of custom templates for your reporting. Forman itself has a web interface. It also has a CLI tool which is called Hammer, Hammer and Forman, I guess that makes sense. And there is a lot of, you can manage a lot of stuff, a lot of resources through the API. And this year will be the 10th anniversary of Forman, and they have a lot of commits on their repositories. So I want to quickly show what Forman looks like before going to the next slide. And let's see, all right, over here. So this is Forman, and this is the installation, partially a default installation with a plug-in which is called Catella. And this is just the Forman dashboard, and it is able to show you a lot of statuses of your host, so-called host collections. I will get into that later. Warnings about tasks, task statuses, if Puppet is applied or not, if new host has been deployed, et cetera. And you are able to simply create new hosts through, yeah, kind of like a wizard for deploying a new machine. And if you create a host group, you are able to set a lot of default values for hosts that you want to spawn up. And, yeah, you can manage the virtual machine. The resources, you will allocate to it the disks or add additional disks. And are there useful... Let's see if I can find it. You then connect it to a so-called compute resources. In my example, I'm using LibFit. And by pressing the create button at the end of the create host wizard, you will be able to launch a virtual machine on your compute resource. There are a number of compute resources of, yeah, add-ons of native support is available, for instance, for VMware or FIAT. You can also connect it to AWS and GCP. But for my slide, I will... Yeah, from a presentation, I will keep using LibFit. And by the way, Forman has a stand in the keyhole. So if you want to learn some more about Forman or talk to the developers, they are, yeah, in the key building. And next up is Catelo. Oh, sorry, space. Yeah, so Catelo is classified as an add-on for the Forman. Catelo handles so-called content management. And what content management means is that it is able to manage your... kind of acts like a proxy for your upstream repositories. For example, you are able to add an upstream repository from Elasticsearch into Catelo. And if you are able... if you sync that repository, Catelo will host those... if you configure it. Catelo will host those packages locally for you. So it's kind of like a proxy. It handles so-called subscriptions. If you're familiar with Red Hat, I will be mostly using CentOS instead of Red Hat. You are able to import your... Hey, thanks, man. You are able to import your subscriptions from Red Hat directly into Catelo. And Red Hat also offers products. And you are also able to configure products or custom products in Catelo. And a product is kind of like an umbrella for multiple repositories. And some repositories are signed with GPG keys. You can also add them to Catelo and assign them to your repository. Yeah, you can also just install signed packages. It handles the sync management for you. So when you configure a basic repository on multiple ones, you can create a so-called sync plan, which is able to say that, let's say, every once a week you want to sync all your repositories or do it daily. And it has the concept of so-called lifecycle environments, which means that you have... Let's say you have... At the base, you have your library. Your library contains all your repositories. And based on that library, you create a new lifecycle environment, for instance, test. And then you are able to promote the packages that are in your library to the test environment. You can also configure another environment, for instance, production, and then promote packages or snapshots from your test environment to your production environment. And that way you are able to kind of pin the versions and packages you are offering to your hosts that will use Catelo. And then, yeah, that is done by so-called content views. If you have multiple... If you have one repository configured in a product and you create a content view based on that, it contains those specific packages that we're seeing at that moment. And if you create... If you have, like, a new sync later, which adds newer versions of your packages, those will not directly be part of your content view that you've published. You will be able to... You have to create a new content view. So the packages that you offer to your hosts are pinned to a specific version of your content view. And which is also on the next part. And then it has activation keys. The activation key is tied to a subscription or a lifecycle environment. And by the activation key, you are able to, yeah, pretty much set up your YUM or I think you can also use it with Shizhi, but I haven't tried that. And then it is able to access those packages in your specific content view from your specific environment. Next up is OpenSCAP. OpenSCAP is a tool for assessment. And, well, OpenSCAP is a tool for compliance and assessment. Sorry. So with OpenSCAP, you are able to... Yeah, it's OpenSCAP's tool as well, and they supply other files, which you can use OpenSCAP for to... Really? Is it that fast? Fuck me. Oh, I didn't have like 40 slides left. Sorry. Yeah, so I'll go fast then. Okay. So as you can see, there are a lot of parts for Catello that can make your life a little bit miserable. And, obviously, there is a solution. The solution is trying to apply your configuration management to systems like Forman, Puppet Service, and Catello. And at the very base, you can do something as simple as this. You can create a bash script, which installs the Forman puppet module, and then cut to a file, yeah, class puppet server true, apply that file, and you get a puppet server. So by using this, you are able to spawn up or bootstrap your puppet servers pretty easily. And, yeah, you can also use the Forman installer, which is part of, yeah, the default installation of Forman is just a regular executable on your file system. And you can specify different scenarios to either manage Forman or Catello. There's also Forman forklift, which uses Ansible. And finally, there is Forman Ansible modules, which you can use together with Ansible to manage those different parts of Catello, which I mentioned previously. So there are a number of tools. You always have your chicken egg problem. If you want to spawn up a new system, yeah, you have to specify either your configuration through a bootstrap file or you want to use the Forman installer. And you can install all of those packages directly in, like, a custom image or just use whatever image is available and bootstrap that using, yeah, a bash script, basically. And the point about this is delegation. In my example, I'll be using cloud in it for the base installation. Then I'll hand it off to Forman, to the Forman installer, later to Puppet, and then Ansible, yeah, Forman Ansible modules for the setup of Catello. So I'll, yeah, use cloud in it on LXD. Cloud in it is also used for AWS a lot and a lot of cloud providers. You are able to specify a YAML snippet to manage, yeah, your server and it is able to run certain commands on your first boot, for example. I'm using LXD because it gives me containers with a pit one so that it's easy and I don't have to use SupervisorD or something like that. Yeah, you can create a proof of concept pretty quickly and I saw that, yeah, I used to run, like, over it with five nodes at home because a lot of money and power so I tried to consolidate that into, yeah, containers. So these are some simple examples of what cloud in it allows you to do. The top one allows you to set up, yeah, simply your host name, your FQDN and then manage the contents of your ETC host file. The other one is a network configuration and allows you to set up, yeah, a bridge interface pretty easily. And then, yeah, Foreman, like I said, Foreman is a, Foreman and Catello basically are provisioning tools and handle life cycle management of your host and Catello itself uses a pulp for the content management and candlepin for subscriptions and they also have stands next to the Catello one in building K and this is kind of like a diagram of what Foreman does so there is your base web interface at the middle, it connects to different compute resources and those smart proxies on top allow you to set up, yeah, DHCP or DNS. And then there is Ansible, Ansible is config management which uses SSH so it's agentless. There are many applications that are also supported by Ansible using plugins and one of them is Foreman Ansible modules and Foreman Ansible modules manage all those different parts of Catello and, yeah, in my example I'm trying to use everything through Git and merge requests so that makes it easily accessible for a team effort. There's a strict workflow by making sure that you have different approvers for your config and finally, yeah, Foreman is the web interface or the API, there's Hammer which is a CLI tool. Yeah, then there's Litfit which I'll be using for my virtual machines. In my demo it's quite insecure but if you would use something like Ofit, Proxmox, you are able to just set it up with SSL off and et cetera. Sorry, I'll try to keep it in five minutes otherwise, yeah, the rest of my slide are obviously publicist. So there's OpenSCAP for your compliance and assessment and finally I'm using Digital Builder to build a custom image for CentOS to be able to add the no-cloud provider to it. So now there is a setup. First I'll make sure that my, yeah, that my LXD is set up correctly. Then I'll provide a small snippet for my machines to boot up that allow them for fuller root privileges. Like I said, this is insecure for the demo. Next up I prepare my network interface. LXD creates a network interface for you and by specifying these boot options who create a PXE and TFTP environment pretty easily in LXD. And then there is a snippet I'm using for setting up my, yeah, Catello informant. On the top there is some configuration to set up my user root password of pseudo access and my SSH key. Then there is the form and installer which I've cut off and it's running for both. Then there is runCMD which installs all the required sources that I need to run the form and installer correctly. So the workflow is this. First up I boot my LXD container and specify it with the snippet I previously shown. Then I attach a network interface with a specified IP address if you have, yeah, with either Windows server or something different. You can do this upfront, obviously. In my example I'm setting the environment the language I can do a complex show to show if everything is correct. I start the instance run my Ansible playbook and yeah, then it's pretty much good to actually boot up the machine. How many minutes do I have left? Like two? Fuck me. Sorry. Yeah, so network config, container config, Ansible setup, populate the farce file, start the container and then apply the playbook. I create a lot of snapshots in between so I can roll back if I want to and it takes the entire boot up takes about an hour or two hours and 30 minutes with syncing all my repositories. Afterwards I end up with an environment which allows me to simply create a new host to a TFTP boot with one CLI command and hopefully I will be able to demo that. So, handler, create handler, host, create my world. Takes a little while, host is created and now I should have a host over here which is called High World and it is started. It's spanning its installation. I'm not completely sure if the console works. Hopefully it will. Otherwise, oh, sorry. No, it doesn't. So, if you configure Forman you can self, if you configure a compute resource, for instance, Libfit, you specify if it creates a password for you for the, yeah, VSC console. The password is also, yeah, can be retrieved at the bottom of the contents and it's over here. Data password. And then we use fit viewer to just connect to it. Oh, LXC, XAC, C12, oh, list, okay, two, fit viewer which is the password from the source and now you can see that my machine is TFTP booting in my environment. Takes a little while to boot up, obviously and then we'll, yeah, simply spawn up a new host, fetch all the packages from my Catello installation and yeah, then I have a new host added to my infrastructure and yeah, I don't have any time left, right? One minute, okay, let's see if we can do it. Fear, no, not fear, present. So, yeah, like I said, sometimes this also works. If you press console and it disconnects, you can accept the insecure certificate and then use, just use the console to connect and I won't be able to push a package because my host hasn't booted up and I didn't think about my time limit, sorry for that, but through Catello, where is it, sorry, through Catello, you are able to let's see if I can find one real quick host, host, I will be able to install like the hstop package directly to my, yeah, dog's host or directly to a complete group of multiple hosts by simply performing this action. I'm not sure if the host is running or not, I don't think so, but by doing this you are able to, yeah, push packages to your host instead of pinning them directly through puppets or in your manifest or in Ansible, whatever you use and in my example I'm pushing to one host but the goal is pushing it to a complete action, so that's it. I'm here to push you off the stage. Thank you guys. Actually, if you want to see more, if you want, one second. Hello? If you want to see more, there is a conference on Monday and Tuesday in Ghent called Config Management Camp, which originated from this and he'll be presenting the actual full talk there. Yeah, and check these slides, I'll check this stuff out. Sorry. I think that's better.