 Hello everyone, my name is Nicolás and I'm part of the KDE sysadmin team along with Ben and Bushan. Today I will show you some work we started doing behind the scenes that should make our job easier and hopefully should also make it easier for new people to contribute to the sysadmin team. I will start showing a simple example to illustrate the problem. Let's say I want to make this URL identity slash register take you to the actual identity registration page. It's a silly example but it will work for here. So what I need to do is edit the Apache configuration on the server and just add a redirect directive. The first problem is which server is it? We have lots of websites and lots of servers and I don't remember where identity is hosted. So while we have internal documentation, I can look at that and it says virtual server hosted at sabba.kde.org. The problem is that sabba doesn't exist anymore. In fact, we decommissioned it two years ago. So why is this wrong? Well, this particular file hasn't been updated for four years. So, okay, since I can't use the documentation, I end up looking in DNS to know where our own stuff is running. Okay, I find that identity is running on Orbi. Next, I log into the server and I find the configuration file I'm looking for. I open it in a text editor right there on the server and I add the line that I need. If everything went well, the redirect should be working now. If I made a mistake, it's possible that the identity website is now broken for everybody. And I have to hurry up to fix it or revert my change before it bothers users. So this way of doing things works because it has gotten us this far after all, but we could do better. The documentation is updated. The configuration files are not in version control, so we can track changes. We have no record of who did what, somebody could find that redirect and not know since when is this here. Instead, we just make the changes directly in the production servers. If something goes wrong, it will affect users immediately. And it's also quite hard to test changes locally because it's hard to get a local development environment for this. So how do we fix all this? Well, to start with the documentation, we decided to start from scratch pretty much. We have a new documentation. It's not plain text anymore. We are using things for formatting, so we can link within pages and all that. We tried to have more explanations of how the server is set up instead of just a bullet list of what's installed where. This is still a work in progress. We have a lot more to do. There are a lot of things that we need to document. And since there are no secrets or anything, it's public, which you can check out at sysadmin.docs.kde.org. Next, let's talk about managing configuration. As I said, we want it to be in version control, but server configuration is more than just the config files. For example, if I change a file in a repository, how do I get it into the server? I need to know where it's supposed to go in the server and what services I have to restart to make that take effect. And really, the whole configuration of a server also involves which packages are installed and user accounts and databases and crown jobs. And what steps you need to install something from scratch. So in order to deal with all this, there are tools called configuration managers, which do a lot of automation around these issues. We decided to use Ansible, which is an open source project from Red Hat. This is an example of an Ansible file. It lists tasks that you would execute on the server. If I run this on a new server, it would install Apache, it would enable the SSL module and install this file. If I run it again, it knows that Apache is already installed, so it will do nothing. It knows that the module is already enabled, so it will do nothing. It just says no changes. And also if I change this configuration file and run it again, it will update the file and it will restart Apache. So that's another step I don't need to worry about doing manually. Once I have all this, I can put all of it in a Git repository. And now we can track it just like source code. I have a record of what changed and when it was changed. I have a commit message for all those changes. Can you skip blame? Now, if I get the Ansible playbooks so complete that they can install a website or a service from scratch, I could also run it in a virtual machine. And that would get me a local development environment, so now I can test things locally. The problem is that it takes a lot of steps to set up such a virtual machine. You have to configure the network and SSH and your SSH key and all sorts of small annoying steps. So for this, we have another tool called background, which automates virtualization software like virtualbox and makes it much easier to create development virtual machines. You can just run one command and it will download, pre-install this image, or in this case it's Ubuntu. It will start the virtual machine, set up the network, set up SSH and it will be ready to run Ansible on it. And it will also run Ansible for you. So here we will run the same playbook that we are using on the real server. It will install everything that is needed to run the identity website, MySQL and PHP and all of that. And once it's done, you can use the website locally. You can make changes to the configuration and to the website code and test it yourself. So in summary, this lets us get the configuration in version control. We can experiment locally, but most importantly, you can experiment locally. You can make changes and test locally and send us pull requests. So this way you can contribute to the sysadmin team without needing access to the real servers. So we hope that once we continue with this work, this will help us get more people helping in the team. Okay, if you want to contact us, we are on the sysadmin channel on FreeNode and that's our email address. Thanks for listening.