 Hi everybody. Jeremy and I work at Evolix, a seasoning company since 2004. We provide hosting and mostly network and system administration 24 hours a day. Today we manage about 700 Debian servers for server customers. Let's cover configuration file for Debian developer point of view. The main question is how to have smooth upgrades. There are several ways to handle config files. The simplest way is dpkg confile. From Debian policy, dpkg confile are best effort configuration. As you know probably dpkg handles confile when they are embedded in the package. dpkg keeps track of empty file checksum for all original confile. You can use them with dpkg-s. If a confile is modified even a single character, during upgrade you have to choose to keep the old file or install the new one. Just for the record, I mentioned siblings. You can use siblings from slash DTC to files in slash usr or slash bar directories. But very few packages use this way. More advanced way is using maintenance script. You can generate confile with postal script during postal of your package. Then you can include dynamic value like osname for example with command osname. And ask question with well-known dpkg confile. Why not use UCF? UCF stands for update configuration file. It's a tool for preserving user change in config file. It uses three rare merges. Then it tracks the original content and can merge if needed. It's even compatible with dpkg question. For example, samba package use UCF and dpkg conf which is great. Here is the interactive choice with UCF in action. Take a look to the option 6, three-way of merge. UCF use these three commands. Of course you can have conflict during a merge. That's why this option is only available in interactive one. Suddenly I don't know why but only about 200 packages use UCF in stretch. If you know why, please, I want feedback. Then of course all these techniques are not enough for system in common needs. For example, you want to detect how much memory you have on your server and give 50% of it to MariaDB for example. You want to add or remove Unix user and then change other user in SSHD config or sudo configuration. You want to add virtual host in your favorite HTTP server every day or sometimes. You want to enable, disable a backend in a load balancer configuration. Then for all that needs, we need config management tools. That's our experience with Uncivil. About two years ago, we took a first look to Uncivil because we wanted to automate some critical or repetitive tasks. Then for about a year, we have been porting many of our install or maintenance queries to Uncivil Playbooks. And now for DevGonf, we thought it was a good idea to share with you our experience and have your ideas and feedback too. Why automate? Automation tools have gained a lot of traction lately. They bring homogeneity, reliability and speed to server's management because they focus on the state of the server instead of a series of commands. But it's true for all automation platforms on Uncivil. So why did we choose Uncivil? And why is it first? It's a configuration management and deployment automation tool. So why? We settled with Uncivil instead of Chef or Puppet for example because it's simple, yet still powerful. It is agentless, meaning that there is no Uncivil component that stays on the server. It only requires an usage connection and Python. It doesn't force you to organize anything around its model. It can be added afterwards, used as much or as little as you want. And you can remove it if you don't like it anymore. Very important, it's very well supported on DevGon. DevGon and Uncivil are pretty good friends. We have DevGon packages since we see backwards. On stage we have Uncivil 2.2 and soon Uncivil 2.3 in Unstable. Thanks to Uncivil Manager, Harlan and Lee for here. Thanks guys. So we like DevGon a lot, that's why we're here. And we use it a lot for the majority of our servers. And we build on top of it. It's no surprise that we install a lot of packages. We create users for our team members. We apply security policies or performance tunings. And most importantly we customize a lot of services for our need with good practices, what we consider very good practices in our context. We have written more than 50 roles for Uncivil. A role is a kind of package. They are all publicly available on our Git repository. And with them we can install all our servers and have very good common ground, very good defaults for us. But also many options for configuration. We also have an ambitious goal. It's to be able to install all our servers 100% with Uncivil. But then for the management we can go either way, continue to maintain the server only with Uncivil or have a hybrid approach with manual interventions and Uncivil driven operations. That gives us many quite difficult constraints. We have to be extra careful in the way we manage our configuration to not introduce bugs or misconfiguration. Some services can include files within config files. That's what we call includes. There are many variations on the implementation, but the idea is that you can extract fragments of configuration, put them into external files and then include them back in your main config. And you can push the idea a little bit further with automatic inclusion of all files from a specific path, like com.d directory for example. And that's what will help keep original files pristine. Apache as you can see and you probably know is using this technique very extensively for modules, virtual hosts and also other configuration sections. And with packages that use this approach, like by ADB, in a much simpler way, we have a pattern for customizing the configuration without touching the original files. We are adding two new files. The first one is called Evolix-defaults and it contains all our default settings. By default we mean what we think is the best for the vast majority of our servers and this file will always be created or overwritten by Ansible. That's why we are able to change the defaults later with no fear that we will mess with the overrides after that. The second file is Evolix-defaults and at the beginning it's just a placeholder. It contains only comments or explanations for how to use them. And this file is never set by Ansible so we can simply add to it manually of Ansible and Lightning. The naming is very important for this pattern because it enforces the load order and precedence and it clarifies the responsibility of each file. And we try to apply this pattern whenever we can but it's not always completely possible. And for Debian Package, we have packages that does the job very well. They are encouraging the use of overrides and includes. I will give you some examples. Fail2Band is explicitly discouraging the modification of jpeg.conf.info file. MAVs use this company directory, populated with default files and you need to use overrides in a new file. Defcot use confd directory also and use UCF which is great. cctl use cctl.g directory so you don't need to edit slash etc slash cctl.conf file anymore. But we have also some packages with lack of support of overrides and includes. I will give you some examples. sudo use sudo.g directory but you can't override alias for example and the worst sudo will be completely broken if you do it. Nagios and RPE use confd directory but doesn't support option overrides and even re-file randomly in confd directory. So surprising. Postics doesn't support include and upstream doesn't want support it. OpenSSH doesn't support includes also and the order of option in SSHD confd is important so it can be very dangerous. Nginx doesn't support override. Screen support includes an override but Debian package use only one cctl file. So Jeremy let's play with an edition of confd file with all that stuff. I will hurry a bit because I think we're late but Ansible gives us many options to edit files. I'll call them briefly. We have line in file which is perfect to add or change or remove a line in file. But it's not very good for multiple lines editions. We have also replace which is essentially the set command. It doesn't replace the whole line but only what matches the regular expressions. Line in file and reject and replace and other modules use regular expressions and as you may know regular expressions are quite hard. For example try to remove the exec function of the disabled PHP function and remove the shell exec. Maybe you will have fun. And if you can get away with replacing the whole file instead of poking into it it's really easier. We also have blocking file which is very useful to insert and update chunks of text within predefined markers. This one is very useful for multiple lines editions but it's quite dangerous to use it if you have a connection with line in files or replace for the same lines. Copy and template. They both replace the whole file either statically or with template. And with template you can use variables and conditionals to build the final file. But we advise not to put too much logic in your template because the maintenance is harder. So to sum up stay at the file level if you can. We do that when we are not sure about the file content. And if you need get down to the line by line level in the file and using those modules carefully with a lot of testing. We need to have smart config file for automation and let summarize all good practices support override in config file. Use include when possible to have untouched before config files. Use confd directory when possible. Use confide generative by Mandar script and take a look at UCF provide script to validate the config like Apache CTL config test for example and provide tools to enable disabled config file or module like Apache d.com. I'm not sure we have time for questions maybe. Yeah we have a few minutes if you have questions. And if you don't have questions now it's okay we'll be here until the end of the week. I've heard Holland and me are organizing above I don't know if it's down or not yet but we can talk in the hallway about that's about what's available for sure. Thank you again.