 So, good morning. Thanks a lot for joining. A lot of people are here, so it seems we have a lot of users of ETC. I think everybody touched in configuration file in ETC already. My name is not Thorsten Kukuk, and I'm not a distinguished engineer. I'm Ignace Foster. I'm also working for SUSE in the same department. Thorsten is sick today, so I'll try to have his presentation. So, ETC, why do we think or why are we having this talk at all? First of all, we, as open SUSE, as an RPM-based distribution, has RPM files, which declare configuration files as a special configuration file. We have the same mechanism in other distributions, in Debian-based distributions, and I guess every distribution will mark configuration files as special. So, what does that mean? If we want to declare a configuration file and we update the system, we'll get a conflict if the administrator also decided to update that system. So, the question is, what should be done? Depending on what the package maintainer decided to do, we'll either get an RPM save file. In that case, the modified file will be moved away, and the configuration file distributed with the package will be put in place. Or it can be the other way around. The new configuration file will be moved onto the site with an extension.rpm new, and the configuration file modified by the administrator will stay in place. In both cases, we have a conflict, because you can't be sure what actually changed in the configuration file, and the administrator has to actually touch that and check for the differences. So, it has been that way for a lot of years, so that may lead to the assumption that everything is perfectly fine there. But, as I said, it may not be ideal, because the administrator has to do some manual search for those files on every update, and he has to check for the differences, and merge all those changes manually in there. Oh, by the way, I'm sorry, that's a very text-heavy presentation. I hope we can get into some discussion and may use that as a reference. So, if those configuration file changes actually contain the security update, we now have two cases. Either the security update won't be applied because the administrator modified file will just stay in place, and the security configuration option just won't get updated, or the administrator will just lose this configuration completely without even asking, and that's not ideal, we think. The talk is also referencing atomic updates, so what is an atomic update? An atomic update, we have some atomic distributions, we have the most famous one, probably Core S, on the open source side we have Micro S and Cubic, on the Fedora side we have Fedora Core S now, the atomic host, on the Ubuntu side we have Ubuntu Core, I think. All of those distributions are designed to be automated in some kind of way. So, what does that mean? That means that an update has to be atomic, in a sense that it is either fully applied or not applied at all, which usually means you have a read-only root file system, and then you'll just update the system and reboot into the new system, that's the atomic part. I'll be having a talk about that later today at 1450 if you're interested. But the essential part is it's meant to be automated, and the second part which we'll just talk about now is we have to wait until a reboot, until that configuration is actually applied, so that makes everything worse than it already may be, depending on your point of view, because if the update was applied somewhere in the background, then that configuration file which an update may have done is not visible. In the meantime, the administrator or your configuration management software or whatever may change the configuration files within the running system. So, if you reboot, then that's the question which configuration file should be applied. The various distributions have taken various ways to solve that, but the point is we do run into problems in any case. We have to resolve that situation somehow, and the question is how should we do that? I already told so, there are a lot of distributions, most major distributions also provide such an atomic update mechanism, but each distribution is doing its own thing basically. So, if anybody is approaching an upstream project and says, oh, we want to change something, could you please modify whatever we need? Then probably the answer is does it work for all distributions? And the answer is no, so there's basically no interest in doing distribution-specific patches, usually at least. So basically, it's all left to the user again. The user has to just check for conflicting files again to all the manual work, and that completely destroys all the intended automation which was supposed to be done. So, what will I be proposing today? By the way, that's open for discussion, you can always interrupt me, and if you have better ideas or want to suggest something, we would need a mechanism which would separate the configuration done by the package or the upstream project, and the configuration specifically or intentionally done by the administrator. In that case, it says two directories, etc., and something below user. Why below user? Because that's obviously not meant to be touched by the administrator, at least that's what the FHS says. We need to see because that's where the configuration files are nowadays going anyway. We'd need some guidance for new developers to get things right, and also for the distributions, of course, and the goal would be to migrate existing packages to use that configuration mechanism which would be splitting up those configuration files, one package at a time. So, of course, it would be obvious to start with the packages included in the atomic host and then extend it to the complete distribution. Leonard Petering wrote a blog post several years ago about a stateless system, and that's basically what that's trying to do, and you can just delete etc., and you still have a working system which will just use the default settings. I'm not sure if it's Leonard here, I don't think so. Oh, yes, if that's still a current thing you were thinking about. But yeah, that's basically what that could be reachable by that if everything's working as intended. So, in the end, the administrator should just be notified if something got updated. It should be visible what the administrator was actually changing, and it would be ideal if those changes would just be applied automatically. So proposals, what are we suggesting? I should note we've already discussed several things within the OpenSUSE community, and we are trying to get into more, to reach out to other communities now. I think that's a perfect opportunity here. In the end, we have SystemD already, which is doing exactly that. SystemD will provide its default configuration files in the userlib systemD system, or userlib systemD somewhere, and then it's possible to override either complete files by placing a file of the same name in ETC or around, but I'll ignore around for now. And you can also override just snippets of a configuration file by appending .d, by creating a directory with the suffix .d. So in that case, we have etcapplication.conf, or we could also just put a file in etcapplication.conf.d slash something.conf, and we could just override whatever we want there. So in the end, it depends a lot on the application, what actually needs to be done, or if an application is able to support such a mechanism at all. For example, the drop-in part may not be trivial. If you think of Apache, that's a configuration file which is really complex and just merging parts of it will probably simply not work, but what would be possible in almost all applications is to use two directories to actually read the configuration. And if you have a look around on the system, we noticed that a lot of applications, also basic system implications, support such a mechanism already, but at least an open source, it's simply not used. If you have a look at PAM, PAM already is reading from userlib.pamd and etc.pamd. We have sysctl, which is even supporting drop-in files. LDconfig is also using several directories, also with drop-in files or with extension files. And there are dozens of others which will just read configurations from multiple files, usually with that .d syntax. We have a few special cases, for example, a few things about etc, RPC or etc services or etc protocols. The question is, is that a configuration file at all, because mainly it's a static database which will not be touched but only extended. In that case, it should be trivial to also support that to just add additional files into additional directories. Did I forget something? No, not really. There's one special case which we don't know how to handle yet in a good way, and that's the whole user and password logic. Especially etc passivity, etc group and etc shadow. The problem is, how do you want to actually split that up? Because if you do, as I just suggested here, put all these system accounts in userlib, whatever, you have the problem that you just blocked the administrator and will not be able to touch those or remove those accounts anymore. And you still have one file which would contain all the passwords, so there's no real separation which could be done reliably. One of those problems, by the way, is that if you would split it up into several files, NSS Compat, in case anybody is using it, but I think customers will actually do, wouldn't work with that approach. The second suggestion or the second idea we had was to just do it as we suggested, use .d directories to allow applications to each define their own user and just put it in the .d directory, for example, etc pass vd, pass vd dot d, sorry, and etc group dot d and etc shadow dot d. But that's really incompatible with everything and would need a lot of work to actually get it done. I don't think that will be a realistic goal to actually change that. So if anybody has any idea how to solve that user, do you have a microphone? Like at least for exposing new users, dynamically you could use NSS, but that doesn't solve how to do the password stuff, they need some stuff in PAM. So you could have your own custom module that reads from other locations as well as groups and users from there, or however you manage users. Yes, and just the password file would be still shared among all files. Yes, that would be an option. So about the whole bike shedding part now where to actually store such a file. We were discussing a lot of, oh, just a second. Yeah, that will be the next slide. What should such a location look like? We don't want to have a binary file which is not searchable, I guess. So it has to still be grappable to actually find the values, that's what we just assumed. And it shouldn't be spread around the complete file system. Ideally it wouldn't conflict with the FHS, and it should be obvious for administrators what that directory is actually doing because in the end all the administrators have to work with it and accept that. And yeah, for the package configuration part it should be obvious that the administrator really shouldn't touch that. So we had a look at various other distributions and tried to come up with several paths and we discovered that we have a lot of paths already in place. If we have a look at clear Linux and some existing packages which are part of distributions already we'll see that those are using user share defaults and then something. If we have a look at CoreOS, or the old CoreOS I should mention, the container Linux part, then we have user share and something. On Ubuntu Core we have writable something. On our own distribution we are using user etc. something and it's also used by the new Fedora CoreOS thing but for a slightly different purpose. One obvious suggestion would be user sysconfig but the problem is sysconfig is already taking the name basically and we have several tools which are using user share MISC already. Torsten said it's not compatible with the FHS so I guess that's correct. So we all suggested all those directories to the open SUSE community as a first step and all of them were rejected. But almost everyone agreed that user etc. It's in the heading. Yeah, it is. User etc. would be an excellent idea because that's what is short. It is below user so it's obvious it shouldn't be touched. And yeah, it sounds easy and it was suggested quite independently. We didn't suggest user etc. by the way even though it's mentioned on the previous slide. Exactly, user etc. would just be a duplicate of etc. but contain basically what's currently installed by packages by default and etc. itself would be empty ideally. And that's what we are currently trying to implement. We just started implementing it and now the question to the general public audience is that something which should be proposed also for other distributions, is that something you can live with? I see some people nodding. The question was, the comment was if you go one slide back it's already in use by something else. Exactly, it's already in use by Fedora CoreOS. Leonard is approaching so I think you may have something to say. Just to mention this, if you don't specify a source where you copy from we default to user share factory which is what we thought should be the right path actually. So you didn't even list it there. Okay, let me add that. And actually system.dev.stream ships some basic files there so that even the crappiest distribution can still boot up with that. So this is what we thought should be the right one. Like the idea is that either you simulink or you copy from there to Etsy. But it's not the place where application should just directly open. So we have user share factory now? Yeah. So that's basically the temp files template directory. Yeah, it's about templating. It's about copying from there to Etsy if it's not there or simulinking from there to Etsy if it's not there. It's what we thought would make the most sense and documented as part of that. Okay, cool. Another thing to consider. Additionally, what we currently are doing is NixOS is not installing stuff as specified in the FHS but every package is installed in its own prefix. So we kind of do your user etc but for each package we have its own prefix so it would be kind of nice if that were relative directory to its installation directory of each package. So we have, for example, slash nix, slash store, hash, I don't know, cups. And then below that we have been cups and we have etc default configuration files which we don't necessarily use but we build our own configuration file but kind of having this relative directory to some etc default being used would be nice. Which distribution is that? That sounds like Ubuntu but... NixOS. NixOS, okay. Okay, so yeah, we have, as I said, we have several completely different approaches to that already and yeah, the question is how can we get some common criteria? I mean, in the end it's not that important because if we get the applications to support such a mechanism then we have the ability to just change the second path via a configuration option when compiling the application so if we couldn't agree on one path we could just say at least we get the applications to support that and everybody is able to use a custom path. I think it's not ideal because every distribution will have a different mechanism to actually configure things and it won't make it easier for an administrator. So yeah, the question is where can we find an agreement? You have another opinion? Well, more of a question really and do correct me if I'm wrong but how much of the problems go away if you use portable systems given that then everything is wrapped up in its own thing? So both the user story, because a couple of slides back you already mentioned like we could use system D users but that doesn't work because users are only present on reboot I think no problem is going away because also in portable systems you still want to have an administrator or user-adjustable configuration file somewhere so if you are packaging for the supportable app where are you actually storing the user configuration file? So let me answer that in portable services the idea was always that you mount some specific directory from Etsy into the portable service so that both the system and the host see that if you want to make the configuration like this but what I don't really get like changing back to that what I don't really get is like it was always our intention that sooner or later applications would need the second configuration directory but would just run with defaults if they find their configuration files missing in Etsy Why do you need the directory? For us user share factory was just the fallback if it wasn't easy to patch the software like for example Glypsy is very hard to patch because the upstreams used to be quite complicated so it wasn't easy for us to teach Glypsy that it would not read anything but defaults from slash user but we always thought that it should be up to the applications individually to put that somewhere and use a share or use a lipo wherever they wanted Why have a common directory there if it's not just for the stopgap to support the few components that still don't default to built-ins Because we have a lot of applications which depend on files in etc Sure but I mean you asked you wanted to agree on some directory here so that the various applications can be changed to go there instead of Etsy but my assumption was always they should just either have built-ins that they use if the files in Etsy don't exist or they should read whatever they want from user share user lipo if they have a private directory there I mean we have two problems here first of all if the application is only using its internal defaults in my opinion that's not so great because you can't see what the administrator can adjust if you have a basic etc configuration file which will just list all the things you can adjust then it will make it easy for an administrator to just change the corresponding values I mean yeah we do have main pages listing all those options but yeah just I mean what you're suggesting is getting rid of user etc or whatever completely and just using etc with all those remaining artifacts the administrator wants to have changed from the default yeah but then I mean for me this appears to be a documentation issue right like another question is if you have user etc around as documentation I'm pretty sure that if you build immutable images and deploy them in the wild they probably shouldn't come with documentation so they shouldn't be there right like you see what I mean if that's correct but if you change that you'll also change existing systems which are basically those you're changing the many there because they don't know user etc right it's a new thing which also has consequences for existing systems at least on open source we don't distinguish between regular old-fashioned systems and container hosts both of them are built on the same core I know it's a bit different from Atomic Host but basically we are using the same base I just want to respawn what Landon was saying I think you are conflating a bunch of stuff in that list I can speak at least for container linux the thing is the last one that you have in there are defaults that are kind of like application specific so the application knows about that that's the system this style then what you have in user etc for Atomic is not the same thing it's the result of a three way merge in that one is kind of like version as well what you have in the first the second line in container linux it's another different thing which is application doesn't know about that but the first time that you build the machine we are copying those files in another place in ATC and then there is another category of application that are like they can use defaults that are sim linked directly to user so it's kind of like there are at least four different categories into this list and I do agree with what Landon was saying which is I prefer to have ATC as user customization which is something completely different like user distribution defaults and the fact that right now there are some distribution like Atomic or Fedora Koras that are doing this three way merge in order to version ATC it's kind of like a stopgap solution from my point of view like it works mostly but it's not the same thing as the other stuff that you are doing so I think that like we are speaking past each other because you are putting all of these into the same category the question is which directory we wanted to use we know that use ATC is for the three way merges in Atomic in Fedora Koras sorry all the name changes and that's why we hesitated to use that as the default directory for OpenSUSE in that case now but that doesn't mean that it's supposed to be the same thing we just recognized the name Clash and it's used for similar things mainly which is somehow getting a configuration file merge do we have so first comment about the problem with sysusers and the users being created after a reboot so this is essentially a self-program in Mantriva they split out the sysusers configuration to a separate package and have a trans files trigger to create the users when the package is installed so that when the package that actually has files owned by those users is already there so it's not really a blocker and as for the general thing there are two things one is moving stuff out of ETC and I think that we all agree that this is a good thing and we want to do that and the other thing is the details of the naming which is something that we clearly disagree on but it's not really that important so as long as you push people to move stuff out of ETC one by one this is for me the most important part and we will all benefit from it thanks a lot for the summary that's what I also wanted to say I think we all agree that something should be done about the ETC situation not a lot of people are nodding trust the second as well we still have more time for questions and comments don't worry just the last slide I wanted to show we do have a temporary wiki page on the OpenSUSE wiki for packaging for user ETC but the most important link is the second link which is Thorsten's original proposal with all the motivations for the chest to use that repository for further discussion as we didn't come to an agreement nor we can discuss during the conference we can also set up a buzz of a feather or a buff session for anyone who's interested the problem is I will not be here on Sunday so you'd have to discuss without me that's not a problem but and the last link is a link to a project called LibEconf if applications want to use the new mechanism then if they see a C++ application they can just use the LibEconf to read configuration files with one library call which will then just read all the corresponding places let's say alpha software but we're working on that to make that transition easier so I have five minutes left that's my presentation I don't think we have questions we have for discussion yeah so thanks for the presentation I do agree completely with the problem statement what I'm asking myself a bit is if files really cut it anymore or if we need some kind of different storage for configuration files first of all when we are merging things together we have different formats some configuration files might be for instance in XML which is not so easy to merge just by appending files together for instance I think Cubs solved this problem by not having a root element in Cubs printer files and so on but there might be other programs with that problem another issue is that we still want to run for the administrator of the system whenever there is a change needed then it needs to be propagated from USR ATC I don't care much about the naming what we choose factory or whatever to a custom change he made in his own ATC and it looks like this kind of problem has been tackled before like I'm thinking about decon for instance a binary format for configuration files or even I'm going to name the devil I mean the windows registry so what do you think about do files still cut it for our current needs in my opinion yes because it's a really simple format which is quite obvious to search for if you are thinking about the windows registry that's in my opinion a really bad example of how to do it because it's almost not searchable in an efficient way and yeah you simply don't find anything in there now with the different configuration files you have one directory or one file usually at least per application so if you're looking for your application you can just search the default the default directories in the ATC and you will find all the configuration files belonging to that a registry would have one advantage namely being that you get a standard for all the applications it would force applications to obey to those rules of that format but again a lot of linux applications especially ported to windows will still keep reading their own configuration files because it's simply not possible to squeeze everything into that format so we won't get rid of files in ATC anyway in my opinion you have to have some way of configuring it of having extended configuration formats does that answer the question so yeah I think configuration files are easy to understand and to edit yeah that's why I think they should be preferred compared to binary only format but that's my opinion something else I'll point out that config files make it a lot easier to deal with config management so if you run something like Puppet or Chef or Ansible or whatever and you need to write out configs you can use templates you can templateize the configs it's very evident what's happening you can see diffs of what's happening if you're doing this in other environments where you don't have this possibility you end up having to deal with amorphous concepts that mutate the system and while the end result is roughly the same in terms of what you can do with it I think we can all agree that it's definitely harder to understand in a lot of cases it's harder to understand how to get from here to there I think we still have one minute last question we'll come in we're just responding to that too the problem is also with things like gconf and deconf they have sematical problems because they store every bit of information separately and evolution for example you know this email program from the ground people then famously ended up sticking XML configuration into one key because they wanted to have atomic updates and you just think okay then yeah you have to give up so all these things like registry and these things ended up having semantics that are not compatible with how most people who understand and have non-trivial programs actually want to do stuff because they want more atomicity and be able to change the whole configuration at once and yeah files are crap but they kind of allow you to deliver this in some way or another so yeah it's a complicated issue and I don't think any yeah I mean there have been attempts to unify on something like Electra or something there was a project like this all failed and deconf is certainly the most successful one because it has most of known behind it but I see no way in the world that's a political thing where the semantics are not strictly better I would not want to fight that fight all right thank you we've actually run out of time thank you Thorsten