 Yeah, I'm Richard, Linux distribution engineer at SUSE, which is like the most generic job title anywhere. It's like, you know, human planet earth And I'm going to be talking about some of this stuff. I'm not going to be talking about a distribution for like my first time or first time ever Instead I'm talking about a nice library living underneath But starting with that Who is responsible for the configuration of a machine once it's deployed? You know Not learn out pottering although, you know, if you want to point fingers you can you know, you know Is it the distribution? Is it the upstream who actually don't get a mention here or is it the user? You know from speaking from a distribution perspective You know, I honestly believe you know distributions have a hand in making sure the configuration of the operating system once it's deployed You know is right and sensible, you know, we should be making sure the defaults are Sensible and good out of the Xbox experience, you know Secure, you know, whatever sensible means really. I mean, it's you know open to interpretation different distributions have different interpretations And of course different upstreams have different interpretations too And you know, sometimes we don't agree with what upstream who decided of the defaults But also upstreams always do interesting new stuff and that interesting new stuff often requires new configuration parameters to be set or you know the the sensible defaults change and Somehow we need to get that to our users But at the same time, you know Some users believe the mix is about choice or at least there's definitely the choice They have to you know potentially ruin their system, you know, and they want to configure their configuration the way they want it And that means, you know potentially having Unsensible choices or at least, you know weird ones that make no sense to us And they're not necessarily going to be paying attention to all of the stuff going on in the distribution in those upstreams So they're not necessarily going to know that this default needs changing this new feature has some nice new flag You know, just like the system detour Queenie. We just had with all those wonderful different flags You can put into your system the unit files, you know, the you know, right now if a lot of services a lot of demons You know, that's a bit of a minefield and sort of the typical classical problem is You know distributions put up got a package. We've got some service called food. There's a conflict filing ETC The user edits that and sets up the way they like it and then a new version of food comes along and We want to change that, you know, we want to change the defaults We want to introduce the new parameters, you know Or we, you know, there's been some horrific CVE and we desperately have to make sure that there isn't that setting set by default And whatever happens, there's some, you know, horrific or semi horrific explosion. There is no perfect solution to this problem at the moment No matter how we look at it, you know, you can notify users on wikis and stuff, but they don't read it and it doesn't work You know, or if you try and find a technical solution, you know Most of us as packages try something like some fancy post script where we're you know editing was said and just praying we've got our Regular expressions, right? And then we get it wrong and the users scream and file bugs saying the distro broke our conflict More conservatively, you know certain packaging tools, you know, especially RPM and sort of art, you know, we have these sort of You know nice helpers where you know weakened in the packaging tool Back up the user config install on your user config and you know leave their old config as an RPM save file And then the user doesn't bother noticing that and they file a bug saying, you know, the distro broke our conflict Or we flip it the other way around and we're paranoid and never touch the user's configuration And we put our new settings in an RPM new file And then nobody ever looks at it and then they file a bug and we just turn around and say sorry your setup isn't supported You know, it doesn't work at all at the moment. It's it's the hell we've all been living doing distros for so long And the hell's getting worse Because you know, there's a lot of new trend of the topic distributions like with open Susan micro s Where this problem gets even more complicated because like in micro s We have a read-only root file system, you know a user can't modify that root FS once it's there But we want them to be at a modified ETC because you know, they need to be able to change the settings So but when we need some way of atomically delivering those new configuration settings that we need on there without messing around with that conflict So, you know, yeah, it has all of the issues plus more because we yeah, we can't just mess around on the root file system like we used to and When we were dealing with some of these issues that cropped up with what we're doing in micro s, you know We realized there's there's one really configuration heavy service sitting on All of our micro s machines, which doesn't give us any problems at all and it's wonderful and life is great and It's just indeed Would shock me a little bit when I realized it, but yeah, you know system D Solves this problem really really nicely, you know, it has this lovely structure of everything's a unit file And there's unit files live in one of two places, you know The distribution config is meant to live in usr lib system D system And that's where all your unit files live your timers your services blah blah blah blah And then for your user config every change they make is in ETC system D system And then the two worlds are separated and we don't have to worry about that, you know We don't touch ETC system D system. We just touch us are and all is good It gets even better because with system D services. You also have this concept of droppings Well, like a per service basis you get a you know food dot service dot D folder And you can put little drop-in configuration files in there those drop-in configuration files can individually override individual lines or parameters of the the other config file elsewhere So, you know and it's even that is layered so you can have you know USR drop-ins and ETC drop-ins and system D figures out what the right priority is and applies to right one Which is like what we're using with like Kubernetes now, you know You've got you know container run times and Kubernetes and all these other distro dependent stuff and they all want to modify each of the services, so US yeah, and our and opens is a cubic at least, you know USR live drop-ins, you know We have drop-ins off the drop-ins off the drop-ins and then users can put their own drop-ins in and then you eventually get the right Complicate the end so it's great when you've got this sort of multiple layering And we realize well, this is cool first, you know system D services. Why can't we do it for like everything else? Which of course, you know is going to need some work, so let's try and make it easy for people and so Pascal an apprentice at its Suzer Last year this like this entire project is like not even a year old Yes, that basically started writing a C library to kind of first as an experiment to see how we even do this and Then like all good experiments. It's got completely out of hand for basically Providing to anybody using this library API calls and functions to be able to be able to do the same kind of configuration layer in their services in their applications It's an MIT license so you can use it for everything and the two main use cases are literally You know both layering sort of an entire config file like the typical system D service thing of USR versus ETC or If you really want to be fancy during sort of food dot D and and yeah, drop-ins and layering those two things Like I said, not even a year old This is its first foster it could ever be at because it was 26 of April last year. We started it But yeah, we're using in plenty places. It's not magic, you know start with start with the negative You know you hit you need to modify software to have live econ support I don't you know, we're not magically taking over other bits of services And in its current form it only supports of any style configuration files where you know You've got a group. You've got a key with a value. You've got another key with a value It's flexible in the sense of you know, you can have whatever delimiter and white spaces if you want But you know we the moment we're only working with these kind of fit configuration files which which form this structure I'm not going to give like detailed code examples and you know the API list is getting longer and longer But at its core this sort of eight or so core API calls which kind of do all of the work Econ for read file reads a configuration file takes all those key values and you know dumps them in a key file for you use elsewhere in your application Reader basically does that so but for multiple configuration files in a directory So, you know, that's what you would use for the kind of food service the equivalent There's a function for enumerating the groups in there Which you know can be handy when you're looking for that one config file nested in the one config nested in one group Somewhere down the line Get the keys and then we have a whole bunch of get value of various types So you can you know grab your strings and make sure it's all types saying and grab grab your right long int and Get the the correct value of the correct type you expect as you go along That's all the reading on the writing side of things. Thanks We've got a bunch of setters for doing the same in reverse so you can set your configuration back in there We've got a tool for merging files so you can just yeah two straight files merge everything together job done and all of those API calls are up stable versions You know, we're not gonna break them at all. That's we promise right file on the other hand for writing out this to a configuration file We've got people using it, but it's it's not yet stable It's it's you know, because you know, we're not entirely happy with how it's writing or handling things like the file What are you being open already being edited? But it's there if you want to write the file back at the desk In open Susan land, like I say, this is all kind of gone out of hand but in a fun exciting way Open Susie is moving towards using USR for all of its district conflict. That's the goal It's gonna take years, but we're moving in that direction as far as we can so we want ETC to be considered user data and you know treated with the same reverence and avoidance as Slash home or slash SOV, you know, we don't want to be messing in our users sandpit. That's you know, ETC should be theirs We don't want our packages touching it. We don't want our scripts touching it We we don't want to be responsible for breaking anything anymore So, you know in the case of like system D. It's already using USR But for the stuff that doesn't We're now starting to use USR ETC and basically moving our configuration to that In the case of Libicom Libicom doesn't care about these locations. It's all variables So, you know, you could use it how you want to put it where you want So this can handle, you know user lib whatever user ETC, whatever it doesn't matter But for us we're finding it's easier to move things along to that because it's seen a one-to-one mapping of user ETC Whatever to ETC, whatever makes makes the logic a bit simpler for us There's a number of examples where we've already got applications moving across to it like PAM, PAM upstream has Libicom support since three or four months ago, I think now So if in open SUSE, we've already packaged up with this So user ETC PAMD has all of your PAM conflicts Then user conflict can be in ETC PAMD Both get passed at one time and ETC values always went nice simple The same for shadow also up. This is also upstream, which of course they're also using PAM So it also has their conflict in PAMD. So it's fun that when they were sharing the same thing and using the same library But you've also got login deaths and I just realized a typo because that's meant to be used at ETC login deaths But yeah, the login deaths are stored there ETC login deaths, you know can be overridden by users the ETC values always run. That's all done at runtime And all of the UTOLINIC stuff, this is not yet upstream I think it's submitted but there's some discussions about our patch, but we've patched it in open SUSE So all of the su login remote etc in open SUSE are using this and using it exactly the same way But there's exceptions. So reboot manager, which is the the tool in micro s which basically Detects when there is an update pendant Well, there has been an update applied and there is therefore a reboot pending and required to be done That configuration is really simple. There's like it's one group. There's like four parameters It's there's no point doing all this complicated nesting and layering So in that case, you just got a nice simple straightforward config file and you can put another nice simple come straight forward conflict filing in ETC And we just merge both of them at runtime and we we use that Unstable write file to just dump it in ETC reboot manager com And so all the logic in the code for actually reading the configuration file hasn't changed at all We're still reading the same conflict file But yeah, Libby comp is doing a little bit of magic before that to move things along And yeah, we want as many other projects to start adopting this using it in their own way We haven't yet Done this where you know, you must follow a certain sort of system d like or you know Pam like approach where you know, you have to use the same copy You know, we realize upstreams have their own way of you know, absolute applications that have their way of doing configuration We want to make this as easy as possible to fit in So you've got the primitives here where you can Take Libby comp make our life easier and make your life easier. Hopefully But you don't have to rewrite your entire logic of you know How you're consuming your configuration in your application We are thinking of adding more high-level API calls for people who just want to Make things even simpler for them in the long run but before that We have some Yeah concrete plans, you know passing conflict files is hard and you know our parser isn't perfect So I've been looking at using a much more established configuration file parser as you know instead of writing our own and constantly tweaking our own So August is on our radar. I have some patches that are roughly working and you know I'm slowly patching it into all of the API calls It does a very good job technically speaking, you know, it's very it's very reliable It handles far more variations of configuration files than just in ease. It also handles Those any files more gracefully especially the weird ones with weird delimiters and weird white space rules and all that stuff Yeah, all gears have figured that all out already The downside is though, you know, it does make the dependency tree for Libby comp You know a little bit heavier than just a nice simple C library that you can plop on and put into everything And the code is getting a little bit more complex. So this this isn't certain I want to kind of get feedback from anyone here and see if there's any other options out there Or help doing this, you know, if people think this is a great idea But yeah, it's a sort of work in progress and the direction we'd like to go because we realize that just doing any files is is somewhat limiting for this idea and Dominic is Dominic here. No Dominic another one of our friend. This is he's working on E-conft tool, which is a basically to be sort of a generic helper tool for The idea being you know an application started using Libby comp that now means you've got configuration files in USR configuration files in ETC How does a user know what parameters are actually being applied when how so you come towards sort of a generic helper of you know Taking that default logic, you know, you don't you know like ideas like you come to Pam D. What your config is done You know give it give a list out there and a bit like system CTO with like system CTO edits You know be able to edit the config and it'll put the right snippets in the right place and make it a little bit easier This is just the beginning if people like this idea, you know, there's obvious things we you know, we could do to improve You know different file paths have been being the one that I already mentioned, but also language abiding It's not everything is written in C So sadly Especially when it comes to go But yes, if people won't go if people won't Python if people want to ask, you know, this you know This needs to be something used in as many places we can You know, we're not just gonna start here. Yeah, stay here, but we can't do everything alone. So, you know, if you like this idea The code is open it's MIT contributions are welcome changes are welcome Features are welcome You know, please help enjoy start using it start putting it in your distro That's why I'm here basically And with that, I think I've managed to squeeze this into 20 minutes or so. So there is just a bit of time to questions at the end Yes, sir. Yeah So why is the read file just a reader singular file? Yeah That's because there are definitely cases where you just have one conflict file for those cases where you have multiple in a directory That's why we have the readers conflict the readers API call and readers just you know reads the directory and Read files all of the contents of that directory Yeah, the application developer who is patching Liby com support into that application has to do that So the user doesn't but you know Liby com is there to be consumed. Yeah So at the moment, yes, you know, we are planning on adding, you know Higher-level API calls which you know have opinionated views on how you do that matching and merging Which will make their life easier, you know, that's you know Yeah Yeah, if you don't mind yeah, but like the the reboot manager example, it's kind of like a nice simple one You know, it's one conflict file. So, you know, the conflict file now lives in two places you merge it Yeah, you know, not everybody has complicated stuff like Pam. Yeah Yep Yeah, absolutely. Yeah That in in high crew. Okay. Yeah, we will look at that. Thank you Yep No, we do not handle merge conflicts, you know So the moment with with the current calls we have, you know, the you know, you've got, you know With that merge files, for example, you know, you're declaring one is going to override the other So you know by how you order it and you know, there's no there's no magic there And so for the more complicated stuff, you know, you could use those primitives to do fancy stuff with with handling merge conflicts, but There's no standardized and simple way of displaying the conflict the all the conflict files, you know, that's kind of what the Econf CTO or Econf tool, sorry, we was called Econf CTO a week ago Econf tool is is looking at doing is sort of giving us giving us sort of standard way of doing that. Yes I Have a feeling that it doesn't really make sense to port Libby comp a system D to use Libby comp, you know, they're doing it fine We wouldn't really have anything there. Yeah No, this is this is all completely fresh. No, no reason code. No, sorry, but just repeat the question the question was, you know Did we reuse any system D's code with us It's it's what How the moment it's just it's just you yeah, but it's a bit basic at the moment. Yeah, it's a bit naive at the moment We admit that Yeah Most of them are in this slide. I think I might have missed one or two But you know, this is kind of kind of where we are after it. Yeah, so so the question was how many upscreens have we patched? So the most of them have been like the the question was what do they think Most of it has been like surprisingly. Wow. This is called great merge in done Like like so utero Linux has been like the only one where there's been sort of questions feedback like improvements Juggling around the rest of being like, you know, here's the patch find done in Of course in the case of review manager, you know, we are the upstream so that yeah, we cheated but The question yes Yeah How would the distribution chip a change to the default in the case of like we were manager will be writing it out Yeah, we wouldn't You know, it's you know It's in the case of review manager because we're the upstream as well We know we're like probably never gonna do that and if we do like the config file simple enough So we'll still hack around it the old-fashioned way Which you know, it's a road for our own back, you know, we'll do it properly one time But in that case, it's mainly for adding new configuration files because review manager is is you know In a kind of similar state of development relatively new moving relatively quickly So they're just using Lee we come from makes our life way easier for adding that new thing out to existing systems Plus also with review manager being like a really like that's definitely something where you don't want to mess with what the users already decided You know, they've got their policy for how they want their system rebooting and that like those core parameters are basically that right now So the chances are even if we change a default, we're not going to change the default time your system is booting You've picked it. You're running it. You know, it's it's done dusted You would you would hate us if you suddenly your machine started rebooting at the long time Yes, so the question is do we provide any Any tooling to kind of help people move to this approach We don't you comfortable do some of that for users because you know You'll have that there where you comfortable will be showing this is coming from the district of us are this this is what you've got in Etc, you know, so this is what the runtime uses We could look at you know having that or working on that for making sort of the transition easier But we have found because we're going to this, you know, every upstream app is is different They've implemented their configuration differently. They're consuming it differently across with their entire code base And we're not trying to force everybody to doing Everything exactly the same way, you know So this is sort of trying to be that middle ground if it's flexible enough that you can patch it in so your conflict could be split and layered But you don't have to you know, don't have to do everything exactly the same way Time's up. I'm sorry, but we'll be outside so we can catch. Thank you very much