 Welcome. We have Marcus Robb here with the talk configuration revolution. Thank you for the introduction and thank you for joining me here today. So does everyone understand me? Is the audio okay? So thank you. So, yeah, we have the title of configuration revolution. So that's quite a claim and I hope I can back it up in this talk. So my name, as already said, is Marcus Robb and Thomas Vasa will also support me later with a demonstration. Yeah, so let us start with some questions. So the first one, so who of you has edited a configuration file? So, okay, yeah, like I assumed everyone edited a configuration file already. Who has heard of Electra? Okay, so nearly no one, so that's good. So, and someone also using Electra? So, okay, so maybe if someone wants to start using Electra, we also have T-shirts afterwards. So, let's see. Okay, then, so two more serious questions. So, who of you has designed a new configuration file format? So, yeah, there are quite some. And who has implemented a configuration file parser? Okay, so also, okay, so we asked these later questions in a survey and I think it's the same picture as we have here today. 25% implemented a new configuration file parser. So that means every force has done so. And only 30% or only 5% more used some existing configuration system library or APIs. So that's nearly the same. Okay, so, yeah, how to not change the situation is quite clear. To try to establish a standard for configuration file formats would add one more. So, I thought certain years ago this might be useful and we had the idea that every configuration option should have one configuration file. But actually, we just added one more format and it did not help at all. So, yeah, keep adding new parsers and configuration file formats in an unorganized way actually led to the chaos we are currently in. So, this picture basically demonstrates a small part of how the configuration landscape looks today. So we have many isolated solutions for operating system, for desktop systems and so on and so forth. Within these platforms we sometimes have a really nice integration which works out well. Two examples, the GNOME platform or the KDE platform put a lot of effort that everything runs smoothly within them. But other applications or other platforms are not integrated in the system. So when you want to have from GNOME one of the KDE settings, there's no easy way to get this configuration. So, yeah, you might already know this slide from 2004, something very similar was shown. And, yeah, initially, Electra did not consider that there are also huge advantages with this situation. So let's change the picture a little bit and just don't draw the red arrows. And here we see that we actually have a really modular system. We have a system where applications are really decoupled from each other. And this is basically something we want. So at the moment we don't have a single point of failure. So we can rescue any system with a simple editor. And another advantage is that we have a diversity and developers can choose from many different options how they want the configuration system. Do you want JSON or XML? Everyone can choose. And this is, of course, also a great thing. It's a diversity is the strength of the Libre and Open Source System Floss. So why should we care at all about this problem? In other ecosystems we also have a great diversity, many different solutions doing the same. For example, video formats. We have plenty of video encoding libraries or video player libraries and they coexist nicely and there's no problem with this. There is a reason why configuration is different. There's a really important difference of configuration and other libraries and the difference is called misconfiguration. Configuration is not only used by developers but is also used by system administrators and end users from a completely different perspective. And I personally find this duality of user and developer in the same system really fascinating. So what is misconfiguration? As you know, configuration has many constraints. For example, a configuration setting might need to be a valid file. It might need to be a valid IP address. It might be an integer. It might be that it must be lower than the number of CPUs and so on and so on. There are many, many constraints. And at the moment we demand from users, from developer, from users and system administrators that they manually find a solution. So basically the brain as constraint solver to think of all these hidden constraints that are somewhere hidden in the system and it gets even worse. They have to solve all these constraints without proper feedback. So at the moment the application might just crash if you have some memory variable or like experience it myself. For example, I had a U limit of some specific size and after KD upgrades suddenly KD crashed in the middle of starting. The problem was that they have a JavaScript interpreter which allocates something which was not there before and so it's really easy to get into it. Misconfiguration is not only some cloudy problem other people or big companies are having. We are all facing it on a day-to-day basis. So, yeah, then, yeah, and misconfiguration can also lead to system failures. So if some application or some server doesn't start up and this application is central, for example, like this JavaScript part, this is essential for KDE to function, yeah, then we basically make our complete system non-functional. In, of course, particularly, the problem is even a little bit worse because of the huge variety we have. We have to keep many configuration file syntax in mind and so on and so on. Okay, so standardization doesn't really help. So what else can we do? And yeah, we are all developers here. We know that we can solve any problems by adding another layer of abstractions unless the problem is that we have too many layers of abstractions. But for configuration, we are nowhere near of too many layers of abstractions. So the usual abstraction we currently have is that we have bits in a configuration file. So we are still, basically, if configuration would be a file system, we would be on the lowest level. We are still talking about if the equal sign means that key is a value or if the hash sign or the semicolon introduces a comma. So basically, we are on the lowest level we can get. So I propose to not even, to not only go one letter up in this abstraction, but to go two letters steps up. So, yeah, if the one up would be to introduce a registry. This was basically the proposal 13 years ago. And the problem here is that we need a huge consensus of how this registry must look like. And it's impossible to get such a consensus. So instead, I propose we should specify how the configuration access works. So we specify I am this application and I use this configuration file in this format. So this is going two levels up. So then we need only a little agreement. The agreement is basically only how the specification looks like, how to describe the system, but not how the configuration files look like. Okay, so, yeah, a little bit over time. So we have five parts in this talk. So the first one will be a little, I will shorten it a bit. It's a popular ways which are present in configuration today. Then we have how to reduce or hide configuration and why this is a problem. And then I will talk about what I said in the beginning about the configuration design and how it affects modularity. And in this fourth part, we will also have a demonstration of the specification language. And, yeah, if there's time, I will go and show a little bit about tooling which is provided by Lector. Okay, so before we dive into the content again, so please say, so it's a little bit abstract what I'm saying. So it's important to not lose you. So if you have any questions, please ask them immediately. And there's also a link. So you can also post questions there. If you're too shy to ask, he will look there and relay the questions to me. Okay, so what we did to find out what the problem is. So we used several methods to find out. The first one, we made a large-scale questionnaire, where 672 persons visited the questionnaire and 286 gave answers. Is someone here who visited this questionnaire? Okay, so yes, only one. Okay, so thank you a lot for visiting it and participating. We also had to check if what the people are seeing actually reflects the reality. And then we did also a large-scale source code analysis of these 16 applications which are shown here. And we looked at all get-end occurrences which are in these applications. Then we also did case study and user studies. And this user study will be presented tomorrow. There's also a poster outside showing a little bit about this user study. Okay, so the first insight was about popularity in configuration. So this might be not surprisingly for you that command-line arguments and environment variables are the most used ones and the most popular configuration systems today. This was completely surprising for the research community which had no research whatsoever about these two most popular ways. And in the source code we found 2,683 coal sites of get-end. So the first abstraction, I can't go into the details unfortunately, but Electra introduces namespaces. And these different kinds of configuration sources like for example a command-line and environment they get different parts within these namespaces. And whenever, so we basically have a kind of file system with new names and whenever this parses start with slash it basically means consider any one of these different namespaces. Then we of course use the well-known key value, abstraction. So it's widely known that key value is a common structure among all configuration files. So yeah, and here we have two examples of basically the same content once in any and a little bit more of both with XML. And then we have some other formats which are supported by Electra. So yeah, this abstraction is quite nice, but it is not enough because it doesn't deal with complexity and it also doesn't really deal with this user interface problem. So to understand the complexity let's look at the trend in number of configuration options we have today. So Tierni Xul is one of the other misconfiguration researchers. He also advocates a simpler configuration design. And in the study he found that there's a clear trend towards more and more configuration settings. So and 52% of these configuration options are rarely used, if at all. And yeah, so I'm sure you know but to make sure what this means, for example if a patch in 2006 has about 300 options and here in 2014 it already has 600. And this is a double of configuration options but of course it's not only a double of the configuration space. But we have 2 to the power of 300 to the power of 600 if all of these variables were bulge and they are not. So we have a huge explosion in the configuration space. And my own research focused on get-dance because it's more popular and it's widely standardized and I could also confirm the same trend. So the trend is not only limited to configuration files. So here we have Firefox and Curl. And yeah, for example here in Firefox we have nearly 1,500 basically hidden configuration options which we can use by environment variables. So why not simply get rid of rarely used options like suggested? In the survey, as in the question survey we asked the question, why not remove the options? And 30% answered that configuration should not be reduced at all. And I found it really surprisingly because this was a multiple choice question. And if you choose no you could not see any of the yes reasons. And there were many good yes reasons or I thought it would be a good reason to say yes. For example, to simplify a code maintenance or because the options are actually not used. So we are in a quite problematic situation. So users and admins are usually overwhelmed by the many options they have. They basically are searching for the knob they need and these are only very few. But developers according to the survey seem to be reluctant to not remove a configuration. So first there's a relatively clear point which I've violated I think many others do also violate it. We need to stop adding configuration options just because we can't decide in the implementation. So that's really not a good reason. We should see configuration as a user interface and if there's no user who wants to use this we should not expose this configuration option. And then there is a second part to it. So we need a way to derive default configuration and this way we can break the controversy whether or not to reduce configuration. So some software already showed us in really successful ways that this is possible. For example XRWA or GPS daemon. Many are maybe old enough to know in XRWA several years ago you had to write an XRWA file and specify what the resolution is and so on and so on. And at the moment this is not necessary anymore. XRWA simply detects which hardware you have and does some best effort which works in nearly all cases. And GPS daemon is even more extreme in here. But according to the study we should also keep the possibility overwrite individual settings as needed and XRWA allows exactly this so you can still write a configuration file if you want to. So we really need to go beyond XRWA and GPS daemon so only hardware detection but we need to generalize this lesson. So wouldn't it be nice if shortcuts work across different platforms like GNOME and KDE? And wouldn't it be nice if applications automatically switch to the defaults of the desktop we started? And wouldn't it be nice to have a more consistency between the applications? And according to the survey we also agree that good defaults are really important. So 80% said we really need good defaults. Why are we not doing it then? And the reason is really simple. It's too much effort to do it. So it is difficult to ask other sources to get better defaults. There's only little software which actually does this effort. For example in LibreOffice you have a desktop detection code and it's nearly 400 lines of code just to detect at which desktop we are running. And of course we can't put this burden on every application out there to detect everything ourselves. And here there are also other questions. For example to detect network bandwidth and so on and so on. These are things which are not easy to do. So it is too expensive at the moment to do. And I suggest that we add a specification for a configuration and in this specification we can describe defaults. And these defaults are not only some hard coded value but are something derived from the system we are in. For example which desktop we have, which hardware we have, which operating system we have and so on and so on. So we also introduce a concept of black ins to implement this specification. So dependencies which might occur, for example for desktop detection, you might need some x-server specific parts. So these dependencies are not your concern but the concerns of the black ins implementing the specification. Okay, so one of the key features here that the specification can be written like its configuration file. But also always think about it. It's also in the demonstration afterwards. So we have different meta levels. So one meta level is the configuration file we have and the other meta level is which specifies which configuration files are possible or if some value is absent what would be the default. So we always have these two meta levels. Okay, let's go to the configuration specification and the demo. So let us remind us of why we think that configuration needs some special treatment which was the misconfiguration. So not every configuration option can be derived. For example we need to tell the system which IP address we have. So sometimes you require manual input and manual input may lead to misconfiguration. So the state of the art is to encode the specification, validation, transformations within the source code of the application and with some luck the applications that detect problems which are in the configuration file. Often they do not. So there's a lot of research where it is not detected or too late detected. But actually that's not enough to detect the problems at the application start. What we actually want is that the wrong configuration never ends up in the configuration file. We want that already at persisting the configuration files where the wrong values are detected. So Yoda basically tells us if I'm allowed to paraphrase him separate the specifications you must. So we propose that such validation code should move to the black-ins. So we get this picture here. So this is not a configuration file but it's a specification. So one of the important part is amount point basically means that everything below this hierarchy so with the slash we build up a hierarchy should be in a different configuration file. And here we say everything below here should be in etc. hosts. Then we can specify individual options. For example we can say and we will also see in the demonstration afterwards that the local host it's not only an IP for four address but an application might specify or that's an address but the system administrator might additionally want to be sure that it's some specific in the local host network address range. And so the specifications are modular and extensible and so there are basically no built-in specifications but everything is written in black-ins or by external tools. So it is, yeah. Okay. And then we have also different generation techniques which of course not, yeah, maybe a little bit later future. For example we can also generate codes out of these specifications or more practical we can generate auto-completion. So if we specify which command line options we expect we can generate files which auto-complete what our tool accepts. Then we can also generate GUIS, WebUIS we'll show you an example later there. Of course documentation and we can generate all the validation we need in the persistence layer. Okay. And we will show this now to you in a demonstration. Okay. Okay. Let's start with Hello World. So, yeah. Here below we see there is a configuration file we are watching. That is example.ini, like you see here. At the moment this configuration file is not present. So now we do a KDB set of this Hello. Okay. And when we do it we get a section Hello. And, yeah, let's continue with another set so that we can also set a key value. And now let's say Hello World to Belgium at Fostem. So, and, yeah, we have written this down in this configuration file. And, most importantly, we can also introspect. We can also query what is inside this configuration file. So when we get this World, Belgium, we get this value. And what's important here and what's one of the goals of Electra, that applications see the configuration exactly the same like an administrator see it. So at the moment there are different passes involved and not everything is so easy. The idea is that we see completely the same things. So we can also remove this option. So, okay. And, yeah, so it's now gone. And we can, now we are at a different meter level. We can specify which default should it have if it's not present. And so now we have this setMeter command which says the default value should be default at Fostem. And if we get this value, you know, even though it's not in the configuration file as you see, when we get it, we get this default at Fostem. So what we have achieved here, we have it externally to the application. So it's not encoded within the application. So all tools also use the correct defaults. And it's not anything only in the applications anymore. Okay. And so now we go a little bit into the concept of mounting. And so let's list which plugins we have. This can be a list. So we are searching now for a host plugin because now we want to manipulate the et cetera host file like it was shown before. So yeah, there are many here. At last we see storage. So let's list storage plugins, whatever they are. And here we see, for example, oh, there's a host plugin. Okay. Let's try out. Let's mount with this hosts. All the card will be in for hosts. Then we get some information about a host file. And here is a tutorial. It says how we can mount it. So there are these recommendations. Okay. Let's try to mount it. So there was a copy and paste error. We did not use sudo. What's important here is these specifications are a part of user and et cetera. So you need to have a sudo right. And it even suggests to use sudo for mounting it here. Okay. So now we will watch the hosts file down below here. And with this mount command we have connected our configuration system with this file. And now we can get the local host IP address. We can also get the IP for six address. And then we try to get a foster IP address which is there. Okay. And we also see we have tab completion even for keys which are not here because we have a specification of what should be there so we can complete and the tooling can help us with what should be there. And so now let's do what the CAE hackers also try to do. Let's set some invalid IP address like we had in the example before. And here we see that it's not a valid IP address. And this is actually checked in the plug-in with operating system facilities. So it gets its get hosts by address. And it says that this name or service is not known. So we cannot resolve this to IP address. So we can have really sophisticated checks much beyond what we could do with a regular expression or something like that. Okay. So now let's do a valid IP address again. And now we look a little bit at the specification. So we unmount this host again. And then let's see how the specification looks like. So it's exactly the same I've shown before in the slide. And so here we can now mount the specification to the specification namespace I talked about. And then we can apply the specification on the system. It's currently necessary for mount points. And so now we can use the Alice meter. And we see we can also introspect specifications. So the tooling could actually look into the specification and tells us, oh, this is an IP for four address. So in the GUIS you could actually have checks for what this is. Okay. And then let's set this local host wrongly. And here we have the simple validation check which could be added by administrators afterwards. And this is the message. This local IP must be written within this particular network. Okay. So anything about the demonstration? Do I want to see something more? Okay. Thank you. Then let's go over the rest of the presentation. Okay. So basically we get three features. I repeated a little bit here. We have an introspection of the whole system configuration and also of the whole specification of the system. Like the elephant looking on the world, we can introspect what we have. And we have a reusability. So plug-ins can be used anywhere in the system. So you write a plug-in to check network configuration once. It can be used by every application. And we have a system which is much more safe. So we prohibit that any invalid configuration, so everything which does not adhere to the specification, reject such configurations. And we achieve this integration we have. So Electro does not introduce some new technologies, but it basically allows to connect these different technologies. So for example, if you have a GNOME application, we can simply say in the specification language, if you are running in a KDE desktop, please use the shortcuts which are in here. So we have some external specification where we can connect all these applications together. And we only need very little consensus. So one distribution could say, I want to connect this, and the other distribution can say, no, I don't want it. I want the status quo. And we can use all the brilliant parsers and validators we already have here in our system. Okay, then a short about how to electrify, so how to use Electra. So there are three ways to do it, with a degree of how much you like Electra basically. So the first one is, if the application has nothing to do with Electra at all, so we have a completely unchanged source code and an unchanged binary. We can use interception techniques, so we can use preloading mechanisms, and we already did this for get-then and for configuration file. So the application thinks it gets its own configuration, but in reality it gets a generated configuration file from Electra. So if the applications want to use Electra, they can write a specification. And in the specification, and they can also link again these bindings they want to use. And the third option we have is we can also generate source code. And with this generation techniques we have the advantage, it is actually statically guaranteed that every configuration option which is used must be in a specification in the type as specified. So for new applications I would recommend to use this. In all three cases we do not have any configuration logic inside the applications, but instead we have the logic within the specification. So you do not need to change the architecture of your application to use it. So now let's strive in the last minutes about some things what Electra actually is. So Electra is a small and fast library. It does not have any dependencies. It is completely written in C. It does not have a demon. So the security is as it was before. So it does not matter if Electra is used or not from security perspective. So applications using Electra without specification work as before. So they get their own configuration file and write to their own configuration file and basically the rest of the system is not affected. So it's a non-invasive change. So there are many language bindings at the moment. For example, Shell, Lua, Python, Java, Ruby and Haskell. And these language bindings cannot only be used within applications to use Electra, but you can also use all these languages to implement new plugins which do this logic and do this validation. So if you have fluent in some of the language, you're welcome to contribute. So there are 80 plugins currently used. Best you look at the homepage to see what we already implement. And there are some things you can do with the KDB command. We have already shown you before. But what's really important here is that it's nearly trivial to write new tooling like this. So, yeah. And then we also have web UIs and graphical user interfaces. And in the web UI, we actually read the specification. And if the specifications say, oh, that's a Boolean, then we get checkboxes and it's not even possible to input something wrongly. So that's because of the introspection we allow of the specification. There are already packages of various distributions. Not all of them are official. In the first row, they are official of the distributions. And I really thank you for all the distribution for the maintainers. Maintaining is really a hard work. And Electra is already used widely commercially. And so that's also the reason why I'm here. So the whole reason why Electra exists from the beginning to now is only to support the Floss ecosystem because I think that's really a key point of usability that all of the box behavior works as it should. And I think here we really need to catch up. And in the Floss ecosystem, Electra is not used as much. So it's basically only used in the Auronauts color management system. And in progress are these open ICC standards then LCD proc and machine kit. So if you like for your project, you're welcome to do and you also get the T-shirt as said already. And for administrators, there is a talk tomorrow. So it's already used also as an administration facility. So who is working on Electra and it's not only V2 and unfortunately not everyone is on this picture. It's hard to get everyone on the same picture. So here we have a list of the names who are working on Electra, but unfortunately we don't have the time to go into these topics. But yeah, we are basically doing the hard stuff which is sometimes not so fun like some optimizations. But there are also many fun parts like implementing configuration parsers for example. And yeah, so we basically left these fun parts for you. So we are at the moment we have a very key focus that we want to publish 0.1 Electra. There are different parts relevant to it. One are these fail-safe user interfaces already talked about. Then we want to improve the type system and then also optimization are important. For example, one is you might have asked about so if you are using so many different languages in the plug-ins, it doesn't the performance suffer. And for example, there's one optimization and M-Map cache which basically if you have just passed the configuration file before and did all this validation we just get a result. So the conclusion is what we want to achieve is a really great out-of-box experience. So we want to calculate better defaults using the specification I propose. And of course it's all about sharing these different options. And yeah, and I think that with the specification configuration settings are shareable across the growing communities we luckily have. And we need a language to communicate between these different communities. So for example, if you look at the free desktop there are endless discussions which configuration file format to use and I think we really should stop these discussions and use a language which is appropriate for this kind of problem. And so we really want better tooling to edit configuration files. So Electra is not some magic box which automatically will resolve all configuration file programs. Of course the design is still so which configuration options an application has is of course still done by the applications. But Electra builds up a framework which enables us as a community to better face these problems. So here are some resources. So please join the new configuration community Floss never had. So please drop an email or contact me so I will be around here. So don't be shy if you don't know how to talk to me just ask for a sticker. We have plenty of sticker and can give you one. So see you tomorrow at the talk at 12.30. So thank you for your attention. Are there some questions? Thank you. Jack. Hi. Thank you for your talk. I have a question. So this all assumes that configuration is just relating to file formats. If you believe that configuration is more than just file contents and file formats is there a way to extend this to specify other things and other types of states? Can you repeat the last part? If you believe that configuration is more than just what's in Etsy files and so on is there a way to extend this to support other kind of declarations of other states? So declaration of states that are not just in text files. So states? To other states. I'm not sure if I got this question. So basically, you mean if there are configuration scripts or which kind of states? Okay. But basically we tried with many configuration file formats. They are easy to integrate. So we did not have any problem. Okay. I'm very confused now to be honest because I already heard about and tried AGs or how that should be pronounced. The library for parsing and generating configuration that is probably behind LibElectra and now I'm not sure what really LibElectra is because from what I remember from my tries with AGs it looks similar or the same. So what is different between AGs and Electra or which part is part of AGs and what Electra? So it basically has nothing to do. So AGs or AGs, it's basically a way how to implement configuration file parsers and that's not really a business of Electra. So Electra can use AGs but it actually does. We have a plugin using it. So the idea of Electra is to write the specifications, how the default values look like, how the configuration design looks like. And AGs does not concern about this at all. AGs only adds another parser. So AGs actually introduce one more way how to parse configuration files. You do not get disunified ways. The idea of Electra is that applications actually move their parsers to plugins and it should be the only way to get configuration. So AGs adds one more parser which can be used by different tools. When you show it, getting some variable settings to configuration files this is exactly what AGs do. When I try it, the interface is the same. No, it doesn't get you the few the application has. So it parses the file and gets some new replicas.