 Okay, let's welcome Martin Kraft. He will talk about recursive inventory management and this will be a subject that is interesting to all of us who do System automation so welcome Martin Hello, is this good with microphone excellent Yeah, I'm here to talk to you about recursive inventory Management, and I don't assume that everybody of you knows what I mean with that just from the title as As a matter of fact, maybe I can say up front before I go into my slides that the name recursive in this is Born out of a very very long night in Switzerland actually And I should have chosen a better name, but I guess it's stuck. So whenever you see recursive in the Following couple of slides you can think of hierarchical if you want Back then when I was implementing it I did it recursively. So that's why I called it recursive But we shouldn't expose too much of the internal implementation designs, right? anyway I'm going to be talking about configuration management and because that is also a name that was used for some time For what is known now as version control. I Specifically want to say that this is about system administration. So if you hear from version control, then actually that's not the right place I'm configuration management as I understand it as most of you will understand it I'm sorry Thomas that phi is not on there And is something to do with one of these or some other tools you will have probably seen Possibly used some of these tools at some point in time. They are listed here in somewhat order of increasing or decreasing age So see if engine predates pretty much all and one of the latest newcomers has been ansible and salt And I'm actually going to not be talking about see if engine BCF G2 and chef But only about puppet salt and ansible and actually not even about those But I can tell you that what I'm about to tell you works with those three products When I talk about configuration management, I also mean system automation. I mean that My assumption is that every single system administrator is lazy or if they are not they should be You should be able to do something once and not have to repeat the same task all the time As a matter of fact if you did the second time then you've done something wrong That is sort of the mantra of system administration and over the last couple of years I have noticed that there's a very strong divide. Well, maybe it's not that strong But it certainly divide into two separate approaches to system configuration management or automation The latest type has of course been the cloud and we see that this conference is already very cloud Centric in some ways a lot of the talks are about clouds So obviously I had to put that onto my slides. Otherwise, it would be completely off-topic But there is also and I beg you all not to forget the sort of classical approach to system administration If I identify two approaches, I better go ahead and sort of like separate them for you Because maybe I'm the only one that perceives it that way And so I just kind of just want to use the opportunity to give you an idea of what I mean the cloud is sort of What what exists when you have buzzwords that could range from unprecedented ease of system optimization to orchestration of course cloud is all about scalability And because you might have a heard an example yesterday used you might know that your television ad is about to go live So you need to be able to spawn very quickly ad hoc provision very quickly a thousand new notes The one thing that you know about all these notes is that they are going to be homogeneous They're all going to be exactly the same more or less, right? But they are all web servers that are just designed to handle the requests by your potential customers and generate you money System administration classical system administration is a lot older than this concept of the cloud and System administrators are as you all know those grumpy people that are running around in the university building Actually, they're never running around right they enter it early in the morning or leave early in whatever You never see them and when you go approach them you get a bf bofh excuse usually right when When the system doesn't work, but fundamentally What these system administrators are doing are they are maintaining systems that have some sort of longevity so these systems are installed at one point in time and Ideally run for a very long time until the hardware breaks at which point you try to move that exact same system because you don't want to Reconfigure your LDAP and mail server at this point after the hardware crash You just want to have the same system up So you're dealing with longevity across heterogeneous systems. You have your mail servers You have your web servers. You have your computation cluster. You have whatever, you know better than I do in your use cases The real difference between The cloud provisioning and the classical system administration is the following We use the classical system administrators use themed host names whereas cloud provisioners use WW001 through 999 so a Little bit of my mission today is to bring back the themed host names. I Just put that in there laziness I already Mentioned and I did have unprecedented ease on the last slide Obviously, if you want to spawn a thousand systems and you pay by the minute then it should be really easy and quick to do Laziness is somewhat different from that laziness is you don't have a thousand systems You have 10 but they give you nightmares or wake you up at night or actually cause you to stop your vacation early and All of that would be much easier if you could just lazily press a button on your smartphone And like from the Caribbean islands have it all be fixed I'm not promising that that's gonna work and There's also The concept of orchestration and classical system administration. I'm just mentioning that here because I you know I don't want to rule out that we don't orchestrate if you have a nagios Installation or munin or even post fix with client certificates You need to know about the other nodes on one hold node So in some ways you would like to make sure that the mail server will accept mail send with a certain client certificate Before you tell that client to send mail with that client certificate Obviously if it's five minutes plus or minus and some things don't work That's okay for the use case. I guess you can do something against that if you want But we also do orchestrate we as in the classical system administrators so What I'm going to be talking about today it comes very strongly as you may have noticed from my speaking from the perspective of the classical system administrator But it will Potentially if I do a good job or if you take the time to look at it suit your cloud needs as well So it is not something that is exclusive to classical system administration It might actually be something that revolutionizes the way you think about and Do system administration on your cloud? I think one of core differences between these configuration management systems that I've introduced puppet and ansible and salt Which also relates to the distinction between the classical system administrator and the cloud provisioner? I should invade actually The cloud provisioner is in my opinion the one that thinks about system automatic Automation in a way of targeting configuration at nodes Whereas I think that the classical system administrator thinks about classifying a specific host now the distinction between those two is not at all clear-cut and I'm really hoping for your patience here and for your understanding that you will just let me get by with us because I'm not willing to answer questions about terminology But let me give you examples of what I mean When you do cloud provisioning with the tools such as salt and ansible These are the latest newcomers to the game and they are very much targeted at the cloud and whereas puppet sort of like Existed before the cloud came and then try to Implement cloud-like behavior or features into its feature set When you when you come at the cloud provisioning problem from these tools Perspectives then what you're doing is you're saying my mail servers are expected to work such and such Have port 25 open on their firewalls for instance and then you say this should be applied and Obviously I have my systems or cloud somewhere. I should have made that a cloud. Damn it You have you have your machine somewhere out there and then you say I I will target the configuration that I expect from my mail servers to the following IPv4 network. I Will target my Debian specific Configuration to all of the nodes that export the fact I am Debian or has some sort of Debian related fact Fact just very quickly is something that is generally Generated on the node and then export it back to the system so that you have some knowledge about what the node is like Which system is it running which which version? What's the host name? What's the SSL certificate fingerprint? These are all facts that are exported to the central location and use there and what we're doing here is that we're actually relying on these facts To target behavior at nodes or for instance if your domain name ends in Zurich corporation Then you might be you might you might have all of the Configuration that is specific to being hosted in Zurich at these nodes that fulfill these criteria Now I think that is backwards Maybe that's because I'm limited to my classical system administration thinking but imagine these questions What classes does a specific node belong to which behaviors does a specific node or isn't a specific node expected? to have and You can only answer that by looking at all of the data generated your entire inventory and then walking it back So it's it's kind of inefficient on the other hand Of course, you will see that the other questions that you can easily ask answer with the system of targeting nodes are not as Trivially answered with reclass, so it's really just the opposite perspective of going at it but Another thing that I find very disturbing is that the behavior is actually dependent on data on the host and While I can see a lot of benefit in that because you do Infrastructure of sorts I still think about system administration as Actually being about keeping an inventory of all the stuff that you have and sort of centrally defining the deep behaviors that you want From your systems and then making sure that the software Forces those on the systems rather than asking the systems first whether this is what they want It I think it gives you much less space for errors If you can look at a central location and know that when this is enacted all of the nodes are going to be the same rather than having to worry about Data that is exported by one of these nodes so How I think about it and this is not going to come as a surprise and To many of you and this is also Probably not even that different from what I just showed with targeting as I said the differences is not clear-cut What I what I think about is that when I have a node that's called blue dot example dot work I want to say this node has a certain number of classes this node has this behavior Rather than saying this behavior should be targeted at this node So I say that this node belongs to the classes male server NTP client for instance and host it at Zurich And this is very much see if engine Inspired that's what the first one did the first configuration management system that I noted and I got to know about it through fi Actually, and I really like the idea and it just stuck. I couldn't get rid of it anymore and This is this is very much like definition of behavior and controlling the software the configuration management software that you're Dealing with and I'm not actually going to go into that at all today So no puppet no salt no Ansible specifics here And but one of the problems that comes with all of these systems is Where do I keep my data? Where do I specify the values for the parameters that these systems expose because they are not all equal? They are as I showed earlier part of a mostly heterogeneous environment. I Might go ahead and say That my NTP clients should all just benefit from pool dot NTP dot org and specify the server to eat that But then for some reason I mean ideally that's a distributed system, right? I might want to say that actually when the server is in Zurich. It should use a different value This case only the servers that are in Zurich use a different value all the others fall back to the default That's not rocket science Same example here. We also have a second node called white Just hosted in Munich now. Let's look at puppet Let's look at how we do this Approach with puppet. I'm sorry that I just noticed that the NTP server is actually not zero dot De dot pool at NTP dot org, but red at my own infrastructure I hope you will forgive me for this And my point here is that I want to tell puppet that the server at blue should be using an NTP Should be an NTP client that is why I include the The NTP class here or the module which provides this functionality But that it should actually not fall back to the default, but should use red example org as the NTP server now If you have five systems, that's fine if you have 50 It's probably still fine if you are not quite sold by that concept of laziness that I earlier introduced But if you have a couple of hundred servers, then this is not going to be okay anymore because suddenly you will find yourself with an Site description that has a lot of these stanzas and they are just all going to be the same so in puppet You can actually factor out some of these behaviors into Something that is not really a node, but sort of an include if you want to think about it like that Also programming programmers will understand what is meant here by inherits Now in the common nodes all nodes that inherit from this node definition There is a default set For the NTP server and it includes the NTP class and just by the mere fact of inheritance Blue dot example org now also includes an NTP client configured to use that server But blue dot example org is in Zurich and we want to be using the Swiss specific. They run better, you know so I go in look the NTP server for blue should be red, right? No anger That does not work. This unfortunately does not work in puppet in addition. They're multiple inheritance So what I mean with multiple inheritance the programmers might run away now What I mean with multiple inheritance is that obviously you're not just not all of your hosts in Zurich are also NTP clients ideally and you don't want to be creating a class for every single combination of potential parents that you have so it's really nice to be able to multiply inherit from from different behaviors and Puppet unfortunately doesn't do that In fact puppet actually says in their docs specifically that you should not use inheritance So what I just showed you this what what seems sensible to many of us to factor outcome in behavior Rather than having it all in one place multiply multiplied a thousand times Puppet actually discourages this in the documentation now. I'm not trying to take a stab at puppet here There is Puppet was one of the very first Tools to actually say if you want something more complicated than the simple note definitions then use this functionality that we have over here But I just showed in terms of the overriding the NTP server it does make Parametrization unnecessarily difficult simply because puppet only ever knows one instance of the variable NTP server And unless you want to go into scoping nightmare You are basically stuck with Reimplementing a lot of the data or multiplying a lot of the data and generating a lot of redundancy which as We all know is going to bite you at some point in time So let me Before I go on let me go a little bit into the what I consider to be system of optimization Principles and a bridged version because I don't want to have any questions about what I'm leaving out and suggestions that this list is incomplete I know it is These are the important things for me I want to be centrally in control of something and I want my data to be versioned with Versioning control system of your choice. I Need to have Parametrization I need to be able to say that NTP clients. That's a very simple example, right? But think of postfix. It's like it comes with 500 parameter value configuration Possibilities and some of those are actually sensible to parameterize because they are going to differ between the code and I have seen too many Installations of puppet where you actually have in the module code in the one that is responsible for Installing your NTP server or postfix special casing based on the host name and that just hurts Not right there because it makes it work But at the next upgrade or when something changes and suddenly you have all this redundancy in the code That is going to come back and bite you. I Think that system administration should be about no redundancy do it once do it in one place only and be able to always find That place where you did it very easily without thinking too much about it at three o'clock in the morning after being woken I Think that this idea of using information from the node such as the IP address or the free memory or the SSL Fingerprint is actually very good, but I don't think that it ever should be used to Steer the behavior that is going to be applied to a node Which is very difficult to say because obviously I want to describe my infrastructure independent of having to know Which one is a fedora server and which one is a Debian server and if I you know, I need to kind of ask those systems that that is at least the fundamental motivation between behind this This functionality, but what I'm about to show you Is a system by which you actually take it away from puppet you don't say puppet You know how to deal with all the different distributions. I just have to tell you to install a package and you'll do the right thing My approach is to to say I know that this host is a Debian node and I'm going to treat it as such And if that ever changes, I think I have a different problem So reclass as I said a misnomer of sorts you can think about it in terms of hierarchy, but the name here Ah has Been taken up before I realized that I misnamed mine Um It stands for recursive external node classifier external node classifier is a term that comes from those configuration Management softwares and it basically just means instead of requiring you to specify your infrastructure in note groups or in note Stands us as I've just shown in puppet. Um, you have an external data source and classifier while I try to Make a statement about my use of classes in this The CMS uses reclass But asking it questions such as what applications what modules should be applied what applications should a role have How does this note differ from all the other nodes that have the same application which is parameters and Which nodes belong to a group because essentially I want to be talking to groups of nodes and not just individual nodes and not always only to all of them at once For the system administrator It will obviously as all of these tools to allow you to deploy and manage site-wide configuration changes It will allow you to say things like upgrade all the nodes that are tagged with sable Update the message of the day in Zurich because of a power outage and fetch logs from all of the hosts tag mail Server because the BND or whoever secret service has actually knocked on your door Um Important for you to realize before I dive into the details of reclass is that reclass just assembles and provides the data It doesn't do any changes. It doesn't your hosts don't even know about it It is actually just one abstraction layer on top of all the configuration management systems I'll skip that and It is a single data source which means that if you use it with one system, you can use it with other systems There are adapters that interface between these different systems and reclass For instance mode of indication and output. Basically, it's an API definition of sorts system administrator specific API definition by which I mean We call programs and then parse output The adapters that are provided puppet, which is actually not yet provided But I get into that in a second salt and ansible as I said earlier So if you're a user of any of these and you think that you might want to use a different one This is where the slide about Remote execution and configure I'm done For instance salt and ansible they work very well together I mean ansible is very great at remote execution and is not so great at configuration management where salt is not so great at external execution and or remote execution and somewhat better at configuration management the important point is you can use both of them And you keep all of your relevant inventory data in reclass and Those tools just get the data out of reclass and do what you asked them to do So puppet. Why is it not actually in puppet? It was originally written for puppet Because puppet didn't do what I wanted and then I puppet really didn't do what I wanted and I rage quit puppet And I removed it that was the best thing ever tell the new configuration management system to just purge puppet It's the first thing you do get rid of your predecessor It's a good feeling all these Ruby packages and so I rewrote reclass since I ditched puppet and I wrote it for ansible and salt which is what I'm using at the moment and I could really not be bothered to Reimplement it now to all the puppet users who are wanting to use reclass who are interested by this They are a couple. I'm not trying to scare you away. There are a couple of ways to do it Either you can sit down. I'm sure you'll have it done in an hour or you buy me a beer and I'll do it for you It's trivial Salt who have you use the salt? That's almost a representative sample It provides top and pillar data for salt It has been actually integrated as a salt but not reclass itself But the adapter for reclass has been integrated in salt since 016, which is not yet in devian It is an unstable but not in back ports are stable node groups if you're interested in if you if you are Would like to Completely get rid of the top data in salt then we need node groups and so ask me if you want to help here Ansible who have you uses ansible? That's an equally big representative sample size It provides inventory and node information. It is actually implemented as an external script Unfortunately, it does not support this really new feature of batch calls But enough of the sporing stuff, right? I Think that Parameterization to system administration is key if you manage to somehow Remove all of the salient bits of information that differ between your nodes in your infrastructure and Factor them out keep them in reclass ideally then you're gonna have a much easier time Switching hosts around or doing this and that or switching configuration management systems around because all your data is independent of that Of course, we should only Do the sensible approach don't special case, please Reclass even allows you to keep your parameters modular so that you don't have to in reclass duplicate information and Ideally non-redundantly define your data in one location. So let's look at what it's like Yammel is probably known to everyone here. And if not, then I trust that you will immediately understand this Here's the definition a bridged definition of blue, which you've already met It is a post-fix node. So it has an application of post-fix defined And it is an NTP client and it has a server defined Which is the canonical default server? What this means is that it's independent What you see here is independent from what your configuration management system ends up doing Applications might be called modules. They might be called states and salt for instance, or they might be called I forget what it's called an ansible. It doesn't matter But you can understand that this is actually going to translate Into a host that has post-fix installed and a host that has the NTP client config installed and configured to be This value here, but this is not recursive What we have in reclass is basically two directories. We have Yammel files for nodes in This directory and then also Yammel files defining classes in another directory and And the recursive part is that nodes and classes may specify other classes to inherit from as many as you want You can think of these classes as tags. I very often do Because you might want to address all your systems that have post-fix installed So just simply talk to all the nodes in the class post-fix and It does smart deep merging on return from a recursive Descent walk I had to show up a little bit that I didn't computer science student here by which I mean that obviously there's going to be some data replacements going on and This is implemented in Python and also Yammel has a merging feature, but when you have a set of Values defined on one host and the set of values on another host and they all happen to be kept inside a dictionary Inside of a hash then obviously you don't want to just replace The existing hash with a new hash and forget all of your existing information But you want to merge the two dictionaries and the same exists for lists and the same exists for some special cases of scalars Yeah, what about order for off the text very good question have slides prepared for that But let me first show how the next step would be so here's blue and We have a class down there called common and we now have ntp defined for all nodes So it's in the common class and other than that I changed me. Oh, yeah It's now classes mail server not applications post-fix anymore But I'm sure you can see the direct relation between what I just showed and what this is now And obviously what you would expect this is the value that gets installed as the ntp client on the blue host In general and this is a very very important Baseline assumption that you can rely on in reclass, but that you also need to understand When you have something that is more specific that that talks about less nodes Then it overrides the data that you specify in a more specific less specific case for instance all your Debian at stable Nodes will be able to override parameters that were defined in the Debian common class. I Hope that makes sense So in this case now and to get to your question about the ordering We have a new class here hosted at Zurich and it defines the ntp Client or server to be the Swiss specific one But we also know that common This class up here already includes the ntp server and already defines the default and as a matter of fact We might have multiple of those But through this hosted at Zurich simply because the class appears later in this list it gets to override it There's a well-defined order Even if you don't want to worry too much about this because all you have to do for instance here Let's change to a different program. Let's say that in general. I want my SSH servers to have no root login allowed What about a backup client that needs to allow root login or you can find other solutions for sure But what this is one of the ways to do it? So now you define a class backup client and you set in it You override in it permit root login with a without password and because the backup client comes after the SSH server Expected behavior, but this might not be enough for you So all you have to do now is that if my backup client actually relies on SSH for transfer Depend on the classes on the class for SSH server, which will now cause when when the backup client is hit to walk through the data structure and Do everything that is necessary including the merging for the SSH server? Unless it has already done that right in this case. It would just simply say I've already seen the SSH server I don't need to do it again It would actually be bad if I did it again, right? No, it's only on first occurrence And then there's also something called parameter interpolation and that actually wrecked my brain and really made me question whether I am a CS student Because inside these this entire data structure that is being created when you merge reclass data Up the tree or down the tree should I say? You now can in reclass reference any other key So in this case, for instance, I might want to have message that is a reference to Floyd in My message of the day and then in diamond One of the hosts in my infrastructure. I simply Include a little message and that gets automatically merged as well And this is looks a little bit trivial a lot of the other configuration management systems do it themselves already But then it's either single pass So if you have two references, then you're it gets stuck with it with another reference or it converts everything to strings Which is not necessarily something that you want you might want to have a list after all So future work I'm almost at the end of this I need to package reclass I feel really stupid I come to depth cons to give a talk about reclass and it's not actually in the archive But that should not be a problem It's a Python module and you can very easily pull it from git at the moment and install it with easy install But I'll try to work on the packaging this week as well I was thinking and also during depth conf you could actually integrate this with preceding and Di so that you could provide your data even before installation and then have the same data Reused later on when you are done with installation and you pass over to configuration management In terms of cloud you if you have your name, sir, if you have your server's name WWW 0 0 1 through 9 9 9 you don't really want to create a thousand identical YAML files that all inherit from the same class So I'm thinking there should be some sort of policy classification where you can say that all of the nodes that are basically wild cards on on the node name That should be implemented membership lists. That is the concept of Postfix and client certificates what I mentioned earlier or that you want your Nagios clients to be also monitored by the Nagios server A lot of the configuration management systems address that by having some sort of wacky Communication between nodes or a central data collection called in puppet for instance store configs Which I think does not belong there Even though I agree it is nice to think about that Nagios only starting to monitor a certain Node as soon as that node has actually started the client package Yeah, sure, but if 15 minutes later through the when next time the configuration management system runs I will know that now definitely all of my nodes are in the same state. I can just statically tell The Nagios server about all of the clients, but obviously I don't want to have any Redundancy in the data set so it's somehow reclass needs to figure out membership lists of classes and be able to pass them on as parameters Haven't figured that out yet Other data sources potentially in terms of performance YAML files could become a problem They are after all opened and read and closed for every single run of reclass at the moment You could put this into a database There's actually already a plug-in infrastructure in place better unit testing without any philosophical debates and your idea here if you're interested by what I just said and you have some Usage ideas, then I'd be very very glad to hear them and with that I end my talk and thank you for the attention And I'm open to questions Can you move back? I think three slices There was where where you said there's a parameter and then yeah this one Down there if you add classes SSH server, is it important that classes is written after parameters? No YAML is basically very compatible with Python and all this causes. This is going to be read into a Python dict and The dict is unordered anyway, so it doesn't matter So so it's it's it's defined that the interpretation of this syntax is that even if the parameter is without password and then the classes SSH client which has Has a higher priority Which you define in classes SSH server or in backup client? This is it's a very good question because it allows me to also address one more time this distinction between targeting hosts and classifying them With the targeting hosts approach You would now say what's the parameter that all of my web servers should get and with reclass or the way that I like To think about things I start here I say I have blue right now in my screen and I want to configure that and so I then start a recursive walk of the tree So in this case I load the SSH server class and then continue to do my work here And it merges the this value no into my tree and then it comes back And then I go to backup client and backup client then merges this Scaler over this scaler so that at the end when I come back to my class. I now have the without path password Set if those two were we reversed if SSH server was after backup clients because you just you know Quickly set up the host and you then went out to dinner or something like that Then what would happen is backup client would be loaded. It would actually set Sorry, no, it would actually see there are classes So then it would go into SSH server. It would then read this Write no into the tree and then come back to backup client and overwrite the no with without password before it then returns to blue That example that work Okay, I'm not sure if I really understand seems to be a bit complicated It could be because it can be a very big tree. This is a this is a very classic example of where a Diagram would have been a better slide than just code On the other hand what you just said this could be a very big tree that made me stop the idea from drawing a diagram because it Was not going to be very informative I suppose I can either try to explain it again or let just let it sink in and ask me if you still have problems because I'm Fairly sure that everybody who has dealt with recursion at some point in time or who understands what it means to even walk the Directory tree on your Unix system recursively for instance using find Will immediately understand what this means When what I would propose I think If I have a very big Configuration with a lot of yeah with a big tree or a very deep tree It would be very nice to have some debug support so I can say oh Which passes are walked through the tree for a certain client? Yeah, that would be nice. So I have this to-do list and reclass and debugging is number four It would have really helped this Especially the parameter interpolation to implement that correctly because it requires you to do a topological sort of all the dependencies between parameters That really kind of would have been much better if I had debugging done first but It is very simple in the end and if you look at the code it is actually very simple And once you understand the general concept once you understand what actually happens when you walk a tree the important I guess the only important thing here that that is a little bit different from walking a tree is that what you're doing is You're doing the action at the end when you come back up. So it's tail recursion and I'm sorry. I'm failing at this It's just too slow Tail recursion means that basically you don't just add to your list when you find a new directory and you enumerate the lists but in terms of Unix file system imagine you had a sim link somewhere under a To a tree that is somewhere under under zed and now when you basically Walk the entire file system. You will see the contents of the tree Reference by the sim link first even though it only appears in zed and what what then is the important part is that you? store what you've already seen and that you go all the way to the leaf and then you do the operations when you Come back and that's when the merging happens and that's when this Specifics of I'm having more specific Data override less specific data comes in that's how it's implemented Further questions I'm happy to walk you through examples and everything Maybe you're already answered in the last slide, but I was distracted. Do you plan to do you plan to? To add possibility to use an external node classifier to a class I mean to be able to extract from a database the set of nodes with some properties which will allow to To do most of the class if I but we will still be it will still be possible to override Most of the things with the class. I mean I already have an external node classifier This is an external node classifier So I don't understand is your question about can this coexist with another external node classifier or reuse the data from another external node Yes, that is something that I recently thought about how to do it's kind of difficult because I Mean unless I want to implement an adapter here for cobbler or whatever you have right Into reclass. I'd really rather have you configure cobbler with salt or with ansible or puppet and then let me get at those data Once they are ready pulled because I don't think reclass needs to get any more complicated on the other hand There are there's a plug-in infrastructure for storage and this YAML FS Which I showed you which is the nodes and classes YAML files and so on you can actually it's two functions that you have to write You can overwrite that and put it into a database or XML RPC to your cobbler server I think that should be possible, but I don't really want to put this into reclass proper at this moment But what I do want to do is figure out how to get at the data that your configuration management system has already collected which includes things like the SSL fingerprints or Then free memory all the facts that I was talking about at the moment You can't use them in reclass. You have to use that in templates later You have to combine the values and ideally, you know, be nice if you have all the data actually available in one place Are there more questions? There's one. Yeah, it's on. Yeah, how'd you handle in? Well, I was gonna ask but I think I figured I You know in your blue example say you don't care that blue itself is an SSH server So you go down the tree backup client and that was an SSH server and then you know So you know that permit root login was overridden in backup client. So that's okay when you walk back up How'd you figure how'd you handle it if? You end up with sort of down going down the tree somewhere at the same level To overrides to permit root login There's no there's the same level, but there's still a well-enforced order in this case Whatever backup client does would overwrite whatever SSH server does and this is I don't think there's any other way to do it And this actually gives you a nice nice control over the system because I often Tag a number of my notes with test which I then append to the list of classes And that means that that actually overrides whatever I set in there overrides everything else And then later on I can remove remove that test class again and and resume normal work Of course, I do all my testing in a testing environment not a production environment never That's a joke more questions good then thank you. Thank you very much