 Next up, we have Greg. Greg is, I would say, one of the foremost community experts on the foreman. And he is talking to me about the foreman's puppeteer. And he is very tired and very sick, so I'll give him some slack. But I'm very curious to see what he has to tell us. All right. So you hear me OK? Great. So just out of my own interest, who already uses foreman? OK. Good. That's more nice. And OK, who just uses puppet? And I'm not using either. Yeah, so it's going to be a slow talk if you do. So this talk's really in three parts. I'm going to do just a little bit on foreman for those who don't already know it, just mainly for the video, I guess, and also the full time. I'm also going to talk about octocatalog diff itself. So that's, what's the other pointer? Yeah, so this bit here. So even if you don't use foreman, there's something in here for you, maybe some of the concepts and the command line tool is useful to you. And then at the end, I'll talk about the plugin that I've written to bring that together. So very, very quickly on the foreman then, because you guys already know it. But again, just for the video, if you don't know foreman, it's a configuration management and lifecycle management tool allows you to take the whole set of operations that you need to act upon for a server. You need to do provisioning. You need to get the disk up and running, whether that's pixie and unattended network installs, whether that's booting an AMI or a digital ocean droplet or a disk template for VMware or whatever. You need to get something up and running. You need to do some one time configuration, get that hooked into puppet or Ansible or Salt or Chef. And then once you've got it hooked into your infrastructure, you want to manage it over time. So that means driving the configuration management, using the ENC or defining Ansible roles to play. You want to get reports back. You want to get inventory data back. You want to talk about whether your infrastructure is performing properly, behaving properly, whether hosts are out of sync or have errors. That stuff can be difficult to see if you don't have a nice front end for it. So we can do that. And we can drive it as well. We've got the ENC. We've got hierarchical parameters, parameter storage, all that kind of thing. And we call it monitoring. I personally don't like monitoring. I like the word reporting, because we're getting that data back and checking everything in sync. And then you want to deprovision. And you want to clean up everything at the end of the host life. So delete DNS records and delete VMs and things like that. So we do the whole lifecycle management. And I guess most of you already know that. So I'm not going to spend very much time on it. But no form and talk is complete without this slide, which I'm sure you've all seen before. It's horrendous. But if you want to know what it means, come by the booth and I'll explain it to you. Assuming I'm still awake. So let's talk about what we really want to talk about today, which for me is about this idea of puppet diff, catalog diffs. But I want to start by just laying some groundwork for you. How do we test puppet today? I just want to get people's opinions here. Who uses, who writes aspect tests? Yeah, a few people. Who actually runs aspect tests on the modules they download? Who uses a VM to test on? Like try and copy a production machine and actually run the modules there. Yeah, you see, I do that. Who does peer review when they change their puppet code? A few people there. So there's a different mismatch of ways of doing this. Who actually just commits it and runs it in no op mode? Yeah, you see, I knew it. I do that too. Because actually, whilst this is fine, things like aspect, no, that's the wrong button, things like aspect, my hands are actually shaking. So things like aspect are great if you're the author and they're great confidence in the quality of the module. And if you're running what we might call a cattle system, where you have 40,000 houses and they're all identical and you never change them and if you make a change, you just rebuild everything, right? That's ideal. But I don't actually think that many of us are in that position. And I can certainly say that I've got crusty old hosts that have multiple roles and I worry every single time I make commits that affect those hosts. Anyone else in that situation? Right, so this is the problem, right? We're left with this really awkward situation. If we upgrade all of the modules, if I've downloaded a bunch of modules from the puppet forge and now I want to upgrade those puppet modules to the latest version, how do I do this? Do I diff them? Do I actually look at the code changes from version 3.4 to 4.0? That's a lot of work. Do I trust that these are good changes? Okay, yes to some extent. I'm assuming the authors are doing a pretty good job. But it's a valid aspect test to ensure that it execs rn-rf. So we want to do some checks. So what do you do? You end up merging it and you just run it with maybe no op. But if you've got a lot of hosts, some of them are going to run it before you finish testing it. Or maybe you take one host out of production and you put it in a development environment instead. Slightly better perhaps. But if I've got variables that depend on the environment, maybe passwords should be different between production and development. Things are going to change again. It's going to be diffs that I don't want to see, things that I don't care about, and I don't want to spend any effort reviewing that. So all of these tools are fantastic, if you're the author. And as we get further down the list, they get increasingly worrying and slightly more sketchy. So let me introduce you to the idea of catalogue. Is this coming and going? Am I losing audio? No. Turn your head to the right, it gets better. Okay, right, fair enough. I'll just, no, done. So let me introduce you to catalogue diffs. I think the key idea here, so for those of you who maybe don't know, Puppet obviously takes Puppet code, compiles it into a catalogue, and then that catalogue is sent to the client in order to execute upon. So we can actually diff those objects. They're serialized objects. You can take them and you can look at what's in them. And there are a number of tools out there to do this. But what we're really important to notice is this is not code anymore, right? We've taken out the code. We've compiled the code. We've dealt with all the facts. We've dealt with all the variables, the branching if statements, the optional switches. We've taken all of the hostname-specific stuff. We've taken out all the whitespace, the comments, the Puppet tags, the things that don't matter when it gets to the agent. All of that is gone. So that's a lot less to think about. It works if you don't have test coverage because what you're actually looking at is the actions that the agent would take. So it won't say there's a diff in these lines because that's what's changing the .pp files. It will say you were going to write a file with this content and this mode, and now you're going to write a file with that content and that mode. So even if you don't have test coverage for your module, you still get to see exactly what's going to happen on that host. So this is really valuable, actually. And if you have Puppet code in two environments and you calculate a catalog for each of those two environments, you can run a diff between them and you can actually see what's going on. Now, as I said on the previous slide, and by the way, I'm going really fast because I'm tired and all the rest of it, so I hope I don't run too quickly. But even if you're the author, all the other tools are great, but if you're just a user, then this is really useful. Even at the very minimum, it's a confirmation of things you think you already know. You can look at the code review, you can look at the changes in the module, you can look at the change log, if there is one, and you can see what you expect to change. And at the very least, you'll get a confirmation of that. But in extreme circumstances, as I'll show in a minute, you can actually end up with a much more easy to parse diff and you can have some confidence of what's actually going to happen on your host. So let me introduce you to OctoCatalogDiff. Firstly, this is not my tool. I did not write this. I found this GitHub released it in October, I think. And it's not the first attempt at a catalog diff. There's one already on the PuppetForge that works as a PuppetFace. GitHub have decided to go a different way. They've done it as a standalone Ruby gem that has a binary and you can just run it. And they've done that because their intention is for you to run this from a Git checkout. So you can run this on your local... If you're using Git to store your Puppet modules, you don't even need to be on the Puppet Master, right? Okay, you have to supply some data, which classes to compile, you know, ANC data. You need some facts from the host, a cache of facts in order to generate that input into your Puppet manifest. But ideally, it's staying away from the Puppet Master. You can keep running it while you're working on the Puppet code until you've got it to where you think it's good and then you can commit it and push it, which is a really, really, really nice tool. And it means you're not loading your infrastructure in any way. There's faster turnaround on your code while you're working on it. Let me give you a quick example of what that looks like. So what I have, if I'm not mistaken, here... I have a couple of modules. This is... So I'm using as an example here. The formant... Is that big enough? Do I need to make it bigger? That's good. Okay, cool. So the formant itself, as an organization, publishes a whole set of modules. And I'm using just one module here, the DHCP module. Now, we recently released Formant 114, and that came with a whole new set of the module releases that make up our installer, and DHCP is one of those. So I have, in the production environment, I have the 1.13 version of the formant DHCP. And in the development environment, I have the 1.14 version. So I think I used the numbers earlier. This is version 3.4 of the module to 4.0 of the module, I think. Sounds about right. So here's what happens if you actually just diff the code itself. Okay, there's a whole new license file. That takes a while. New pixie parameters, new code inside modules, optional blocks, support for Arch Linux, new versions of things, new changes to defaults, change of all sorts of things. It's long. It's really long. I don't want to read that. And I don't think many other people do either. That's useful information as you go through the process of developing a module. Again, it's useful for authors. But as a consumer of the module, that is not helpful to me to know whether or not I am going to have any changes on my DHCP server when I upgrade the module in production. And I can do the same thing for DNS as well, for example, because actually it looks almost exactly the same. I do this for both because I've actually got both in my ENC. So these are really, I hope you'll agree, really complicated diffs. They're hard to review as an end user. They're not going to give me a lot of confidence that my DNS server is not going to break. And if I don't do something like a catalog diff, I have a worry as to what's going to happen next. I'm going to go back to those hacks. I'm going to either move a machine into another environment or I'm going to use no op or something like that. It's going to concern me. Maybe I'm going to do my upgrade to my modules, one module at a time, which is really painstaking and takes you all day, right? But let's have a look at October catalog. Now, I must warn you, there's a lot of options here. Yeah, I know it's huge. There's various reasons for that, and I could go through every single option, but the point is there's a lot of different ways you can get data into October catalog diff and a lot of different sources of things. So very quickly what you're seeing is, since I need to fill time, right? So that's the machine we want to compile for. This is where to get the code from. So normally it works from Git, and I'm obviously not working from Git here. I'm actually on my Puppet Master. So here's a checked out version. So you see from and to a lot because obviously we're comparing things. So we have from, production, to, development, where to get the facts from, where to get the E and C from. So node.rb is very familiar if you're a form and user. Don't use your R because it makes a mess of this. Where's Puppet so I can actually calculate the catalog and add some color. So it's actually not that difficult to understand. So let me run that, and I hope you'll agree this is a simpler diff. Sorry? What do you mean by where to get the facts from? So in order to calculate a catalog. So the question was, what do I mean by where to get the facts from? In order to calculate a catalog, what happens normally in a Puppet operation is the agent first sends its facts to the Puppet Master. The Puppet Master uses that to generate the catalog. So if we're going to generate a catalog offline, we have to get that fact sourced from somewhere else. Now the facts are cached by the Puppet Master. So I'm just pointing to that directory basically. So I think you'll agree. Nope, that's too much. There's the command and the only diff is one blank line in one concat fragment. That's easier to parse. I have a lot more confidence now. So I'm pretty happy with that tool. And I discovered this tool in October and I thought, okay, that's pretty cool. What's wrong with it? Why is nobody else using it? Why have I not heard about this earlier? And it turns out there are some caveats. So as we just mentioned, and thank you for the question illustrating it, the facts are not necessarily live. Now in the case of how I'm executing it, the facts are coming directly from the Puppet service so they are effectively live as long as the machine is checking in randomly. But if you've copied that file to your development laptop to run this command offline, that's going to get stale. And that's something you have to be careful with. And it's also something if you're working on custom facts. If you're changing facts as part of the catalog, that's going to affect the compilation of the catalog. So be a little cautious if you do catalog diffs with custom facts. The second thing to note is that catalogs don't specify everything. Agents do get to specify some things themselves. And the main two things to watch out for are dependency cycles. If you cause a dependency cycle, or you're changing dependency information, that probably won't show up in the catalog. That information is there, but the agent gets to figure it out for itself. And so that won't get caught out. The other one, and this is slightly more insidious, slightly harder to realize what's going on, is to do with providers. And I should have probably grouped these last two together. If the provider doesn't support an operation or you have a change to a provider, that's not going to show up in the catalog. Let me give you a couple of examples. Firstly, imagine a file system provider. I could have a diff in my catalog, which says go from, let's say I've created a file system and it's XFS and it's 100 gig. That's fine, probably can do that. But now let's suppose I make a change in my catalog, says I want it 50 gig. Well, out to catalog diff, we'll happily report that as a diff. It's gone from 100 to 50. It's a diff in the catalog. What you won't see is that that's an error, because the underlying provider can't do that. You can't shrink an XFS file system, so it will fail. But you won't get that from the catalog diff. Even worse, if you change how a provider works, you won't see that at all. You won't even get a diff, right? So suppose, again, I have a file system provider, which in an old version of the provider, it simply checks to see if any file system is present. And if it is, it leaves it alone. But in the new version of the provider, it checks to see if the correct file system is present. Now, there will be no diff in my public code, because I have changed nothing. But if I've got XFS in my public code and it's an XTree file system, the new version is going to wipe that out and replace it with XFS. You can see that in public catalog diffs, because it's not changed in the public code. It's changed in the underlying provider. Now, I don't want a new off using catalog diffs. I think they're very useful, but I think it is very important to be aware of the differences in the providers, because to be honest with you, if you don't know that's going to happen, it can really catch you out. So this is all very cool. And oh my goodness, I'm going fast. How much time have I spent, Walter? He's gone to sleep. How long have I been going already? Because I've got... 20 minutes? Oh my goodness. You're all going to get to go and get coffee really quickly. Okay. So let's expand a little bit. Catalog diffs are lovely, but not everybody wants the command line. And the classic example that has happened to me in Jobs Past is where you have people just outside the core ops team who maybe want to contribute. It's great to make your, you know, puppet code, infrastructure as code means other people can contribute, right? Just like with any other type of code. But you don't necessarily give them access to the production branches. And in some cases, you don't even get to see the production branches. So knowing what you're changing can be a little bit awkward, but if someone sends you a patch, sometimes it's nice for them to be able to go and see what that does. And sometimes you just want to click a button and get a nice UI, right? So the other thing that's worth noting is that we already are an ENC for Puppet. So if you're using that, great. So we can provide the class data to Octocatalog diff. We have the fact that's provided by the Puppet agent. It's stored on the Puppet master and it's uploaded into the form and database. And we have a Puppet master which usually has a form and proxy on it, which has got all the code available. So we don't even need to do get checkouts. We have the whole lot. So the question that came to me in about, and one of the things that came up for this dev room came up, was can we make this into a plugin? Can we show this in the UI and is that useful to people? And that's the question I want to put to you guys today. It's whether what I'm about to show you is actually useful to you. Because it's really, really hacky. I wrote it in like a week. And if it's good, there's a lot of places we can go with it, but there's no point spending time on it if it isn't. So let me show you this. It's a really quick demo, so not that one. Here we go. Which is a shame because we're going to be done in like 20 minutes. But I'm sorry. So this is the same thing I just ran on the command line. So you can see, so first of all, for those of you who are not familiar with the form and UI, I guess I should just go back a little bit. This is what we call the host page. It represents all of the data we have for a host. Things like IP addresses. I can go and look at the fax data if I want. Configuration management reports, all that good stuff. I'll just show you the ENC very quickly. So you can see that I'm basically, this is what the same data that went into the command line execution that I showed you a few minutes ago. So it's just the DHCP module and it's mandatory to provide at least one interface, so there we go. And the DNS. So no custom parameters, nothing. All fine. So that's what we're going to pass into PuppetDiff. And then here we have this. So what this is doing is it's looking at the, it's actually live when you load the page. It goes out to the smart proxy and checks what Puppet environment you've got. Now that could get slow, so maybe we should make that use what's on disk on Foreman. So for now it's live and it goes out and it gets it. And we can say, give me, actually before I show you that, you can see that we're currently in the production environment here. So we're going to start, this is our from environment in the command line way of looking at things. And we're going to say what happens if I move this host to development, into production, right? It's the same thing. So now I can come up here and I can say this. Now this is a really hacky UI and I'll talk about that in a moment. So it's the same diff, same information and we can say, okay, that's pretty cool, maybe I'll do the merge. Useful. Let me show you a slightly worse diff. So I have another host here. So this has a slightly different YAML file. In this case we have a Foreman proxy and a puppet server. Now that's, those are much bigger modules but the same diff is what we're going to do. So we're going to go from the modules contained in the 1.13 installer up to the 1.14 installer. And this is I'm just going to give you fair warning, it's ugly. So this will take possibly a little longer to calculate. It's about a thousand lines I think. There it goes. So what you can see is we've had a lot of changes in the concat, how we use concat underneath and it just goes on and on. Concat, concat, concat, concat, concat, concat and it's not very valuable information. But if I keep going long enough, where are we? It will stop being concat stuff in a minute I promise. So I'm going to talk about road map in a minute. Here we go. So actually useful information. You can get a little bit better out of this. So you can see ETC Puppet Master has been added and said don't start yet. We've changed from our Puppet 3 style execution to a Puppet 4 style. You can see Puppet lab has been there in our proxy configuration. There's been some changes in node.rb which is our emc script. And I hacked in an extra change just to prove a point. These all look like content changes but just to show you how other things look, you can see the mode has changed here as well. That's not actually in the module I hacked that in as an extra bit. You can see content stuff here. But the reason I put this mode thing in here is to show you that actually anything in the metadata of the Puppet code can be shown here. So if you change any of those parameters it should show up in your diff. As long as it changes to the code and not the providers as I said earlier. This goes on forever and it mostly goes back into being concat nonsense again. Changes to passwords and a whole bunch of Puppet config. So this is a define we've got in our module so you can see a whole bunch of these have been added. So this is not a massively valuable diff because it's really hard to parse. And that's what you get when you write a UI in two days. But the question is is this useful because if it is we can do better than this. At the moment what we're doing here is we are calling what you saw me do on the command line and parsing the output. Which is really bad, really cheap, really nasty and we can do much better. This has a JSON output mode and it can be called from a Ruby API as well. So I can embed this in the smart proxy and I can probably build a UI where you can dismiss parts of the diff as you work through it. You can say this is not interesting, this is not interesting, this is not interesting. Okay, that I'm worried about, that I'm worried about. Work through it. I guess it's a bit like a GitHub review or something like that. And that wouldn't be in a modal either like this kind of thing. It would go to a separate page and people are interested. So I guess we have plenty of time. I can hack around on these modules. Do people want to see maybe another example? Those are the two examples I've prepared. Is there something about this diff style stuff that people would like me to try and show? Right, exactly. Yeah, I don't know if that's possible because of how you calculate a catalog. I guess you could probably parse the output, right? So we've got things like this here, right? So I guess you could probably group it by those entities. That might work. But I can't do, I can't show that right now because obviously that's changes to how we parse the output. I don't think there's a switch for that in OctaCatalog. Just to show you this, actually. So if you've not seen OctaCatalog, this is their example here. Let me full screen that. So you can see, this is a slightly better example. I might as well show you this since we have time. So that's an exec that's gone away. Some changes of parameters here in this file system, change of devices, change of file system. So this is the sort of thing to watch out for if your providers change and so on and so forth. So it should cover everything you can put in your puppet code itself, including things like defines and so on. Just be really careful with providers. Okay, so that's that. So what are the pros and the cons here? It's still putting load on the puppet server. So one of the goals of OctaCatalog was that you run it locally and we're not doing that anymore. So maybe we don't want to use OctaCatalog if maybe we go to one of the ones that implements it as a puppet face or something like that. How we call an underlying tool is obviously up for debate. It's better than doing it locally on your laptop in the case of fact data because we can continually use that cache data that's coming into the puppet server all the time. So you should always be up for data to the OctaCatalog call from the form and database if we wanted because we have it there as well. So one potential use case might be to offer an editor or something so you can change the fact data before we send it and things like that. So that's potential work for later. It's particularly good if people don't have access to the code itself. And not everybody does, but they can go and look at the UI and do a review and you can say just hey, can you just go and see if it looks the same to me or to you rather. So it's an edge case but it's useful. And it's just generally good for verifying things and for quick and dirty kind of what's changed within a UI. So roadmap, I guess the big question I want to hear from people is is this useful? Is this a tool? Hopefully if you like catalogue diffs go play with OctaCatalog diff, it's fabulous. If you like form and you want to use form and you think these two things should work together come and tell me. We have a booth in the K building. I want to hear from you and hear your thoughts. I mean we can have a bit of Q&A as well because we've got a lot of time. So is it useful? Is that something people want? There's lots we could do. So one of the big caveats right now is the ENC data is the same for both calls. So for the production environment and the development environment. That is a problem if you change your class parameters. So I can actually show that actually since we have plenty of time. ETC, Puppet, Labs, Code Environments, Developments, Modules DHCP Manifests. If you recall in my ENC I was specifying a thing called interfaces. If I change that to I'll just put an extra S on it just to break it. So we no longer have a variable in the class called interfaces. So if I specify that in my ENC it will break horribly. And you can show that. Boom, big crash and this is the keyword here interfaces. So it's gone away in the class that we're developing and now my ENC data no longer works for the development environment and that means I can't calculate a diff anymore. That is a major limitation of the current version of the plugin. The only way I can think of solving that is some kind of modal. So this page here we could send that through and use it but you could also offer it to the user and say what needs changing for the new environment. I can't think of a better way of doing it right now. There's no functionality in the form of UI to have different ENCs for different environments. It's just that is the ENC for the host. So thoughts around that would be welcome but an editor would be a first approximation as to how to solve that. The ENC itself is not environment-aware. The host is environment-aware, foreman is environment-aware, you can obviously assign parameters. So I could change this host into the development environment and assign an ENC with the correct variables. But the problem is for this plugin specifically I need to be in two environments at once because I need to calculate it two ways and that is a limitation of this. Now it's surmountable. We can deal with it. The question is whether or not the form itself was the environment-aware. Do you have a question? Do you have a diff before doing your git commit? Absolutely. I'd like to compare what I've got now in this directory for the plugin. Oh. Or do I have to check it out separately? I believe the way out to catalog diff works is you give it two branches and it sets them out into a temp directory and calculates for both. Right, exactly. And that's local to your laptop. So they use it, for example, in their CI so they can basically run through a whole bunch of tests including catalog diffs. Can they do it now by checking out independent version? Well, actually, yeah. It basically changes to temp and does two checkouts and calculates. Yeah. So, okay. So, okay. So, ENC data is right now the biggest limitation on the plugin because if you, obviously, changing incoming parameters is a thing that happens quite a lot with new versions of modules and that just cannot be handled at the moment. So that will be probably my first thing to sort out. Alternate facts I mentioned, if you're working on custom facts or you need to fake something for a different environment then you want a way to provide that data. New UI is really, I think, important. You saw how completely, completely unreadable that diff gets once it goes more than a page or two. So the ability to dismiss it to have each section in blocks and say, okay, this is okay, this is okay. We absolutely need that. And I'm not a UI expert but we've got plenty of people on the Forman team who are, I'm sure we can solve that. And then other than that, I want to do some of the things I've thought of but I'm sure there's more. If this is interesting you, let me know. You can find the code, I did say in the abstract that I'll tell you how to install it. Right now I wouldn't is the short answer. It's quite hacky and it needs some extra sudo rules in order that the Forman proxy can run the ENC in order to get the data and things like that. I'm going to make the requirements to run it much lighter in the next version and then it'll be much easier to install and set up and I'll make sure that's working properly. But if you want to play with the alpha, those are the URLs. You'll need probably a source-based Forman install because it's not packaged yet. And as soon as it's sensible we'll package it. So I guess that's most of the talk. How much time we got left, Walter? I thought you were going to say that. I do apologize for going very quickly but yeah, I'm not feeling super well right now. So I'll finish with a couple of URLs for you. The big blue bit is on all my slide decks. If you don't know Forman, those are the URLs to work with. If you do know Forman, there are three more important ones. So the community survey is open right now. If you use Forman, come and tell us your experiences. There are prizes. There's a raffle specifically. And if you're coming to Config Management Camp, we have a dinner on Tuesday night and a hack day on Wednesday and you are welcome at both. Provided you at least sign up for the dinner so I know how many people are coming. That's me I think. So yeah, questions or sleep, you know, that works too. I might have missed that for the dark. You did fall asleep as well. I'm going to give you credit because that's totally what I want to do. Does it need to run on the master at the moment? Yes. Specifically it's being so specifically there are two plugins so you can install it in the smart proxy and it's the proxy on the public master that's calling the execution. So yeah, it's part of that. Yeah, so basically Forman is responsible for telling the proxy what environments to Diff and what the host name is and then the proxy calls back to Forman calling the ENC. That's not ideal. I'll probably pass the whole data stack down later. But yeah, it does need to run within the proxy at the moment. There's so the question is if there's a UI for the for octocatalog Diff itself. There is. Oh, okay. What was that mentioned in the website? I was in the camp to camp one. I must have missed it. I just started playing with it like I said two months ago. It means full credit to GitHub. It's a fabulous tool. But apparently there's a UI for it. So if you're not a Forman user, you can still get nice UI stuff. Anything else? I believe so. I haven't tried that myself because I don't use Hira. All my stuff's in Forman, obviously. But yeah, so I guess very quickly that you can see here I've got a minus minus no Hira config, which is it I've explicitly added that to disable it. So by default, yes, it's going to pass that stuff and bring it in. So more or less information then. So the question is whether we're getting more or less information than a no up run on a host. I would say it should be the same information, but you don't have to commit it to that node's environment first. That's the key. I have an issue with that because if you move a host into a different environment or if you merge the production code, both of those have consequences. So you don't have either of those sets of consequences if you do it this way. But it should be the same data, yes. That's a very good point, actually, yes. Missing includes might not show up, so, yeah. That's true, actually, yes. So is that a limitation on catalogue this in general, in your opinion? Yeah, I mean, I've only used the old one, the face. Okay, yeah, sure. Right, yeah, so sorry for the audio, I guess, yeah. You could trigger service restarts that are not going to show up in the Diff because they're not explicitly changed in the code. So, yeah, that's fair. But that I kind of did cover that as a caveat because I'm picking hairs, but it's covered by the idea that the tendency ordering is handled by the client, right? So if something changes that has a knock-on consequence, the agent gets to deal with that and it doesn't show up in the catalogue, right? So, yeah. But you're absolutely right. It wouldn't cover it. That's true. You could do it that way. As I say, I still consider moving a host into a different environment even. So the first problem with that is whether or not your ENC is enforcing the environment because then it will fail anyway. You'll say, run in this environment and the ENC will reply, no. You're in the production environment. You will stay in the production environment. Now, you can disable that, of course. But even if you do, you still go back to the problem of saying, well, if I move a host into another environment, what does that do to its variable data? You may be making branching decisions based on the environment. But you're right. I mean, you could do it that way. But it's still an opera on another environment. They should. I haven't tested. So the question is whether it will include exported resources. Again, because I'm a very long-term form of user. I've long since ditched exported resources from my set anyway. So I haven't had anything to test on. My gut feeling, and maybe you can try and learn as well, is that they should if I think there's a puppet DB terminus in octocatholic diff. So you can say go and get data from that. So I'll just, if I might be able to show you. Let's have a look. No, I don't. OK. So actually, I could do it here, I guess, but we're running a bit long time, but I'm fairly certain that octocatholic diff. There you go. So, yeah, there's a ton of options for setting up your connection to puppet DB, which would then presumably also give you a fact data result. All right. We appear to have exhausted the room. Thank you very much.