 Hi, guys. Good morning. I hope you're not too hungover. I wanted to talk to you today about what we sort of started talking about yesterday, which is dealing with the complexities of packaging all the tools that Google wants to ship in Google's Debian image. But in multiple versions of the Debian operating system, it turns out that there's lots of versionitis and other kinds of issues we're running into. And I'd like to go into a couple of examples. And hopefully, you guys can help me figure out what we need to do, either inside Google or between Google and Debian, to figure out how to deal with these sorts of problems going forward. Like I said yesterday, historically, when we built our images, we just de-boostrapped a file system image, and then just copied all of our files inside. And that was suboptimal for various reasons. You can't update packages and so forth, and you can't remove them. It's just less pleasurable and less smooth customer experience. We'd like to make that a lot better. We'd also like to get those things into the mainline Debian release tree, just like we were talking about yesterday, in order to maybe one day brand our Debian image as official. We've been making some progress on this, and so when we first announced and released our Debian images, we've been starting to convert some of our tools. And we've actually got some nice automation, almost complete, due to WeNugan. I've never really learned how to pronounce our last name very well, but basically, once we cut a new release of our tool and publish it to the outside world, it goes and does a whole bunch of things to build a Debian automatically and stick it into a repository that we use to build images. So exactly what software are we talking about? We're talking about the Google Cloud tools. We're talking about tools for compute, tools for storage, and tools for other products, BigQuery, and so forth, and so on. We're talking about various kinds of software that are used just specifically to make Google compute a functional product, the ability to create snapshots of your disk images, things that manage accounts, and manage IP addresses, and things that set up the host statement and other things like this. We also sometimes want to modify configuration files like SSH and SSHD. There are some issues around the cloud where, for instance, if you don't keep your network connection active, it sometimes will time out. And you'd like to keep SSH, keep alive, and stuff like this to keep the connections alive. But it's not clear exactly how to do that. So some of these are harder than others. So for instance, we found that packaging up our tools is, we don't know exactly how to do it right. Some of the things, and mostly it's due to dependencies we found, some things are easy, mostly because they're just simple files. They don't modify anything else. They have very few dependencies. It just needs Python, execute, or the Python runtime things just work. And then some things are questionable. It's unclear to me that if you wanted to modify the configuration files from another package with your package, how you might do that. That's fairly easy. What was that? A policy? Modifying the configuration file from one package to another is forbidden by the Debian policy. Good. So that solves that problem. Mike, too? On some package, you can include files from the directories. I mean, if you want to modify atc-sudo-us, you can include a file in atc-sudo-us.d, your file. Yeah, so if you're saying if the existing package has a .d directory, you can add files into the .d directory. And that means sometimes patches packages to support that because of this exact kind of use case. And I think we're using that for our syslog. I don't know if that's a patch or upstream. And we should probably switch our sudo-us for the account management to use that as well. Yes, I would agree. In general, we love that idea. And whenever we find it, we use it. But in some cases, like SSH, I don't think that's currently done. It's not supported by the tool. So maybe this is one of those things where we just figure out how to transition all the existing tools. We'll talk about that later. There's a section later. We should maybe come back to that. But those are good ideas. Let's see. So let's go into a couple of these examples of what makes things hard. I talked about them a little bit yesterday. I don't know if we're all of you here yesterday. Yesterday's main talk for Google Compute? Only a few of you. OK, good. So Google Compute, GCUtil, is a command line tool for Google Compute. It's a pretty simple program. It's just 100% Python Apache license code. It doesn't do or it doesn't need anything fancy. It uses several packages here. Google App Utils, Google API Python Client, HTTP Lib 2, IP Adder ISO 8601, and Python G Flags. These are its dependencies. Back in the day, about two years ago, we were using a PyPy to build this tool and suck up all the dependencies that we needed. But we found that, in general, the Python versioning system wasn't doing a good enough job making sure that we got exactly the versions that we want of all the packages. And we were getting many support calls from people who had a system in which some module was a slightly different version because of some crazy dependency chain or something like this. And we eventually just switched over to a worldview in which we basically took the exact versions of all the dependencies that we need and we copied them into our source tree, into our package tree. And we import modules directly from there. Yes, go ahead. You know that it's not possible to do that in Debian, right? I don't know. So this is possible. It's possible. It's possible, but for a video. It might be on Cozier. It might be on Cozier. But in general, we found that it's the only way that we can ensure the stability of the product. Yeah, I mean, I can see both sides. Well, I mean, obviously, I'm wearing two hats, right. But I've mentioned that Debian typically avoids this, but at the same time, whatever, the solution should not, we need to figure out a solution other than having an unstable product. We need to find a solution that respects the reasons behind Debian's policies and also allows a useful experience, like a stable experience, for the customers and users. Is there anybody here who could talk to why the policy exists? I can see HTTP lib2, IP ID, HDR, ISO H6501. Oh, this is in Debian already. Yes, but so many of them have versions that we have not been able to qualify against. And maybe we just need to qualify against all the versions of all these things and all the versions of Debian, but again, that increases the testing matrix, and it's a large support cost. I mean, one of the questions I might have is, why would copying statically into a single package, what the issue would be? So there are many issues. One is code duplication, and the main issue with code duplication is security. When you have a bug in a package, Debian makes sure that that package gets updated. But if you have that package statically copied, then that bug is not fixed in your statical copy. And so you may have a security bug that is, Debian is not able to fix it because you have the copy. Unless they also fixed all the things that depended on it. In front, it's this case, GCU till depends on this and this and this. Your code is not even the Debian repository. So that's not possible. But for packages that help copies, and those copies are in the Debian repository, it means a lot of extra work for the people who are providing patches to also go and patch all the statical copies. So it's more work for security, mainly. So I feel like your best solution here is, if these libraries are in the archive, you should be able to, you should try and work with the version in the archive. Because if you were to make our Python HTTP lib2 a new version, then you'd break other things in the archive. And if you ship your own copy of it, well, then you're responsible for the security. That works as well. But it's now your problem. It's not integrating with it yet. There still is the challenge, though, that from Google's perspective, we need to make it work with Squeeze, and Weezy, and Jesse, and CentOS, and Red Hat, and SUSE, and so forth, and so on. So now we're talking a dozen different versions of all these packages that we have to qualify against and change code to support, right? Of course. Unless Debian wants to take over the responsibility, I guess, maybe, of when, if somebody wanted to create a package of these things, I don't know who. But maybe. Can you say the front desk for new maintainer? Say that again? Say the front desk for new maintainer. You can do that. He's suggesting that you join Debian. There you go. And fix it once. At the same time, the same argument about duplication of effort across different operating systems from Google's perspective also applies from Debian's perspective across different packages that have the same logic at the end, so the same logic has both sides. It's also, from a customer point of view, if I, as administrator of something, when everything uses the Debian packages, it means you have to test more. But it also means I, as user, only have the package once. When I have multiple copies of the package, I have to make sure all of them are secure. All of them fits my needs. If I have something special, I have to fix all of them. So if there is something that duplicates everything, I'm very reluctant to use it as user. Because if I have a big installation, it means much duplication for me. Sure, and I understand the issue. But with respect to this code duplication problem, it's quite conceivable. I could see, at least for some of these cases, the easiest solution for the program developers, which is not actually me, it's the other teams at Google who are building some of these tools, is maybe just to drop the dependency on the library and write a whole separate chunk of code that recreates what happens in that library just because they can control what's happening in that code. And if there's a security bug there, you're still screwed. You can't update that either, independent of this tool. It's, of course, easier for the developer to only have to test with one version. But for me as user, most things that are easier for the developer are things that make me not use the software. If you don't have a bug tracking and no possible way to report bugs, it also means it's easier for you because you don't have to keep up with bug reports. But for me as user, it's a sign that I don't want to use it. And it's the same way with duplication because it means there's multiple things doing the same functionality. And if there's anything special I need, and from bigger installation, there will always be something special. I have to keep up with all the little different versions. And if I am in a more complex setup, I have to verify all of them and keep the paper trail for all of them. And if I have to push a library once through change management, it's worth enough. If I have to do it for a dozen copies, I don't want to do that. But that's still assuming that this tool is using that library, right? If Google Compute tool one day decides that, well, I'm not going to validate against 12 versions of HTTP lib2, I'm just going to write my own. What's that? Well, for some of these, it actually is that much, right? So it's not so much a problem for GpnLtyl. It's more of an issue for the next tool. Just for the leaves that you mentioned here? Yeah, for this one, you're right. It's not that big of a deal. But for this one, some of these libraries change much more frequently. So Bodo and CRC Mod and PyOpen SSL, there are many more versions of these things floating around. And this team would be much more likely to not validate against 1,000 versions of each of these. Can I suggest we let Luciano speak? Sure. Oh, yes. In security team, there is a public file called embed copies, which is basically the database of packages that includes embed copies of other packages. So when we need to patch something in one package, we check in that embed copies. And we patch all those versions too. I mean, embed copies is a problem in Debian as a whole. And I don't see your solution that far, I must say. A lot of upstream developers made it daily. And it is a global problem in Debian. It is how it works. So there's other possibilities. Actually, the biggest thing I want to say is that you should make sure to get through the rest of your slides and bring up the rest of the issues. And this conversation needs to happen. It just maybe should extend beyond this session. Can I say one comment on that? Just on the Boto thing, obviously, we have the same thing. Boto ships in Debian. And right now, it's quite out of date, obviously, on stable. And that's something we're going to live with, I guess. And obviously, the other important point is it's important for these not just to be on our images on the cloud, but our customers are using this on their own installations, on their own hardware, to talk remotely to us. So I think we need to have a copy of all of this in main. It's all free software. So it's licensed compatible. And it's just going to be out of date, which means newer features won't be available by default. My suggestion would be that we have separate repositories where optionally, you could go and add that to your app sources file. We've got app sources, is it? It has a dot date. There is a dot date. So you could just pop a file in there. That's as simple as an echo from the existing bootstrap that we've all got. Yes, it's an extra hurdle, but customers who want to do that can actually copy and paste. Here's how you get the latest copy. So in a way that's relevant to quite a lot of the slide deck, does anybody know much about the PPA main type proposal that the SCP masters mentioned on the mailing list in May? I was hoping one of them would be here to talk about it. Do you know about it? Take a microphone. If you know about it. I know as much about it as you do. So there is a proposal to have PPAs in Debian, which is basically going to mean any Debian developer can create a private independent archive. It's laid on top of the main archive, but you can upload your packages to it and they don't interfere with packages and other PPAs. It's not a repository so that you can upload any random version of things which are already in the archive. If you want to make a back port, you're ready to do so. I think it's best to upload to backports.debian.org. Also, if you're going to be back porting all of these libraries to support your utility in a separate archive, you've got exactly the same problem as if you just dumped them on the machine. If it breaks something else that depends on that package, you've broken something. Yeah, you can break everything in them, like all the other stuff that's been qualified inside of Debian. Part of the issue here seems almost as if you need to have the ability to have multiple versions of packages on a system for different kinds of programs. And maybe balancing the needs of security fixes with different kinds of functional requirements for each of the pieces of software on the machine. I'm not sure. Wookie, you had something to say, I think. Yeah, I was just that Ganev said he was going to work on the PPA thing this week, next week. So actual some progress might occur. That's good. But any useful details? We should probably move on. There's one more person? One more, yeah. Well, if you have to maintain multiple versions of a single package, we already have a 30,000 package. If we have to maintain multiple versions of the same package, it will be a lot of work for us. And we will have to maintain and maintain version because usually upstream only maintain the latest version. So we try to package the latest version. If we have to maintain older version, it will be a lot of work for us. But for your package, I am pretty sure we can work to make them work in Libyan as they are. I am looking at BOTO. BOTO is 296, the latest version is 299. Do you need 299? I don't know. I don't think the version of BOTO is terribly out of date for us. I think it's pie. I think the main ones are CRC Mod and PyOvots SSL. We have a very in Libyan, a very healthy Libyan piton module team that can help you to package the dependencies that you are missing. And we can work to be able to package your tools without bundling the dependency. I'm pretty sure I can volunteer today to help you do that. OK, that'd be great. Yeah, the general problem outcome hopefully will be more collaboration to solve the problems within the existing model and maybe to extend the model where necessary. But so he's also thinking with regard to Jesse, with regard to Squeeze and Weezy, it may be a matter of putting things into the backboards repository to allow people who opt into that to use those where they know it can require some other. I mean, if maybe Google just subscribes the backboards and makes sure that all the versions are the same, but then that might break everything else, who knows. Well, OK, let's move on just a little bit to the next kind of issue that we have. And it's a fairly similar issue. So we're not actually going too far away from the last one. So here, this is, again, we kept coming back to this slide. Where are the version ideas happening for us, right? And in this particular case, it's PyOpen SSL and CRC Mod. And I'm possibly even socks the pie branch. I can't recall. And again, in Jesse, I believe all the versions that we need are there. But in Squeeze and Weezy, which are the released and stable versions of the operating system, the versions are just too old. The situation here in GSU-Till is a lot more complicated, because we can't even do the copying and the code kind of trick to solve the problem, right? Because the binary dependencies, we'd have to build a version for each version of the architecture that we care about, et cetera, et cetera. It's just a lot more complicated. And yeah, let's just move on from here, because I think we've talked about much of the issues around this already, I think. And let's come back, I guess, a little bit to the config file thing, which is another kind of problem that we have. I think we had some ideas about using .dfiles and stuff like this to solve this sort of problem. Yes, go on. Depending on how long-lived your cloud instances are going to be, one thing to consider is to use configuration management, by which I mean that you actually put your changes into central location, and then have some daemon running in the background that enforces these changes on the local system at all times, puppet, or whatever it is. There are small tools, not a whole infrastructure like puppet, although it might be interesting to just be able to actually take that and integrate it straight with your whole infrastructure and have all the nodes be centrally managed. But the benefit is that you will somehow, or at least the daemon users that I know, would prefer to have a very exact description of all the changes that you made compared to the default installation. So before you create a file that says, we changed, permit root login to no, and we changed this, and we changed that, put it in a form where this can actually be enforced. Yeah, I would agree. This is one of the problems with the approaches that I think in some ways Amazon is taking and Google is taking that the build image scripts, they change things, and you can't tell exactly what's been changed, and you can't revert it when it's after it's been changed. The thing that I like about using some of the .d files and stuff like this is that you can install a package that says, this is the Google secure lockdown config package, and I just install that. It applies the updates to all the files and all the right directories to create the policy that you want, and then you can remove the policy later, and you can list it and see that it's there. But again, not all programs support that, right? And do we go and change all those programs so that we can support it? So does Cloud have a mandate now that we have to go and change open SSH so that it supports .d files and stuff like this I don't think it does? So I have two quick things. One very short. Obviously, we don't have any passwords by default, so the SSH locking down isn't particularly important until people create accounts. It's not just the locking down on the SSH. There are several other things that at least Google cares about doing, so network. The other thing is, as an AWS user, I'm not interested in any of Amazon's images. At my company, we use the images Ubuntu publish because we know exactly what they are that got everything we need. And we customize them beyond that ourselves. We don't need any utilities on them. We just need an Ubuntu machine. Yeah, a lot of customers that come to Google, though, want something that works sort of smoothly out of the box. What's very smoothly out of the box for us, but. Yes, but Ubuntu doesn't exist on Google compute at the moment. We don't have a problem with Debian, either. Yeah, so while your company might not care about using tools that talk to the other Google systems, most of our customers need that. He actually meant smoothly integrating with the rest of the cloud platform. Can I go back to what Martin was saying about enforcing policy? So what we're doing here is we're talking about policy when the machine initially boots. After it's initially booted from an AWS point of view, it's your machine. And if you want to go and enable root SSH and you want to go and enable password login, we don't want to enforce that policy from Amazon. We want you to have whatever policy you want, because you could be running that behind private networks, which are not internet-accessible in any way, shape, or form. It's yours to do. So that, I don't know if we need to, I agree, Puppet and Sheffa are brilliant orchestration utilities. I don't agree with that, but. But as soon as it's yours, it's yours to change whatever you want. But I think Open SSH and SSH access, I think that one of the things I raised yesterday that I think is quite useful is what do we feel about having, we're going to install Open SSH into cloud images and that is your way of accessing it. What do we think about having fail to ban or deny hosts installed also alongside SSH so that it's going to lock you out if you do misauthenticate a couple of times? No one? A different IP address. Here you go. I think that all of these suggestions, like including fail to ban and so on, they're great. I mean, I can imagine that there are some people who would really like to have the carefree package and just install an image and know that this is already tightened. But on the other hand, if at some point, if this is a short lift instance, then I don't think we need to have this discussion, right? But if this is an instance that is supposed to live for the next three years or survive a couple of Debian releases, then you, at some point in time, you need to know what changes have been made. And if you make changes and I make changes, I try to keep them in some sort of log file so that later on I know what, when I screwed up, and we need to merge those log files. And all I'm saying is perhaps the abstraction of putting it into something that can be used to enforce policy. I agree that it actually does, right? If you say permit root login equals no in puppet that runs locally and I make the change back to yes and then it gets overwritten half an hour later, that's probably a problem. That's a communication issue though in many ways. But before we have a redundant file that is your log file which I tried to merge with my log file of all the changes we've made to the system and potentially I'll grab a number of the year now, 85% of all the cloud instances are ideally going to be centrally managed with some system like that. Maybe the best approach would be to just enable, at least give the plugs, you know, provide the interfaces for this sort of management from the start. I just wanted to get back to this .d configuration files. You mentioned that you don't want to update all packages to support it. I have heard of a package, I think it's named .d which actually allows you to use the .d folder for configuration and it will then, when the program actually tries to read it, it will concatenate all of them together. So it allows you to support it, even as a program itself does not. This is like a named pipe kind of program. I think it's called .d. I like also to point out that over multiple clouds we need to have consistency, meaning that if you disable root loading in on the Google Cloud on what we want to call the official Debian image, then the behavior has to be the same on AWS or the image that we provide for OpenStack or whatever. So I would be against anything that would be cloud provider specific. I would love that we have only one single image that would work for all of them and that the behavior stays the same as much as possible. Yeah, I would generally agree. I don't know if there are going to be differences amongst what cloud providers really want. So one of the things that I sort of feel I hear is that there's maybe a need for a general configuration mechanism and a policy expression engine of some sort that sort of like a check config kind of thing where you basically register with some place that I want SSH to take over, take this flag and these flags and these flags and then keep track of that history and then make modifications to that via some mechanism, right? That's sort of what I hear people saying, right? I don't want to necessarily install a package that locks down the system. I want to install a policy and make that combine with the existing policy in some way to create a new policy and then have the ability to add another one, yes. I think that's what Martin is saying. I think what Tomah is saying is that it should be done at least for the official images that are the default that are promoted generally not for any local customizations. It should be done in a sort of debbing consensus way with an eye toward the cloud context. Yeah, but if we had this policy in a way that was easy to understand and represent, then it's not as concerned. It's not as, yeah, what I'm saying is that if such a policy were easy to understand for the customer, I don't know that it would matter if it was so much the same as it was just the same structure, that they can look at the policy and they say, oh, that's the environment I am. I'll flip this switch if that's what I want. So you were asking whether you thought the cloud had kind of the right to dictate config mechanisms. I just say that all the embedded people have exactly the same problem, right? That you need to do some customization, you know? And I'm not, and we've done it by installing crappy packages that overwrite files. That's what people mostly do and it's pretty crummy when you come to update stuff. So the Freedombox people have this problem in a big way as well, because they want to make something totally idiot-proof and maybe make a web interface to all the config. And we don't really have a good mechanism for that. So I think there is a good argument for better config update mechanisms, but you know, probably we should try and use something that already exists. So the open-work project has a really interesting config mechanism which integrates well with the open WRT. So, but it's really different from what everybody else does. So it's difficult to use in a Debian context. That's one way of turning all your config into bits which you can then easily just play in a web interface and change that way. But you know, presumably this is what Puppet and Chef and things already do. And maybe we should just use those. And we already have Debian Conf. I don't know what happens when you try and use both of those together and everything breaks. But yeah, this is not a problem unique to the cloud, right? There is, if we had better ways of applying policy and all our installer stuff, you know, the FAI has mechanism for all this as well. I don't know how that applies. Ask somebody if you need to ask Phil Hans or someone who actually knows. I think it's more of a problem for cloud because we have to show it to users and give it to them as their default image, right? And we expect them to change the configuration more than I think in the embedded space. But I totally agree, otherwise. What is cloud config? There was just an imaginary package name that is like, what is the right configuration for cloud? If cloud configuration, it should be different from the basic hosted configuration for Debian, like the center host, the CD version of Debian. Maybe you get it by installing cloud config. But after listening to you, I'm not sure that's the right idea anymore. Maybe we need something stronger, right? You need something that's not just install package. You want something that merges policies in a sane way and keeps historical logs and things like this, right? So you can roll back and roll forward and so forth and so on, right? I think that's what I feel I might want now. One of the things that comes out of this discussion is that ideally, the Debian system should be parameterized completely, right? Ideally, all of the things that you might ever want to change are somewhere parameters. And in some ways it is. We have DevCon. It's not a configuration mechanism, it's a cache. But there's a whole lot of infrastructure behind it requiring the packages in their post-ins file to reconfigure themselves according to choices that the administrator makes or that can be preceded or automatically introduced into the system. So theoretically, that is exactly what you need. However, it's been, I don't know, 12 years that I've been trying to permit root login in SSHD to be parameterized. And for some reason, it has not happened. What are the obstacles there? Is it just SSHD has to be changed? I didn't write the patch. It's all my fault. Okay, so, okay, I'll talk with you later. The point is more that you're not ever going to make it, to find all the parameters and then to convince all the package maintainers to put them in. And then you're also unleashing a whole can of worms because suddenly all the DevCon translators have to start working on the stuff that you've just prepared for them, right? So potentially that's not going to happen, at least not fast. And I don't think you wanna be discussing this for the next 10 years. So another, and since we can't convince Debian to include Ruby, which is required for Puppet and Chef into the base distribution, at least I will object. Maybe there's something much simpler that we need. And in essence, I really like the idea of the .d directories and especially, maybe we can find a way to use that paradigm to make changes. And I imagine there's this tool, August or maybe there's a Clint brought a tool called Deets which is a locally running tool. It doesn't do any network configuration. It just pulls your configuration out of Git and then applies changes locally. And if you can just make it so that, if that, I think that tool is very open to influence. It was a good idea, but I don't think he took it anywhere near 1.0. I imagine that you'd have a Deets.d directory in ETC or in SRV or wherever and then you just drop your policy files in there. And they actually get enacted. And they will make the change to SSH. To the SSH file, just updated by, exactly, and then restart the service or something like that. Now you still have the problem that this is enforced. If I go to ETC SSH.d config and make the change back and then it reverts 30 minutes later, then obviously that is somewhat of a policy violation. But this can be solved potentially with comments or it's a communication issue, as I said earlier. So with any of these tools, that whatever, whether it's .d or Deets or something that doesn't have d in its name, we should, nothing you can do. Okay, there's nothing I can do. We should make it, so we only want to enforce it by default, like with Amazon at the launch, right? And then maybe the user may want to have a way to reapply the policy, but we don't want to automatically override their choices. So remove the crum d. Right, so I think the idea would be to maybe do it once at boot possibly and allow them to rerun the tool that applies it if they want, like not a daemon type thing. So what's that? No, I mean we've got a fun alley side, that's about it. I like the idea about the Deets-like thing, the Deets-like policy thing, because it at least puts all the policy in one spot. I don't love the .d directory ideas as much because you end up in a place in which if I install a package that throws some files in there and you install a file that changes something and in some other way, right? I have to go and look through the evaluation of all these .d files, right? And I have to figure out what the heck this guy turned it on, this guy turned it off, I have to know the evaluation order and hopefully that's the same in each program that evaluates that .d directory. But that's a dash dash test invocation on the Deets tool, for instance. I mean that just shows you exactly what is being done and you could say dash dash what changes and then give it a file name and it tells you the order of policies that are being applied to it. But like modifying SSH itself to take an SSHD directory and then he installs a package and I install a package and you install a package that I don't love, right? Yeah, I think I'd just say that I think we're a little bit fatalistic about using Debcomf. I mean for some of these things that is the right mechanism and we should just use it. But not everything, I don't think it solves the whole problem but it probably solves parts of it where actually a lot of people want to change that particular thing and we should just put it in. And that would actually address the goal of keeping the system as close to real Debian as possible. On the one hand that's a benefit, it makes it very Debian-y. On the other hand, it means it's not like anything else people are used to. So if you're puppet people, Debcomf's really annoying, but that's kind of it both ways. How does this mechanism, so I'm actually not very familiar with that mechanism. What does it look like? Debcomf? Yeah. Essentially it's a protocol that makes questions up here and either get asked to the administrator or fetched from the pre-seed file and then suddenly you have a parameter equals value type system and now when the packages are expected to look at this thing? When the package configures itself and runs the post in script, then it is supposed to merge what it finds on disk with what is in Debcomf giving precedence to the thing that is on disk. I see. So Debcomf is only a cache, it's not the Windows registry. Okay. And the point is that it's easy for it to be, the question part is translated and appears by, could be in a terminal or in GTK boxes or whatever. So there's various front end mechanisms depending on what's appropriate for the situation as well as pre-configuring and translations which is why it's quite nifty. When they say translations they actually mean human language. This is not always clear in the technical context. Gotcha. Yeah, so the package is the one that manages its own Debcomf questions and it's done by the package infrastructure for the package. Obviously it's relatively heavy weight in terms of affecting each package and you have to put the variables in. So, it's a bit of a slow process than just having ETC Keeper or some other magic that just does whatever we want to do to our ETC directory. It's a slightly different thing. This is automatable at the installer level for example. The concept of getting something like Deets into Debian proper is long ways off, probably. Very feasible for Jesse and backboards. Okay, good. Any other quick questions? Comments? I don't know what you mean with Debian proper. It is in Debian. I mean official mainline Debian official release whatever you might call it. It is. It's just not in the base system so it will not get installed by default. I see, I see. So you just have to add it as a package. It's in there. It depends on Lua, which may be a showstopper, but no, that was supposed to be a completely objective statement simply because it has dependencies and then that makes it difficult to get into the base package. But I think that Clint, who's pretty well known and knows Debian very well. This is a very Debian specific approach. This is not at all like what Puppet is trying to do to be like the solution to everything. But I think he's no longer maintaining it or no longer adding stuff to it. So it would basically someone would have to look at it and shake over. Which I think it's probably worthwhile to check it out. Oh, dang. Ooh. Just relaying something from IOC. Petter reminds us that there are levels of debcon questions that will never show up. Hidden questions for things where you want to automate or we want to precede something, but you would never really want to ask a user for it. I think we're about out of time. So I just want to thank you guys for chatting. I'd love to talk with some of you more about Deets and Virginitis and stuff like this and how we might deal with it. I mean, really it's Virginitis either and for the like in the operating system or in the person who's writing the software. Right, you have to deal with it somewhere. I wanted to just let you guys know that this week we actually are building new Debian Google Compute images. I meant to say it yesterday, but I forgot. Jimmy and I are probably going to be doing that Thursday around noon. And if anybody here wants to like help, you can actually have you guys like run the commands and install it and send it to test and all that stuff if you want. Yeah, it doesn't require being a Googler. You know, any DD, it doesn't even have to be Gmail account, just any Google account. We can, you can sort of see how Build Debian Cloud works. You can run it on your local laptop. You know, the account is only needed to upload to add the image. It's a really neat thing to get involved in. Thanks. Thank you. Thank you. Thanks.