 I see entry in the audience. I'm entry and I'm working on Taze, which is a derivative. I'm Miko and I'm working on Grimmer, which is another derivative. We pronounce it Grimmer, it's G-R-M-L. Questions of? I'm John Vert, working on SteamOS. Jeffrey Thomas, I used to work mostly on Debathina, which was an MIT lightweight derivative of Ubuntu. Colin Watson wearing my Ubuntu hat. I suppose I used to do a bit of TV and stuff, but we just kind of closed it, so it doesn't really count anymore. Adam Kohnrad also Ubuntu. Ubuntu? Right now. Oh, you want me to talk a little louder? Adam Kohnrad, Ubuntu for the next 50 minutes. Do we have any of the HP guys here? Up the back. Bill Brothers with HLinux out of HP. Okay, that's everybody. Anybody who's a deviant contributor interested in getting involved in derivative stuff? Looks like no. Okay. Anything any people want to bring up? General discussion. Here we go. Paul? I noticed I was looking at the apt bugs with Newman TfMine, and we noticed that the format for repositories is not fully documented, and basically I understand that that's something you're also working on in terms of derivatives. We emailed the bug for apt to say so that we were interested in finalizing the docs there, but I thought I would ask what the status is, and if repo generation is something that's hard for derivatives, or if they just deal fine. As far as I know, well, in the derivative census, there's a field about what repository generation tool. Rep repo or apt FTP scan thingy. And the format was documented like a year ago on that Wiki page, I think is the name of it, I'm guessing. Do you also want to have Blance under this topic? Deviant Blance? Deviant Blance, yeah. We can hear from a Blance perspective, because they're two sides of the same kind of coin, adapting Devian for a specific use, and the difference is only whether that's integrated into Devian or not. Well, to some extent. Pardon? Okay, so a Blend, well, there's a Devian pure Blend, and there's a Devian Blend. The difference is that a Devian pure Blend is already all inside Devian completely. The Devian Blend is aiming to be a Devian pure Blend. Yeah. So did you want to... I think there are plans for Jesse to have the options to install some Blends with the default Devian installer. I think this is good news. The Blends people are really looking forward to that as well. So I was wondering if their relatives have any plans for CQB, except for, you know, Ubuntu, and if there was like some shared work for that? This is kind of a backwards... Hang on, let me put this down. This is kind of a backwards answer, but I would suspect that it wouldn't surprise me if a lot of people were waiting for us to actually get it sorted out in Devian, which we are working on. I haven't heard much... Well, I've heard a couple of people asking about... There were a couple of people in the UFE circuit, or possibly the Grubbuff, asking about, you know, how they would arrange to apply their signatures on top of Devian's. There are a couple of people at the UFE circuit, if I think, asking how they could apply their own signatures to top of Devian's. So I know some people are interested, but I would expect those people to be waiting for Devian at this point. If anybody wants to contradict me, please do. As a follow-up to that, I wonder which derivatives actually ship their own kernel and therefore have a need for actually signing their own stuff. And which of them don't really want to go through the microsoft signing process. So regarding the microsoft signing process, the direction that this is going, I'm told by Shem folks, is that they're working on making their builds reproducible so that the difference between the same version of Shem with a different embedded key will just be a different elf section at the end. And they're working with microsoft on that to make sure that microsoft will be able to fast-track such applications so that you won't have to wait months for an answer, well, weeks or months for an answer. So that microsoft will just be able to see, oh yeah, this is the same binary with a different key. Here I have a signature. Doesn't save you from having to pay the sort of relatively nominal cost to microsoft. I gather if, in case anybody has ethical qualms, I have heard that it's a cost centre for microsoft, as in we pay them less than they spend doing the work. So you can consider that. I'll make a brief comment here because I want to possibly solicit people to work on this. So with my school alum had were using the Ubuntu kernel which doesn't care strongly about like signed kernel modules at the moment. If it does start to do that, we may have to think how to do that because we care about open AFS, which is an out of trigger kernel module. And there may be some, we may have to think as we're using DKMS, so we may have to do something to be able to load that if Ubuntu ever changes plans there. At the moment where we are able to use Ubuntu DKMS compiled modules because Ubuntu doesn't do signature checking, with my day job hat on, we are including proprietary out of tree modules into the product we ship, but we are shipping a whole image product where we aren't, you know, saying that you get to do arbitrary things in user space, you get to customize user space. Here is an image and the whole image boots and we're actually totally fine verifying the whole image because that's the security model we have ourselves. I am interested in working with people who are following the same model because that may, for people using live systems and so forth, that may make things easier than pulling in all of the secure boot patch set to enforce the strict user space and kernel space boundary. So the root versus kernel space boundary. So that is an option and I would be happy to talk to folks offline if you want to collaborate on making this easier. That's fine. I have a question maybe to get a more conversation happening which is how is it being a Debian derivative? What could Debian change? What pain points are there? Or is it always just great? So H Linux is primarily being built for use to being the foundation of OpenStack solution set delivered by HP called Helion. You guys probably have heard of that. It is a closed bottle. It is an image based model in OpenStack, but there is a number of other initiatives in the company that have gotten very excited about the fact that we have an open source operating system in HP that we can do things with and we're seeing a lot of demand for custom pieces or configurations. You'd be happy to know that we've made a concerted effort that anything that we put in H Linux will come back into open source, period. There will not be ever any proprietary stuff put into that solution that doesn't come back into open source space. Having said that, there's some real tricky pieces that are being worked on and it's been difficult. Some of you attended the session by Rocky Craig where he talked about how we're building an ocean of every repository piece and you can pull an arbitrary repository out of that ocean at any given point of time. And so versioning has become a really interesting thing as well as following all the rebranding rules associated with that and being able to uniquely identify all the right pieces that belong to the right parts. I think probably the biggest issues that we've had is to try to follow the rebranding rules and not break Dev installer or bootstrap and use all the right tools to get the pieces to fall in place. So it would be great if you could help update the branding stuff or problems that you found and we'd like to fix those issues. So unfortunately, Rocky and another fellow in the team in Josh Hawkins have really done all that work. I don't have depth to describe it here but we can certainly provide that. You can do that over email bug reports. You can definitely do that to describe the problems. If you would bring up a thread of the derivatives mailing list, that would be great. So one problem we are facing is changing configuration files like the wording configuration files without going crazy in Boston store scripts. So for example we had this discussion about the SSH in the script. We have different needs and so we can ship the upstream one. We had this discussion which I fully agree with that our modification can't be shipped upstream but we still need to divert in its script accordingly. Are you familiar with config package dev? Have you looked into config package dev as a system for doing these sorts of diversions? Okay, I'll briefly advertise it. Config package dev is a tool that's in the Debian archive. It's not intended to be used for packages. Thank you. Thank you for backporting it by the way. It's a tool that's intended to be used. It's a dev helper tool and also a CDBS tool. It's not intended to be used in packages in the archive because it will make them stupidly policy non-compliant but that's not what it's for. It lets you say in a fairly straightforward way in nice dev helper style syntax I would like to replace this particular package file with my own copy by doing de-packaged diversions and undiversions in the post-inst and pre-RM as appropriate. It handles all of the cases. It also has a thing called transformations where you can say I'd like to install and build depend upon this package, take the current version of the configuration file as I see it during the package build, apply the set script or purl script to it and then package the result of that as file.conf.myderivative and when you build the package and when you install the package it will automatically create a diversion. It also has for various technical reasons involving de-package and diversions and edge cases and it will not directly drop your file in place. It will make a sim link and it is supported for the end user to decide to point that sim link back at the original version of the package. We do keep the original version around so you can use it for wrapper scripts That is the other thing. If your upstreams can be talked into using .ddirectories that also solves the problem but config package to have is great for wrapper scripts. Any other thoughts? So de-package vendor is very useful for doing slight variations between things but at what point does it become craziness when we try how much can we merge into the packaging and any derivatives In general it is often used to have the debian and the blunt 2 versions and we ignore everything else but if we were to put like 60 derivatives flavors of config files in that might be a bit crazy and I just wonder what we think we should be doing with that. I suspect that a lot of distributions will actually want kind of similar-ish things so we will end up not having 60 but more like a few variants so I merge the only package I am aware of with more than one use of de-package vendor is Grub 2 for which I got code for something called a derivative called Tenglu recently and that is merged in as well but that turned out to be essentially identical to at least a subset of the open-to-pouch set so it really didn't add very much code at all. Can you explain how it works? Ashish, can I explain how it works? De-package vendor in general? De-package vendor Okay, so any distribution any derivative may declare its relationship to its parent in a set of confiles that are read by bits of de-package so in Ubuntu we have our distribution name is vendor sorry, it's Ubuntu, bits of similar metadata our parent is Debian you can say de-package vendor dash dash is Ubuntu returns true, de-package vendor dash dash is Debian, returns false de-package vendor dash dash derives from Debian returns true et cetera so that's in de-package dev you can use it in development type context so you can use it in rules files if you need that then you either use LSP release or you build stuff dynamically to do the right thing preferably the latter de-package vendor then you can use it in shell conditionals in your Debian rules file and compile stuff conditionally there is also some support for distribution specific patch series so you can say Debian-patches-series.ubuntu personally I don't know what other people's experiences with that are I've never been very keen on it because it complicates management of your patch series in version control systems and also you have to duplicate because it supports removing patches to get the entire contents of the series file in series.ubuntu or series.derivative so it's really easy for it to get out of sync so yeah I don't like it very much for that reason I prefer to write patches that implement conditional behavior but apparently not everybody does I'm just wondering is there any Debian derivative you're not using in-it-from-FS tools so like tracart or something I think tangler might be looking at tracart I mean yeah I'm with my in-it-from-FS tools looking at tracart and I think we already discussed on IRC regarding how we could handle that I mean the main blocker to have what I would like to see is that we don't have our own in-it-from-FS implementation but have something like distribution independent like Red Hat whatever and the main showstopper might be the live boot stuff and tracart not being as flexible as we might need it and just as a follow up is any one of the Debian derivatives carrying any patches against in-it-from-FS tools or custom against live boot stuff which isn't upstream yet I'd like to hear yeah so we we really like in-it-from-FS tools I think I'm reasonably safe to say and I think it came from Bunty originally it kind of seems to me that moving to dracart is a big bunch of effort with not a lot of upsides yeah we would get to share some patches maybe some hooks but it's I suspect there's a lot of work to get there and it's not clear it again from it the as far as live boot goes we're still using Casper and one of these days I really must switch things over to live boot but we've again big bunch of effort with not a lot of immediate upsides I'm having internet issues but you can check the patches against in-it-from-FS tools at this URL this is based on the Debian snapshot service that downloads all the source packages from all the derivatives in the census and compares them to historical Debian source packages and dumps a patch for each one that looks different where it can find something so I mean it's not quite a derivative not using it but certainly during the ARM64 stuff for like a year we didn't have Calib-C which means we didn't have an in-ramps so I find myself not using in-ramps on real things quite a lot just because it's broken so the degree to which things can still work without that is worth bearing in mind and in fact things need to work moderately well where you lose a few things but it's not nothing too critical just old-fashioned booting just knowing it read just boot and fix up anything that's broken yeah I mean that's fine as long as you don't have anything very complicated one thing that needs to happen in the derivative stuff is integrating these patches any information about those into the patch tracker the package tracker we need people who know Django Python and the census itself probably needs a little bit of work as well that's all Python if there are any volunteers or people interested that would be great doesn't look like it right now maybe you're not ready to admit on camera I've got a question for the room speaking of tangenting off what are people using as installers or using Debian installer or using Ubiquiti or something custom what's popular what is popular in terms of being used by derivatives we have Debian installer heavily preceded and customized so it's kind of a one-click install and then we also have an image-based install we basically take a clonezilla snapshot of that installed system with the steam and everything on it which is I think much more for what we're doing much more effective Debian installer is way more complicated than we want to inflict on our users basically at my workplace we use DI preceded one of the issues that I found for Jesse is that you always get a grub prompt where do you want to install grub even if you've only got one disk it's kind of annoying for Jesse but I'm hoping we can fix that it's problematic because we do both installs on physical servers and virtual machines and therefore the device name is different between the two VDA versus SDA because we're using Vert.io so we use Debian installer one of the things that we're doing because we're feeding the OpenStack bunch they use a tool called DiskImageBuilder so they end up with an image that they raw boot the image rather than using Installer so really the installer for us represents the way the developer on the OpenStack side can prepare their stuff and for the DiskImageBuilding process one of the problems with image-based installation is that you get stuff on the image that's specific to a specific system like SH Hostkeys how do you guys deal with that in SteamOS and HMX? okay there are other files like machine IDs and stuff like that so SteamOS SSH isn't installed by default so we don't have SSH keys and we set the machine ID we have some scripts that run after the imaging process so we can set a unique ID for the machine at that point I don't know of anything else that we need to clean up that's per machine there so Debian LiveProject does some of that sort of stuff as well I wonder if we could merge all the things because there's like cloud in it thing, Debian Live thing and probably more things ubiquity it sounds like that's something we want to tag basically in packages I mean for like in systemd for instance there's the machine ID file and if there was a way for us to just tag it as this is the file you want to remove if you're building an image like this is unique to your installation what about not generating those files in the first place how can we achieve that like a very well how would you go about doing that basically that's my question in our process what happens is the cloud development team ends up building an image with just exactly the packages they want and just exactly the configurations they want that's built into a hard image that the customer can't crack open they have no access to the OS in any form or fashion there's no way for the customer to reach in and touch it so there are actually no tools that allow them to do that that end up on the final machine so part of the reason that we have a really flexible a process for people to pull an exact set of packages and exact images and exact set of configurations so that they can build that kind of image and they call that compliance so the embedded people using embedded W often wanted to cross install so tend not to use DI and usually use multi-strap mostly because Debootstrap can only use one archive and we usually found ourselves using mostly the Debian stuff or the MDebian stuff plus some other bits possibly some proprietary bits whatever and it was much easier if they were in a separate repo and Debootstrap's crap for that which is really irritating because otherwise it's great so multi-strap because it uses app rather than some shell can do arbitrarily complicated things but because of the way D package works it can't run the pre-in scripts which is really annoying and mostly that doesn't break but some things do like MySQL it creates its users in the pre-in script which is fucking irritating it used to anyway so you didn't get MySQL users that didn't work so you end up basically the mechanism is that there's a script you need a script for any fix-ups because you didn't have pre-insts and another script for effectively first boot stuff because of SSH keys and things and yeah a proper mechanism for first boot would improve our lives greatly and what do we think that should be dealing with the multiple repository problem either by making Debootstrap cleverer which is hard in shell or fixing the problem with the pre-in smithing which is really hard either of these would be a massive improvement but yeah pretty much nobody uses DI it's all image creation for various reasons and often cross-image creation which adds a new little set of wrinkles and things to go wrong My suggestion for the cleanup things it's basically just a .d directory with scripts where packages can drop their files into and then what our clients like they just do run parts on that I kind of propose that for anything that can be done without having to write shell scripts we should so we should a lot of these are just you need to remove a file and it will be recreated on the next boot by lacking a script or a system unit and if we can all we would probably need to do would be to add a new control header to the binary package that is just like the com files fields but instead says files to remove at first boot think of an M for it and we would only need to agree on that there would be a couple of other things where you do need to do something a bit more complicated I think that the pepper I think needs to be deep because it includes it sets itself up with local dependent information that sort of thing there's a couple of other things like that but mostly we could deal with it just by removing files and I think we could make that declarative and in fact we discovered from messing about with this that a lot of packages deal quite well with having some of their config taken away and they usually do the right thing and put in a sensible default did quite a lot most of the low level stuff actually does a sensible job isn't it more to do with the problem that a post-it conflates things which should be done on the image and things that should be done on the instance and what you actually want is two scripts so I mean like a generating SSH key you only want to do it if you're actually booting a machine that becomes an instance or an SSH key is a particularly interesting problem there is a pepper that somebody that some security folks put together a while back which surveyed the internet for insecure host keys and it turned out that most of those were due to were due to people who had naively separated out this image and instance thing for SSH host keys and then they ended up generating a host key in an environment where by definition they had no entropy so it's a really bad idea to generate host key on first boot don't do it put dirt in an environment where you actually have entropy instead isn't the trivial workaround for that to basically delay the SSH starting until you actually have a bit of entropy except on anything where you actually need SSH you probably need it really early I mean cloud instances say you need it right from the start just in order to provision the cloud instance you need SSH so I mean the only other alternatives are things like installing Havagd or something along those lines that assumes that your per instance thing is on first boot I didn't say that so whatever is baking your instance should do it and have lots of entropy and then should push the button which runs the post-post-inst which actually sets up the SSH key in the machine ID and reinstalls lib paper or whatever right so you could have an argument for post-inst which would allow you to do that crap later or it could put it into a trigger that doesn't get run if you're inside the bridge to happen the environment variable or whatever is set that stops so yeah the instance setup could be deferred in a way that you can tell a post-inst of a package needs to care and we can probably write a list of them in the room there are probably five of them or whatever that can say but you know a fairly small number of packages could just opt into a mechanism where you say I'm making a system image I'm not making an instance, please don't do instance stuff and let me know how to do it and there could be a trigger or a way of re-invoking those post-ins later to say now I'm making an instance please do that so you bixie whatever would be just be able to poke those bits just to add one more thing about SSH keys in many cases it's completely reasonable to try to generate an SSH key on a cloud VM or something so long as you have proper you could possibly have a hypervisor hand you high quality random number generators there's some work on that but anyway so I did a bit of work on this sort of stuff on the wikipage reproducible installs a way to build an image or a file system twice and get the same thing but the main thing I found if I remember correctly was the mandib databases that's a that's a time stamp because that's a time stamp because mandib needs to know when the database it needs to know the time stamp of the tree that it's comparing the database to so that it knows when to regenerate could you look at the use the time stamp of the latest file like the latest the latest time stamp of all the manual pages you really really do not want to el-stat all of user share man it's not quick I think if I sped up mandib some more in other ways then maybe it might be justifiable at the moment it's a really important performance optimization people get very upset if I break it there might be some tricks like actually probably a sensible thing would be to touch the instead of putting the time stamp in the file touch the database to the right to have the right time stamp itself and that's probably the fix for that I was just doing this but I if you did the time stamp on the file that would change the images also right that could predest me and haven't thought about it very much now that you missed is a problem please file out as a bug so that I don't forget about it and I can certainly work on moving the M time out of the DB so related to the declarative config for files to remove at first boot where you want to get rid of like any admin stuff or similar and also to reproduceable installs is anyone solving the problem with the host name of the host where you are running the CH route leaking into the actual system you're building because there are plenty of maintenance scripts relying on host name information for setting up configuration files which are quite painful to locate especially if you're not at any time logging what's going on I guess you could do two installs with different host names before and afterwards and then compare them but I've only the river distiller installs things was only the bootstrap I haven't got any further than that so it would be quite handy to have something like host name namespace available from the outside to set for the CH route just fake the host name for a reproducible boot no matter where you're building there is a tool with Linux's namespacing feature there's a thing called the UTS namespace and you could just use that and you can set the host name in the CH route where you're doing the build it should actually be fairly straightforward to do this with nothing more complicated than the unshare utility unshare dash dash UTS and then run hostname inside that problem that's probably easily solvable it's probably worth, I'm curious what image building utility you're using because probably worth your image building utility growing this feature natively so we basically have our own tool Gramalive based on 5-year install and I mean we can't control it anyway but I'm just wondering if anyone has already a working solution like we do this already so but yeah thanks for the hint General, would it be helpful to have packages not actually copying the hostname it's a configuration should we as a Debian project strive for that should we encourage people to not do that I see various mail servers copying the hostname and people tend to forget to change that at this point There was some discussion in Josh's system detox this is my hostname solving this and storing the hostname in one point there was one and also speaking of systemd there was talk of systemd having what's called first boot or first run configuration where it tries to boot up with an empty etsy and this configuration like just the minimal configuration needed for that one system which actually might be have people been playing with systemd in derivatives and just going full ahead with the assumption that the only thing you care about is systemd I know that Tanglo has been playing with this I don't know if anyone else has been experimenting with that we're basically out of time just for okay we started doing new things that have been just doing we started to assume that there's no point inflecting a migration on people for new stuff we'll just go straight to systemd we have systemd in our repose we informed all our consumers that they can pull it and start playing with it but the default is still all-star Colm has wanted to finish that migration to systemd do you want to say something the only thing that refrains us from migrating to systemd and our easy tails image is the weights and those kexec which is not really compatible with kexec tools right now we can discuss this later Paul asked me to comment on ventures migration systemd I don't know if that's the place but we're working on it it's partway through we aim to be done somewhere in this kind of meta LTS cycle so by 1604 hopefully well before that but I'm not one of the people really working on it so I can't give you exact details so thanks everyone for coming I guess we'll close here