 Så jeg tror det er tid til å begynne. Min navn er Petter Reinholtzen. Jeg kommer til å snakke om den Debian-boot-systemet og hva vi kan gjøre til å begynne det. Først ... Nå har jeg vært i Debian. Jeg har vært med i Debian-installer i Debian-idue for en lang tid. Nå har jeg vært på Debian-gis, Debian-java og mange andre ting. Jeg har vært med i Linux for en lang tid. Jeg har vært med i Linux i kernel patchen i kernelen. Jeg har vært midt i 90'erne. Hvis du har vært med i problemet med å runne ut av file-deskriptorer når du ville logge innen til en overladet system. Jeg er den som har vært med i en prosess til tre prosess. I dag kommer jeg til å snakke om den Boot-systemet. Debian-systemet har flere opanskninger for Booting. Jeg har vært med i en summari om hva de avværende opanskninger er. Jeg har vært med i en prosess til køren og å demonstrere hvordan vi kan solge problemet. Nå har jeg vært med å spille hva pakketmentaner kan gjøre til å make dette en realitet i det defaulte Debian-systemet. Partiet som jeg har vært med er gjørd med studenter i summari-kode på Google de seneste året. Han har vært med å spille boot-systemet. Det er bare en fraksjon av hans arbeid. Debian-systemet er en kompleks og en viktig del av linuxinstallasjonen. Hvis det ikke fungerer, systemet ikke boot. Det er så svært. Normalt er systemet 5-boot-systemet som jeg vil gå i detalj litt later. En veldig åld alternativ har vært med file-boot-systemet som er bare en optimisering av systemet 5-boot-systemet slik at du har mange små file- og sim-links, du kan se alle fra en file som speeder opp bootet litt. Det er en ny effort fra Ubuntu til å adjøse til den nye ordene av kernelet. Det er en oppstart-system. Jeg kommer ikke til å cover problemet med event-basisere boot-systemet i dette tal. Det er en interessant og kompleks problem. Men jeg kommer til å fokusere på en annen del av boot-systemet som ikke er interaktivt med kernelet. Det er også in-it-RG boot-system som var eksperimentet med i Debian. Det er en dependensi-basis. Det er flere fleksibelt enn system 5-boot-systemet. Men det trenger hver pakket til å reivere de kjølste boot-skriptene. Så jeg har aldri hatt det. Det er en minut. Det er to simulere projekter for å få en dependensi-basis i Debian med alle simulere projekter. Og bare til å mennes brevlig det meta-init-initiiv taget for en veldig dag til å løse det mye av manualet editering det pakket maintainere har å gjøre å generere boot-skripte for de kjølste boot-systemet. Jeg er ikke sikker om det er en solusjon men jeg tror vi snakker om det during a boot-system buff during this debakon. Så å get started with the meat jeg er ikke sikker om jeg har løpt at the boot-system her så jeg har startet with the question om jeg har løpt in to the ETC RC directories once in a while. Det er god. Det er en masse av de mennes. Det gjør det sikkert for meg å know that I'm not missing the audience. Så det point er at the boot-system starts by running a script that starts all the scripts in rcs.d. They are started in sequence with the start scripts first with the start scripts with start us argument and when all of them are done it continues with the run level directory which is normally rc2.d and runs first all the stop scripts which are the scripts starting with k and then the start scripts which are all the start script starting with capital S and that's like all there is to it. Of course the complex part is what's being done within the scripts but the boot system itself is very simple. And there is also the small complication that .sh scripts are sourced within the process that is running this script fragments and the ones without that .sh are executed directly so they are in a separate sub-process. And I'm going to one nice way to visualize the boot system is to use the boot chart program and this is the boot chart of my base system running in QMU. And as you can see here the scripts are started in sequence and some of them are running for a longer time and some of them are just running for a short time. One can see here that the .yudev system starts fairly early that's the one initializing devices and loading kernel modules. That's the event-based part I'm not going to talk much about. Then you have the networking starting, you have DHCP running, you have syslogs started, you have different croons and get the and stuff like that. I'll just continue. So that's the current boot system in Debian. And these are the script fragments that are run normally during boot. As you can see there are s01, glibc, s02 hostname and so on and so forth until s99 stop bootlog bootlogd single and then it's continuous in rc2.d with s10 sysklogd blah blah blah s99 stop bootlogd. So it's fairly easy to get an overview of what's going on during the boot. As always the details are what's making the complexity but the overall sequence of the boot is very easy to get a handle. The problem with this approach is that the numbers are fixed. So if you need to put a script between s30 and s30 for example between checkfs and procps that's not going to be very easy. If you need the procps to run earlier than s30 checkfs then you have to renumber procps or checkfs and of course there's only an argument which package should actually modify it. So there are it's not too flexible but it's kind of working. And the same is happening during shutdown. The run levels 0 and 6 are used for halt and shutdown and as you can see here the scripts are running in first the stop scripts the k scripts the start scripts and the important question to ask oneself when one is working with the boot system that is this really correct and I can tell you that it's not. There is a small hidden bug in this list and it's not for all to see and it's not for all to experience but those that actually experience it are really host because there is no way they can actually shut down the machine correctly with default Debian boot system. The S20Sensex script is killing all the user browsers and the S31UmountNFS script is unmounting all network file systems. If you are running a network file system that is not NFS but some other system that for example needs the PPP dial up connection the PPP demon is killed by Sensex and the unmounting failed because there is no access to the network anymore. These kind of problems affect the Debian boot system at the moment and that's just one example. There are several educations where the boot system is wrong. So as I said, the current problem is it's slightly wrong and it's not very flexible and it's very hard to pick the correct sequence number because sometimes the correct sequence number involves three or more packages changing the sequence of their init scripts and it only affects those that install all of them so it's very hard to convince any of the packet maintainers that this is the right thing to do. So that's probably our early hard part of the maintenance problem and another problem with the boot system is the shutdown part because to go back to this one when you see this you would initially believe if you know the system 5 boot system that the K scripts are under the stop argument and the S scripts are under the start argument but that is not actually true because the Debian boot system maintainers know that this system is broken so we actually made sure that the start scripts also start with the stop argument because otherwise it doesn't work at all. So the S scripts should actually be K scripts and reordered to match the well to be ordered next to the K scripts that are already there to avoid confusing users that will try to install the start script and run this well expect the script to be run with the start argument and discover that doesn't actually work and the last part is it's impossible to parallelize the boot in Debian at the moment because a lot of scripts with the same sequence number can't really be started in parallel they have to be started in alphabetic order and that's not really what the boot system is supposed to supposed to that's not how it's supposed to behave because when the sequence number is the same they are supposed to be independent of each other that's not the case in Debian so how do we solve this one easy solution is to document the dependencies in the scripts using the LSB headers which is possibly at the moment and when we have all the dependencies documented we can actually verify that the boot sequence is correct and if it's not we can change the sequence numbers and this is an example of a header this is from the networking script with a few additions as you can see here you have the required start part which is scripts that need to run before this script is executed with a few magic tokens the dollar local FS is a marker to indicate that this script needs the local file systems mounted before it's executed and that's mounted read write before local FS is available the root file system is mounted read only and also there is some information about how it should stop it needs to stop before I have up down it takes down the interfaces and it also needs local file system to to be present and writeable before it is stopped the reason it actually document or it's putting some state information on the disk I'll skip the two next ones and go for default start and default stop which are the run levels where this script should start and stop and as you can see from here it's run level S very early part of the boot so networking should start very early that might be correct or not normal traditional unique systems do not take up the network in single user to be able to do certain maintenance jobs in Debian we have always started the network in the S run level and that's already been executed before the single user run level is executed so it's network already in single user that is probably should address but that's not too important default stop is 0 and 6 so it should be stopping network when you take it into halt and when you do a reboot and recently the to X start before and stop after headers was added to this system it's a proposal from SUSE we have already implemented dependency based system 5 boot system and the advantages of that header is that you can specify that your script need to run before another script for example we have in Skull Linux Debian edu a script that configures X at boot time and it has to run before KDM or XDM or GDM so it's actually configure the X before X starts having a header like that makes it possible for your packet to document this requirement instead of asking the maintainer of the other packet to list all the packages that want to run before it's a script there are similar features like post fix modules that need to start before post fix then you can specify it in the module that it needs to start before post fix instead of having post fix list all the modules that is present in Debian that needs to start before it or start after if you have similar requirement when you shut down the machine and when you have this dependency information available you can actually make a graph of the dependencies and that's the graph you see on the right side and it looks like this when you increase the size of it Glib C is the first script that has to run then you need to run the to mount the kernel file systems then you load the kernel options and there is a bit more on the side here modutils if you have it available mounting file systems below dev starting the device mapper and as you can see it has reverse dependency on udev hotplug and discover and the last one is devfst so if any of them are present they have to start before libdevmapper so I'm not going into detail on this graph it's just showing that the the order the script has to start in to be able to fulfill the dependencies documented in the header of each script and when you get to the end of it the system should be ready to go here you can also see an interesting feature you have to remove the no login file after kdm, xdm and gdm is started to be able to login you can also see here that there is nothing within the dependencies that require kdm, xdm or gdm to start very late in the boot you can actually move it earlier to get a easier or a better interactive feeling when you boot a machine so your login screen pops up while the system boots and continue to start demons that are not needed by the login system so that's all nice and yep or we should get the microphone that's just a comment on the kdm, xdm and gdm case where you said it can be started very early you have to keep in mind that maybe your home directory is mounted via NFS and stuff like that so you probably can't mount it or you can't start gdm or kdm that early because then users won't be able to login just to absolutely they have some dependencies on their own so they are not installed on this system that's why they are not showing up with all the dependencies on the graph but they do depend on local and remote file systems yes so when these dependencies dependency headers are present in the scripts you can check the ordering and detect bugs that's what the script in in-serve is doing it's first mentioning that two scripts are missing the headers and then it's documenting a few bugs in the ordering of the early boot system cruel as dependency on correct time set and time and it's running before time is actually set syslogd also have a similar dependency and it's running before time is actually set and that's using only the headers in the files for a base installation in Debian because it's a lot of work to get all the dependencies in all the headers I've implemented a system where you can use overrides in a directory included in the in-serve packet to provide headers for the scripts that are missing them and if you use that you get this list there are still bugs in it but at least you are not guessing the dependencies on the scripts that are missing them and here you can see that the reading of the hardware clock is done before the local and remote file system is present even though it actually depends on both local and remote file systems to be available that's a long term bug in the boot system which affects everything with the usr and var on a different partition than slash so not affecting too many but it's annoying for those that are affected but the point is that you can actually verify the boot sequence and detect bugs in it if you provide these headers and this is the same check for a desktop install very similar a few more scripts missing it 8plib, kapsus, hotkey, setup and anachronis missing headers so they don't get the correct dependencies basically the same bugs in the boot system because the bugs are in the early boot system very few of the later scripts have problems with the sequencing so detecting bugs in the boot system is one thing and that's my main argument for Debian implementing the headers there is no way to detect bugs if you have these headers you can actually reorder the boot sequence using the dependencies to make sure it always stays correct and that's the second part of my talk where I demonstrate that this is not a theoretical possibility it's actually implemented, it's working and it's possible to enable in your machine on edge today of course the headers are still missing and wrong for some cases for some packages so you have to be very careful but it's really possible to convert Debian to use a dependency based system 5 boot system today and the steps are fairly simple, you install the insert packet which is a port of a packet SUSE is using to insert all the init scripts in their boot system then you convert all the broken simlinks in RC0 and RC6 to stop simlinks then you replace update RCD and you reorder all the RCD directories with the insert program and then you have dependency based boot system which will stay dependency based because update RCD make sure that all the new scripts are inserted in the correct place in the boot sequence based on their dependency information and the majority of packages already provide the information so for the common case for desktop installation fairly normal installation with Samba either the override files or the packages themselves include the information required so you can use it today of course there are I don't know how many thousands of packages with init scripts in Debian and not all of them are converted but most of them are getting a correct sequence when you use them there is a small problem with the shutdown sequence so you need to be careful when you do it it's still experimental because the headers in the files are wrong in some cases and when they are you will get a really strange sequence if you get loops in the sequence you will get any kind of sequence for the boot so if your boot system have a loop in the dependencies do not use insert that will break your system and another problem is that when you use dependency based headers you need also to make sure that you register the scripts in dependency order so if your packet include two scripts you cannot install them in any order you have to install the first dependency first and then the script depending on the first one if not the first one will not be included because the init update RCD script will refuse to install it because dependencies are missing anyway to enable it you run the d-packet reconfigure on in-serve with environment variables set the bad in-serve hacker equals true and when you do that you are presented with a question do you really want this or not and if you flip that to yes and press enter this will happen it will take a backup of everything in the current boot system it will reorder the sim links and replace update RCD and when it's done you are over to a dependency based boot system and I'm pretty sure none of you actually remember the look of the last boot graph but I can tell you that the boot sequence is fairly similar which is good because we didn't really rewrite the boot system we just made sure that the dependencies were used to provide the sequencing and this is the new sequence after reordering the scripts based on dependency information you might notice that some script sneak managed to get into the sequence between glibc and hostname libdevmapper has moved a bit earlier in the boot and this time all the scripts with the same sequence number can be started in parallel if the dependencies are correct there is not much to it you can see that the number is a bit lower and it doesn't really matter because the numbers are assigned reassigned automatically when you insert or remove a init script so the numbers are just the smallest one they could use this is the shutdown sequence and I can assure you that there are some bugs in it for example the k90 reboot script should have been renumbered I suspect there are some leftovers from the suce boot sequence that is messing up the shutdown sequence in debil, I'm still working on that one but it's currently fairly correct and it's not breaking the shutdown for the common case and if you're not brave enough to continue this adventure you can actually undo it as I said the first thing the script is doing is taking a backup of the current boot system so if you rerun the reconfiguring script and answer no it will restore the backup and you will be back to square one with no changes to the boot system if you have not installed any new packages and if you do it only once because if you have several backups it doesn't really know which backup to use and you have to do it manually but there is a hard gset file in the vartlog where you can get the old boot system extracted so it's not a one way street where you are completely lost if your boot system is host after running this you can actually revert it and continue from where you are and last how can the packet maintainers assist in making this happen first of all make sure your scripts have dependency headers and lintian is warning about the missing headers at the moment so I know quite a lot of you have already included them in your script of course the first step is to get them in there the second step is to get them correct and to get them correct you actually have someone someone needs to test them and use them and that's another important thing if you can start using dependency based boot system and detect problems and report bugs and get the maintainers or the packages with bugs to fix them that will be very good there is a weaker page explaining how to specify the dependencies in the header also you need to make sure that the scripts are registered in dependency order not many packages have several in it descripts but if they do they have to do it in the right order either edit Debian rules or edit the post in the script to make sure that the registration order is all the reverse of the dependencies the depending scripts has to be last and the dependency scripts has to be first and all this is correct you can even run the boot system in parallel to actually speed up the boot and then it looks like this this base system doesn't really have many boot script so it's not showing the big advantages but when you have a desktop you will actually speed up in a few seconds by just turning on parallel booting so that concludes my part, any questions there's one over there one problem I've had with the current boot sequence is it supports LVM on RAID but not RAID on LVM which obviously is one of those things depends upon how you configured your system static stuff in the init scripts is not going to be able to handle something like that is there some way that this is going to be handled in the future or am I going to have to manually tweak the boot sequence to get my system to boot properly yes and no the override files is made in a way that some override files can be included in the insert packet and some can be stored in etc and if you have different requirements for your system and the LVM over RAID is not the only one there are other systems where you have to start LDAP early because the system needs LDAP to boot and other systems where LDAP server is just a server provided for other machines so you can start later and these kind of things will have to be adjusted on a per system basis and for that setting you provide your own override file in etc to specify what your boot sequence has to be like or actually what your dependencies are and then the system take care of reordering the boot system to a correct state question over here so I really like the idea of checking the correctness of the boot sequence and I think it's more feasible than like changing our boot system to a radical new one and so I ask you if you have ever thought about harvesting our packages collecting all initd we have checking the default run level which they are run and like doing some QA work reporting bugs against packages I haven't thought about it but I have not had time to look into it I hope someone will start at work when the headers are available like they are now and actually try to collect all of them and see if they can form a consistent system Just a quick comment for anybody who might want to work on something like that to be aware of that the copies of the init scripts of every package in Debian along with many other things are in the Lintian lab on gluck.debian.org for Debian developers you can do all sorts of really interesting package analysis by looking through the Lintian lab because it has an unpacked copy of every package it doesn't work if you enable this system does parallel starting of init scripts get enabled automatically or do you have to enable that separately ok, because you said it is possible in principle to run the scripts that have the same number parallel but it isn't done by default if you enable the system to enable it you have to run this shell command set concurrency equals shell in ETC default RCS and that works well that's possible to do even today with the current boot system the sequence is wrong you have to modify the sequence by hand to get it working I'd like to use the chance to make a quick drumming for the meta init idea the idea is that maintainers should not write the init scripts by themselves or just copy them from somewhere because it's just tedious work that's going to be repeated from package to package instead what we are proposing is that wherever possible the demon is simple enough that you just describe the demon with two or three lines how to run it what dependencies there are and then upon installation the init script will be created and it will get the right number hopefully in the system so if you think this is a good thing to have but don't want to care about it you might want to look in providing meta init files for your packages it's still very alpha we are currently developing it so if you have questions just talk to me I'm also interested in people with packages that want to test it and want to service guinea pics for the system I was quite surprised in your instrumentation here that your system was it seemed to be CPU bound on boot and I know that my systems don't feel like that I don't know whether they are or not but I wonder whether you had any opinions about this is because of the simulation environment you are using and whether your parallelization has how much of an effect that has on multi-processor systems I can assure you that the lack of in-out problems on the machines is due to the fact that the entire disk fits in memory that's what I thought there's one question over there this isn't directly related to the work that you're doing but it seems like the work you're doing a lot of Debian packages have some variety of different weird hack to say I have this demon installed but I don't actually want to start it it would be really nice to have a way that we can tell our users to just use this method which doesn't involve renaming simlinks by manually inside the XERC directories you have an override file for dependencies does the override file also let you say don't start this at all no when you want that you actually have to modify the simlinks that's the correct way to do it to switch from start script to stop scripts for that packet do you feel like that should continue to be the correct way to do it and we should therefore write a tool to make it easier to do that or that there would be a better way of doing that in the longer run I believe that is the correct way to do it and that for example update RCD should have a disabled option to do that for us I just want to say does console let users usually disable and enable I don't remember its name it used to be the case I don't know if it still is but if you turn off demon starting then when the package gets updated security update it turns itself back on not if you do it the correct way the correct way is to rename the start scripts to stop scripts if you remove the scripts I believe it wasn't installed and installed it that's the behavior you are describing but if you rename them it actually works any more questions we are planning a buff about the boot system the time is not decided yet so if you are interested in joining please show up here after the talk we hope to get all the boot people together and talk about how we can make upstart easier to enable and this one enabled by default or the meta in its scripts more useful and working properly please show up I am the maintainer of upstart so I just wanted to add a few comments on that but you also said it I want to with you together organize this buff we haven't found a time yet but anyone who is interested maybe should come up front so we can find a time for discussing this and what I wanted to say because although I am the maintainer from upstart and I think upstart has great potential there is still room for improving our current init system so I think there are two or three points where our current system is lacking still and this is for once I think is the lack of policies for example the return codes of the init scripts some return zero if starting a script fails some provider option to give a status command some don't some behave differently when it's running and it started again some fail in this case some just do nothing, some yell and the second point that I think is really lacking is our current logging system I mean with the LS base package it got a little bit better base logging function but do you think it would be useful to push for a better solution there do you have ideas for that to implement a better logging system to probably integrate existing splash demons like you splash or splashy better in that case or other cases like syslog where you can use that for logging currently it's only printed to standard out mostly I have looked slightly into the use splash logging problem but I have not really worked hard on it and I have no better solution or ideas on how to improve it I do on the other hand plan to work on the progress bar situation with the system 5 boot system and make it easier to plug in your progress or your splash screen system into the default system 5 boot system to get progress information in there there are patches floating on BTS which seem promising but I haven't had time to actually look at it so that's the only thing I have looked at when it comes to synchronizing between splash screens and making things easier for the default boot system and with regard to policy do you think we should specify it exactly what return codes we expect I mean if I remember it correctly the example is the skeleton at the NAD recommends different return codes for eksempel then the LSB specification what to return if something fails or not As far as I know we have specified that LSB is the alternative source on exit codes behavior of init scripts so if the skeleton is wrong or not behaving as LSB specifies a skeleton script is wrong so I think LSB is the correct way to behave and when we have education cases we have to improve LSB or specify how it should behave What part was done by the Google summer of code guy He wrote the lintian plugin to warn about missing headers and he also did quite a lot of testing on the current scripts with in parallel booting and reported a lot of bugs to asking for headers to be added and managed to get quite a lot of packages converted to include this dependency information so a lot of his work made it possible to run this demonstration and prove that it's possible to do it today Do you have an overview of how much is missing like do we have 5,000 init scripts and 500,000 I don't know I haven't done the math I haven't checked the packages Thanks Any more questions? I guess we'll leave the rest for the buff If you are interested in the buff come up here and talk after the talk and we'll find a time that fits everyone Thank you very much