 Welcome everyone, this bot has been scheduled very spontaneously because the talk Peter gave yesterday. I'm the maintainer of Opstart and thought this might be a good idea to gather some ideas how we could improve our init system, how we could manage to transition to a new init system. I have some ideas in my mind, I want to hear from you if you think these ideas are crap or what. I just want to start with some short comments I currently see in our current system. Our current init system is just a bunch of shell scripts. Actually, if you look at one of the shell scripts, it starts to de-man and you check the return code and based on that you do a lock message and so on. If you have to re-implement this again and again and the policy changes over time we have a very inconsistent init system at the moment. We don't really have specified what return codes we have to use. There is the LSB and I think we should in the policy refer to the LSB, what return codes we'd expect to at least improve our current system. There's also the problem with our output, it's not very consistent at the moment. It got a little bit better with the LSB logging mechanisms that are in place but they are by far not that sufficient. Another plot problem I encountered when I started to help maintain code, maintain debas was that whenever we had to restart debas there was a problem that dependent services had to be restarted to. I think there are other cases where this is the same problem. So what can we do about it? The first thing about it is reordering the boot sequence dependent on the dependencies that's the talk Petter talked about yesterday and I thought maybe it makes sense to implement the same kind of functionality and invoke RCD. I don't know, what do you think? Would it make sense that we have something like invoke RCD with depends? We can restart that. I want this to be a bof so everybody should chime in. For some services it makes sense for others it doesn't. I don't know how to determine which one. You need metadata for the led-to-led services so that we can declare whether or not that makes sense. For example, restarting network might restart all services using network for others in certain circumstances. Normally they just listen to any interface and will keep running perfectly without network. Yeah, which right is the old NTP. It used to be that NTP had that issue where you had to restart NTP after you added new network interfaces or it wouldn't listen which was actually eventually fixed in NTP itself but other things do have that problem. How does it do? It sleeps and retries again? It apparently periodically monitors the list of available network interfaces from the system call. So this is one option to fix every D-man and make it pull and try again? We fix a lot of different things that way and already in like for example the LDAP server was always an ordering problem with all the different things you decided that they wanted to if you're using AnnaSwitch pointing to LDAP and LDAP is your local LDAP there's various different programs that have now been modified to be able to back off and retry So you can get somewhere by doing that and it makes it more robust when you can but there's some time when you can't. I think the problem here is that you as I said you'd have to fix every D-man and that you would have 20 processes sleeping doing essentially nothing. So there can be better ways than that. And what's the other problem with our current system? In our current system we can't handle situations where you'll plug in which basic where you'll probably have your home system mounted where you have your network connected by a wireless card that is off to you booted and our current system can't deal with that really good that's why I come to Upstart later on. Another problem is service supervision so we don't really have a way to monitor services down We start them when they are killed when they die so we'd have to refer to other tools like monitor, monitor, monitor, stuff like that but it sits on top of that and you have to use that But if you keep monitor, nothing is watching monitor and you have to screw which will be annoying. The nice thing if we can integrate this into Inet is that Inet has special semantics so if a child before a process dies you can monitor it easily so this is usually the process you want to do that. What are the better units I saw allowing you to have arbitrary numbers of servers and not to restart it? It's a real server that's really really important and one thing you care about is that it doesn't die There's a reason though why everybody invented why starting with demon tools everybody's been reinventing this wheel and without using Inet despite the fact that Inet's had its capability for forever and that's the interface to temporarily running a service down when you have Inet monitoring it is hideous For most Inets we have to comment out the Inet line and then tell Inet to reread its configuration file and that doesn't work The advantage of all these other utilities like Monit and Runit and whatever else is they have a control socket to the process that's monitoring that particular service and you can run a command line and say take the service down, bring the service up issue an arbitrary signal to the service a bunch of other things like that that are actually extremely useful in practice for doing things to that demon. I mean if somebody added that to an Inet I'd be happy to use an Inet but an Inet as it is doesn't have that capability Yeah, it's too stupid, isn't it? Well, Inet is not designed to have that capability It was designed a very long time ago before the Inets arrived so I don't think it can be even engineered to decently do it And there's a strong argument to be made for making Inet as simple as possible I mean in this process one and if an Inet dies So with all this said there were some attempts by Ubuntu you probably know it, it's called Upstart I'm currently the maintainer of it in Debian and I'm very interested to make it possible to get Upstart installed in Debian so we can really use it as an alternative maybe as the default in the future who knows and there are currently some issues that are blocking that and I'll come to that later but first some nice things about Upstart So it's totally event-based What you need are events and events are generated whenever you plug in a hardware device When a job is started itself generates an event so let's say Debian is started the job itself emits an event that's called Debian started and whenever such an event is emitted you can write another job that elicits on that event and is started on behalf of that the only event that Upstart itself emits a startup and from that on you start your boot process What that gives you is that your boot process is completely race-free so you mount a device when the device is ready and not some arbitrary number in the boot process and the actual idea to make the boot process robust and not so much what some call Upstart is for making the boot faster that's not the actual goal of Upstart but it's a nice side effect and I had some nice pictures that shows that I could by 50% my boot time went down because it also paralyzes it it just fires the event and many of them are started in parallel Are you involved in the development? Yeah let's say the upstream, the core development is quite different what I'm trying to do is not do the core development I personally want to develop a debas proxy for for you that's my personal goal to do that as a head project which is a bit separate from that but I'm listening on the mailing list I give comments whenever it's got once it's featured it has discussed so this is my involvement but the main author of Upstart is Scott James Brownman Are there any developers involved in Upstart? I don't think so but there are other distributions one I know of already uses Upstart as my boot system to drive which one? It's Frog Aware if you know that it's a smaller distribution which one are you? Frog Aware don't ask me how it's pronounced and there are some let's say 15 people who are around at the IRC channel of Upstart discussing things and stuff like that and yeah I put the URL Upstart.com Exactly there's a wiki where you can leave comments every new feature that is discussed Scott usually writes a spec where he wants to implement what he wants to okay what he wants to have implemented Well there are still two people compensating my but I think it's really nice about the concept of Upstart is currently the version that is experimental so you can install it today what it basically does there is one job called RCS that listens on the Upstart event when that is started it drives a complete system 5 layer compact layer so at the moment it uses your existing in scripts so you won't notice a difference this is the advantage that Upstart can be stressed as it and we can check if the code actually works and it does as today it does this very well and we can transmit jobs bit by bit to real native Upstart jobs and if you want to have the full power of Upstart we have to do that somehow somewhere to write native Upstart jobs now the problem is how we are going to do that for Danian if every package that chips in in script should do that for system 5 in it and should an Upstart job or should there be a separate package that chips all the Upstart jobs that have been written so far and then when you install a package it installs a corresponding Upstart job this is something I wanted to discuss with you mainly that's my concern there is still some work to do before Upstart can be a complete placement there is something called complex event config it's something when you want to combine 7 events together say if the network device has to plug in and this condition but not this condition this is currently not yet working what it currently does is it only listens for events and or's then so if that happens or that happens or that happens start it that's basically it but it's basically enough to try to complete the process with Upstart jobs alone and I'm using it already in my laptop and it's working pretty well so far unfortunately I can't demo it because my laptop I couldn't get it to work really so everyone who is interested in the format of these Upstart job files I can show it to you later isn't it just like like satisfying that even these command parameters that's a big difference between Upstart and System 5 in its script so it's declarative you just say okay this is the path to my demo these are the parameters then you say okay do you want to have and then you have sections called pre-start post-start but you can prepare the environment for the demo to be started so for example a typical use case for that would be to create a run directory or initialize something below the module yeah that's basically it you write down and all the other stuff is automatically handled by Upstart so if you type in start that's a command start job name it just brings it up if the demo is already started it just does nothing and if you type in in its CTL list it shows you all running processes or all running jobs and you can easily stop one job and based on that other jobs are either stopped if they have corresponding stop on let's say debuff stopped commands in it so they will be brought down automatically and brought up again when the demo is started so it's pretty easy to format how does it decide when the job is already running how does it decide when the job is already running when you go to start a new one for example so as soon as the process is spawned it says okay this job is started and it's fired this job is done so it's kept track of by the running so the other thing if you have a demo that is after it started it takes a while to initialize spawn other demons and stuff like that there is a post start section where you can monitor something like okay you have a tomcat instance which connects to a MySQL server and only after that it's considered to be ready you can ping your tomcat server or your patchy server and only after that your server is considered to be up so what happens if you after you upstart it just tells it to re-exit itself and reloads it and that's it it keeps all the state yeah no problem where is it keeping the state it's keeping it it's just reloads itself so it's keeping it in memory but do you want to keep the state in memory if you re-exit yourself what does it do now it just reloads it sorry it just sends a signal so it reloads the configuration that's all it's not going to be all binary until the next time basically well considering the fact that this was it the data info this in the previous session then somebody described something about this extra info in the lsp info it basically has the demon name if you can just have the parameter it's pretty easy to generate the obstacles from there unless something special has to happen yeah well I'll come to that later so just a moment so one thing I think that our current blocking to get upstart into the evidence of falling that our current system file in the package is required and essential so if I want to replace as in in it I can't do that easily if I'm not sort of work around the next line of wording in it but actually I don't want to do that I want to handle that in a different way there's also the problem that that our packages our maintainer scripts are expecting invoke rcd and update rcd so they have to be around all the time I think even during upgrades so if you look at the I think it's the sysvrc package which does nice copy around magic when you upgrade it or when you move it so I thought that maybe this could be handled a bit easier to have always an invoke rcd and update rcd around so my idea was to have a package called initbase where you ship very basic implementations of these tools which which only do nothing which is return 0 and then the actual implementation installs or the actual init script installs implementation of update rcd and invoke rcd does it sound like a plan I don't think that would work I'm not quite sure why but it doesn't sound like it's going to work for me because installing our boot system do not really convert the machine to use that boot system and like if you install this 5rcd you don't really have all the rcd directories with simulink system and you really need to keep the old boot system until you actually reboot with new one before you can ditch it so it's a complex problem to replace the boot system you have to be very careful when you do it after the machine is installed what happens if you install it at the current system as you replace as within it and when you shut down it just basically calls the old system 5 layer so there shouldn't be that much of a problem and I don't think all boot systems would have to keep the system 5 boot system around until they shut down so which init program to run is a parameter somewhere in the boot process isn't it can you tell me it's really neat you can tell grub to run a different init because it's been in there but you've changed it to something else I wonder if that's an easier approach to switching but I don't know actually you want to pass the parameter to the kernel like boot type so it's not an issue for init energy when you switch to init energy it just tell you to pass another parameter to boot grab that to it but I don't think init energy in that form is a complete replacement because it doesn't ship the usual tools like how that's already taken care of although those tools were moved into a different package already I know that but what I wanted to say is if you install init energy as it is currently it's not a complete replacement because you have to manually configure stuff like how to use that tools and not the old ones so if you want to replace it it has to replace aspin init boot option can be added by update grub automatically if there's something somewhere to use then update grub could insert the right parameter automatically so you don't have to use it if they don't want it I think the point is that some of the other system administration tools are expecting to run aspin init in order to do their action my plan is to replace aspin init and not work around somehow to rename it I think that's the only way it will work properly well I believe your assumption is slow because you have to keep the old one around until you boot and when you have to keep it around until you boot it's going to be aspin init and aspin hold for the few old ones well I could show you I can replace the system with upstart and you can shut down you're replacing it with upstart the transport activity rule what you have to do what you have to do right now that's correct I can replace aspin init process with upstart in the combat mode if we ever the idea of having a grub parameter is that on the first boot you install together with stuff you have like an update script that is executed on first boot and then removes the grub parameter again yeah but if you have removed the system file in the package and it's gone then aspin init is away and I don't want to install it in parallel because then the package can't keep track of it I did notice one thing bitoff is missing from upstart I had a system that isn't good because of that but not completely bitoff is missing from upstart bitoff is missing from upstart at least at the end it was quite funny because the device I was using relied on bitoff to find certain things to get further in it's good it's executable even shipped in sysfile that's one of the ones that says 5 init utils at this point PID of along with last and a few other and it's intended that way that these utils are shipped in the sysfile utils package and you can install them along so it's actually intended to work that way it should depend on the system if you install the upstart compart sysfile package it will install sysfile utils if we don't go this this way how could we make the sysfile init package not essential, not required to have a chance to install upstart if you think this idea is not going to work don't you think it is actually suitable to just keep the sysfile and install the package and via the crop parameter on the next boot just completely suitable to use sysfile and whatever I mean there's not many clashing binaries it's aspen-initab-clash as with upstart but aspen-initab-clash because you have a boot parameter and any other binaries that are clashing how? they could be they could be wrapper scripts check what's the current init script and run the problem is as far as I see it will be the upgrade until the first reboot but that's not a problem if you just replace the boot parameter because that's not active until the next reboot and then all the binaries that are called while the system is running just check what's the current process 1 what's the executable name aspen-initab-clash and depending on that do the right thing so I think that would work the nice advantage is that you always have the sysfile-initab install and in case something goes wrong you have the choice to boot crop with another init parameter and you have another init system the problem is that it requires manual editing I think you could query or check for example even with the process name you could query which init system is currently running and the wrappers would have to be a bit more intelligent and therefore a bit more complex but still they can query what is running and just run whatever is needed based on it there is a minor inaction with the crop because you can have crop update have this functionality so you can just can you pass the parameter to update crop because you have to manually add a boot crop manual no no the kernel install does it for example you just call update crop in your post-inst and of course update crop has to know about the fact that it has to decide what init system to install if it could either have a priority thing or there could be an alternative or any way for the for you to decide what init system to run it does not involve editing boot crop but maybe this is the init system I want or just debug config upstart makes it itself insert to the crop so the question is actually how can you modify the crop system to be more adaptable to the script it's not easy to do without finding a way to do it but it's impossible to do if the script will be like something you're using well so you know and you know the file you just type in init equals sbin upstart wouldn't it be easier to make it to a stage install like you want to install the package I just want to go on with it's really short actually so basically it's a two-stage install install the package the first thing it does puts the binaries in a place where it knows they are that's the grow boots into single user mode actually kind of thing without run levels where it knows that it's boosted that script installs all the stuff removes the old stuff and then reboots again then you start from a clean system I mean what about the old package which has removed it it's still recorded you only remove it afterwards you only remove it when you're in a single user mode where the install script runs it so you don't have a run level because you're in single user mode so it's just one 2RM binaries without having it removed from the the package runs I think we can conclude that switching the root system while it's running it's a complex problem as I said if I go the root system in it, upstart plus compact layer it's not a problem, it actually works today you just install it, you just download and if we really want to move to the system we could make it upstart plus compact system for one release for the next we could go to the new in its upstart jobs so this would be a possible upgrade what do you think? I don't know, it says in its script the nice thing about that we could replace the system in a package with upstart it has basically the same functionality as it has today because it uses the system in its scripts so we don't lose anything and we don't gain anything much this compatibility mode that you said upstart has can be even used in different systems because it's just a binary that calls all the Rc files in order exactly, it just calls the old system file in its scripts so this could be even from upstart put into the base in it whatever indispensable in it that we will have well it doesn't tell us really it's not possible either because it's in the binaries this is basically my idea to have that to make it really possible to have Sys5 in it's not being required and essential to introduce this new package make this one required and essential and let this one depend on the actual implementation default so any init system can provide a complete implementation that means a working invoke rcd and working update rcd plus driving all init scripts that are currently available either through a compact layer or through native jobs could have this provides init system so you could really replace it do we have a documentation describing what that good system is supposed to contain like update rcd and all and all these things I mean update rcd and invoke rcd are the current APIs I would call it in devian for managing init scripts so I think this is a starting point to yeah but that's not complete there is all the all this I could write down this it's probably useful but they can compare the available init systems and see if they actually fill the list or not yeah if we can define a subset which is interchangeable the schema disk for example gdm calling this binary for shutdown and stuff like that that would be if they're all too different then it gets a bit very good well even if you can't get between different systems still going with this gives you the option when you're writing custom installers and things that you could have one install process that gives you cispian it and another install process that gives you upstart yeah that's another idea so if you're installing from here that might be easier to just decide when you create installer what you're going for and make it possible to keep upgrading your system so they don't get a new feature what happens with rather than my load what happens if I kind of trickle to change the old systems in glider are still running unless the manual process to change is done you can't just apt-get installed rather than changes over no it doesn't you can try yeah well that's not what we're doing maybe we just we did 5 to 10 years on the old system it's a very simple problem obviously it's nice to be able to change okay there's another problem I'm currently dealing with and that is how I want to get native upstart jobs or native entity jobs or whatever having installed on your system currently as I said tries to complete system 5 layer so how should we do that there's this approach called metaminate we're going to describe it in a description language and you generate the start file the restart child file for your system in use but I think there's one there are a few problems and this is yeah that if you that we have problems passing it with different developers use different systems we will have possibly problems to verify that the boot sql is correct someone using upstart might not test the scripts or the metaminate scripts for the current system for example we also discussed that metaminate only targets the simpler scripts so we still have the more complex initialization scripts we do the stuff like that I don't think I'm not sure if we can manage to look to handle that with metaminate so there's still a way that we need to have this native start shops for a different system to have to replace the corresponding system 5 in a job Claire somehow what I wanted to say since the metaminate idea is only a few days old maybe we shouldn't start with those like possible problems but maybe first describe it but the metaminate idea is already I thought it was already involved so you discussed even if this metaminate can solve one problem it's actually the quality of the of the start up scripts because there are like start up scripts that like this are spawning seven shells and doing lots of things while one command can do it which actually slows down your boot also I don't want to vary this idea it's not my intention but what I want to do in this metaminate idea is to actually get an idea how many scripts can actually be handled with this metaminate idea so we have our numbers so I think we have around 1,000 in the scripts at the moment in the archive and when we could say ok we have 900 that we can handle with metaminate this would be an achievement so only two out of four are mine to take a random figure yeah I think there's a lot of in this script that do strange things I have one that has a check for the kernel module being loaded and another one that checks a bunch of stuff around whether or not you already have all that database no but as we already discussed this of course discussion changes when implementing but as we originally discussed this ok not every package can use metaminate but most things can be specified I mean not only the most basic installation start up scenarios but also some of the ones that have a bit of complexity I do think that I don't know throwing just a number but over 90% of the packages we currently ship shipping in a script will be available with metaminate so my question is how do we deal with the remaining 100 in the scripts even with the thing is it's a hard question how to replace them in a running system well sure but if you see the scope of metaminate as a way to make most of the scripts consistent and be properly and efficiently for system 5 then it's perfectly possible to generate a lot of these scripts with templates and if you then reduce the number of problematic scripts from 1,000 to 100 the problem is suddenly possible to grasp the remaining we still have it now even if it's very easy to migrate we have to get 1,000 people to migrate the scripts even if it's only a good thing it's not that easy it's also what you propose with upstart worlds but you have to get people to migrate yeah sure this metaminate could deal with the 90% and the remaining 100 I don't know how to tackle this if I as upstart maintainer should ship these 100 in scripts for the more complex ones in my own package and on install time instead of installing the system in it counterpart I take it from my package and test it from there would that be an idea? should the packages itself ship system 5 in start scripts and upstart start scripts? I suspect that the most complex good scripts that have good systems are the ones in the init scripts package the only good system is very complex and it gets a lot easier when you have a file system that actually is writable and enjoyable and if you look for example at the bluetooth start script it's pretty complex and long so this would be broken up in smaller shops usually within upstart where you actually start very little pieces only and react on for example when a bluetooth device is added stuff like that you file a bug in and it is easy for each application to supply upstart scripts at the moment they supply upstart scripts they should supply upstart scripts the problem is that it takes much longer to get started if I want to say people you can install upstart and you can run it now if I have to wait until all the people get there you can get to the point where I can actually recommend people start using dependency based system 5 scripts it takes a while ok if you can do this to 100 instead of 1000 it's going to take a lot less time it might be that some of the problematic might be that some of the problematic ones in system 5 might be really easy in upstarted Windows what does the system use I think meta is a great idea regardless of whether or not we switch to any other different init system just because the majority of init scripts are badly written we can get to stop writing init scripts and instead describe what they want to do and want something to write a init script for them the better off we are this was something we need a better understanding of what our card init scripts are actually doing because another problem I see is that the event based paradigm and the dependency based paradigm are different so I'm not sure if we can handle many jobs with both that could be tricky what kind of feeling does meta in it already even contain dependency information well the idea comes from meta in it is but that's of course very often meta means to be headers so meta in it so where I myself see meta init where it can really help is that we get more consistent consistency with our current system 5 init so as I said before most of the init scripts are chest.demon shell scripts are duplicated a lot just copy and paste mostly from existing ones from skeleton skeleton gets updated and the return filter it's a bit messy maybe it's getting better or not consistent this way sorry to interrupt but even a bit more than being consistent for example when we implemented we adopted having the last bit headers well it took a while I'm sure there are many scripts which still don't ship that information the main the main help will be when going forward yes whenever we start requiring some changes in the way our scripts are made because the thing is we want to trigger well the original idea we had this VKG triggers which are not yet acceptable but whatever the thing is the init scripts can be kept up to date and right now they depend on the update of the package one of the advantage of the LSP stuff though is that we were able to write a lintian check rule for it and some of the things that we've discussed here are complicated enough that it's hard to do that for example I have no idea how I'd even start writing lintian check for your init script is simple enough you should be using that init it's very close to the scan yeah but I mean it's hard to do that sort of fuzzy matching in Perl actually I think Enrique could do that was it by magic or? no using zappian which is magic text search technology which is what dev tags are based on but it just you know it can do looks a bit like okay so I want to make a debut on this as far as I can see a slow organization to turn and that's part of the reason why I didn't jump the upstart back in Scott asked me about it and working on the development because I don't think it's possible to switch them into a new boot system in a short time frame I think it will be hard I think it will be hard but I think Upstart has great potential and because it's really provides a 100% compatibility I actually don't see why it couldn't replace it and then step by step we move to this new system and my effort to actually to make the system 5 boot system correct doesn't really conflict with that I'm well aware that the kernel's event based good features is unsolvable with the system 5 boot system I mean there are many problems I couldn't speak in detail is with when you initialize LVM on RAID and stuff like that where you have a lot of RAZE conditions our current system can't really deal with that correctly I think and it won't in the future and there is where Upstart really can help to improve the situation and I don't think we can handle that with system 5 boot system but there is a very few packages involved in that part of the system there are 10 packages in the system that's handling hardware loading you in it for that in my opinion how would you handle that with system 5 to make that possible but my focus at the moment is to improve the current system to a point where we know that there is no need to optimize anymore it's correct and fairly stable and when we got to that point I think the rest of the LVM community will be well aware of the perks of the early system and ready to switch to something better maybe it's Upstart, maybe something else I think that the devian way of introducing a new internet system would be compared to using an architecture some team of very authentic people start working for seemingly nothing converting package by package like we did converting package by package to additionally include there in the script and using some kind of interesting text to make the internet system run without changing what everyone else uses and after a while it supports 80% of internet scripts and people say ok it's almost there let's make it more official support and push the rest together to support it and in the end actually I'm hoping that devian can be effective enough to give the user the choice of internet system and the problem I see with that is that if you have half a dozen of internet systems and every internet system container, box and container in the script too it will get a bit messy because you have to update it and the container of the package probably doesn't run every internet system but it's still a problem for the hundred complex in the scripts we also have to see how many of those people are going to install those complex scripts because like I said most of them are like in the beginning and some of the complex stuff are maybe servers or read demons that may be minority runs but then again you have some very unpopular packages such as Apache such as Samba I mean the good thing to be able to pinpoint one of them is that it's easier to get the container or the team to make some changes but they're not minority packages they don't have to be complex but some of these very popular packages have relatively complex well I don't think it's the property of the system that it has to have complex in the script I think it's more like how they structure it they can have a script to do magic and run the program and then the manuscript is like two lines run the magic and run the program well people I have to cut you now I see we're in a good discussion but