 Well, as he said we are going to try to have a round table in an auditorium. So I can prepare just a few introductory Slides so that we can get up to speed with the work that was done at depth of two and I really like to end this session with a definite idea of how should we implement the interface and other stuff we will need For people to really use in its scripts that are far more powerful than just the standard ones we have So the definite goal of these beslock filters is to jump-start the design of the script systems We will need to plug dependency basis and parallel execution in its scripts Which has some people might know can reduce the boot up time to I Don't know maybe 20 20 percent of what we have That may be more so we will talk about obstruction layers The hesitory which doesn't exist yet and also I will try to talk a bit about the proposed interface For dynamic dependency need to script systems And this is all work that was done at that comf 2 and survival as a paper at my debian page at people dot debian dot org to Hmh so For this Talk to be successful. We should define our short term term medium terms in long-term goals as in do we really need Such a complicated need script systems. How should we start? What should we strive for at? For just the beginning and then what would be the next step? Should you have a need scripts registry that would help us right now with what we have? I mean a lot lots of packages start Services at really weird Positions sometimes things break up because a maintainer that has no idea who starts where Unless he actually searched for that information and if he does not have all packages staled Please Maybe you could say a quick word about what an in its group registry is I think a registry could be a lot of things when it comes in okay Just a simple table which tells you package who wants to start a sexy bar at level Bash so if you look at the file I see package pretty much Like that. Okay. Thank you Should we support parallel execution one well Probably the answer to the fist first question is yes, we should a lot of people want that But when should we do that from beginning should we try to prove a dependency-based system? We about parallel execution first then try for parallel execution This is something that I would really like to have a Common opinion about I mean parallel execution means you have lots of output that you have to you have to someone Somehow organize or otherwise things get really screwed up on screen Dependences should we support them? when at the beginning at the second stage and Of course LSB should be actually petition to what they say or just ignore them Should we have a logging layer? My personal opinion is that yes, definitely, please Let's have logging and try to design something that so that we can Jump-start it and have less flamers on mailing lists before you can start actually deploying stuff So let's do a quick review on abstraction layers Well for the packages to plug into the system without causing a lot of sorts of trouble and actually allowing stuff like File RC or a future dependency based in its script system to work We have to somehow Have an abstraction layer with which we use on the packaging system scripts To interact with the subsystem We are already have sort of Jury rigged wall abstraction layer the factory works. It's very limited There's data RCD Which in fact resistors and need script that uses to mean just creates the those sim links for CSV In its scripts, but we file our seat means I Added online a table and maybe some other to each script would need something else That is invoke our CD Which runs a new script? Remember we are talking about the package scripts. We are not talking about the local administrator The local administrator supposed to just run the script directly However, your package should not I mean, maybe he doesn't want the script to be the service to be started when he upgrade the package or something like that We had that blockage that breakage for a long long time to invoke RC dot D to fix it and Invoke our CD will probably become a must requirement for edge For any amuse if necessary to fix the remaining packages that not don't use it and that's another one that I think all the gdbc guys are to pay attention to which is telling it so that they can start in it when they upgrade gdbc There's not much to do there as long as we agree to document that Only a certain subset of telling it options are to be used everybody who writes a new need script systems Just have to provide a telling it that we work or do whatever is closer to what should be done Then there are dependencies Start order at dependencies are what we have now a List of numbers which tell you that at number one we start this that and that at number two we start this that and that Full static dependencies, which is what you have in LSB That would be oh, I am need script for I need bar and full bus to run So please run them before and then you already need script system will read an entire list of need scripts Build the graph of dependencies and then start running things after it starts you have no way to change the dependency tree and There's the dynamic dependency Where it's not known beforehand who depends on what? But when you start need script if need something it tells you and if that something is not available yet You just wait block it and when that something it becomes available you unblock it If that thing fails to start you tell it that sorry you are on trouble do something about it so here we have Start this file RC and CSV, which we do know Through this run levels In all information about the order is out of band. I mean on ceilings or the ETC a file RC file the static where we have those dependency informations usually in comments Quite ugly, but it works. It's easy That will be in band Or it could have update our CD or write a file somewhere and have that out of band Doesn't matter much as long as we do not screw up and overwrite the local administration. Yes, please Yeah, I was curious if we're the first distribution to do this I'm sorry. Are we the first distribution to look at something like this? Yes No Can too Sorry, I didn't understood. I thought you were asking if someone else had that right done that. Yes gentle desk I don't know exactly what they do But I think they have a static If they are using something like our unit it's a dynamic Open BSD has the static one already for a long time And they they have an interesting system and I think that we might want to look closely at their stuff Implementing ours. What they do is that every single in its script has two additional arguments like start stop restart They have Requires and provides. Yes, and so you call like EDC entity Apache to requires and it outputs something that we could call virtual names Like it requires network. It requires DNS service it requires something like that and then what happens is that when I start Apache to it checks whether all the required Features are provided and if they are not yet provided it searches the rest of the in its scripts Which one provides them starts those in line? So it's kind of a it's it's kind of a dynamic thing. Yes, it's a Half way there dynamic system. Yeah So That's right down problem with that approach that You have some Some mutations that are typical of the static ones and you don't have the high-gazide advantage of a static dependency Which is that you know it fully beforehand? besides It's probably easier to understand if a single unit script subsystems script like RC does all the parsing find out who needs to be run when and does that The way gen 2 does it apparently Everybody's group decides if they have everything they need and if they don't they started the beautiful thing about this approach of game 2 is that if you have You are the local administrator and you ask for some service that needs another service to rework So we really should think about that idea Oh, what's happening here parenting some people don't think we should stay so much time Same slide Well, we have already covered the idea behind full dynamic interfaces that When you run your need script you call an exact as a table that's called need or something like that to say I need something running or you call something called provides to say I am providing this as if I Exit with okay status and You could also depending on how much headache you want have something like before please start me before something else which Easy to do they get into a but completely difficult to do the are you need to be so we would have to think about it And there are run levels Which actually are naming sets. Yes. I'm sorry. I can't hear you very well With the the need Function call would you be specifying the net script that you needed to have run first or could you have like virtual? Names yes, you could they are called facilities, right? And would you then call a provider in a different Imagine for more or less like that every need scripts provides a facility named after itself And it can provide any facility once using the provide Oh Sorry quite a nice quite a nice way to wake up everybody Well, so you can really you see we hear me Yeah, okay. Well last track of the fuck. Yeah, what I was thinking was if you had the need and provide pair you could conditionally have the script goes to say Normally I'd want to do this but that bit isn't working. So I'll provide less Normally I'd need this but that's not available. So I'll need something less You could do it that way. I mean it's kind complicated for the user to understand But yes, it would support that using something like a test. Please tell me if said facility is available or not Instead of just give me that facility if you can This one we should have a long time now, but nobody's really interested in implementing it It's the idea that you can he start something Okay the run levels run levels are an image sets of Service that you have started or stopped and you can also Think about a third set which says that Which ever isn't set. I don't know anything about maybe they're running. Maybe they are not Since we have that idea of incremental run levels run level one then run level two which add some services Then we're level three which add yet another service and so they've been uses only to run levels single user and multi user We also have the Let's call them. I don't know system wrong levels of startup and shut down we actually have reboot as well, but For Debian, I don't know any of anything that runs on shut down reboot other than the Few scripts that actually do the shutdown or reboot and who cares about that others are just act the same if you are doing a dynamic dependency system, you are really better off if you only have naming sets and So that you can roll back to a previous run level or something like that really should not have shut down and reboot Separate it gives it would give you a lot of headache Maybe you can actually do that, but it's an additional Complication that we really don't need and it's about time someone fix that. I mean How many how how much is of how many of you know exactly how that How the shutdowns and the reboot our run levels work? Yeah, is that obvious? No, it first runs all Start scripts to do the shutdown then it does runs this top script. It's It's really a key Then there's try to start Which has the very obvious idea of please restart something that's running, but do not start it if it's not running It's a fool not to trample over the local administrator. I mean, he just stopped the service then he changed the wrong level Should that thing Started or not? Usually we can Read his mind. I mean if should have been running and it's not we are to suppose that he stopped it So we don't start it again At the need script level that just means you have to actually implement something that tells you Whether some service is running or not That alone should fix a lot of problems We have in its scripts that try to go the easy way and Do a very poor maintenance job of taking care of this Things like calling starts top-demon with that create feed file, please function, which doesn't really work something that dies leaves Peel file behind and It just doesn't we have to fix the demons so that they create and take care of the the P files themselves And that's the fresh. Oh, yes, it's on I Was thinking of a related thing to the pit file stuff when using a system like Run it or Damon tools or M in it We try to keep services running by restarting them if they die. Yes, and I would like for standard interface something like start stop them and that to be Included because right now when I run an in its script, I have no reliable way of knowing which process was the related one Yeah, that's something you lose track of when people start doing the demons the demon thing properly I mean they set a new session idea and you lose to lose track of everything Problems that most most of debon's work to think this is V a mode So they start demons Probably the easiest way to fix that would be to have two in its script One that's supposed to work in the background Which is what we have now and one which is supposed to run the foreground which is what you need to do the Are you in it way? I mean you keep you fork a child and that child should stay there Running on foreground and you policy to if dies you start it Could be But is it really worth the thought or should we have another action that says running the forego? Or maybe an environment viable That's a question. I would like to have asked answered here. So please web and has a very good idea about that You can even do a quick vote Do you think it would be a? Good enough idea to be worth the fart to implement foreground running services Please vote yes with your hand up. Please vote no if you're with your hand up So we have more loading yes, that means a flamer later. It could also use another approach we could Define that the in each script itself should policy whatever where whatever it started the service it started and keep it running That would be a separate suppose It's not exactly the CC way, but stuff like my skill does that I read I think Yes, okay. How about if we had the in its script? Register with something like the need and provide scripts the kids which it started and then Some part of the in its system itself can go back and run stop and start when it dies Well, the problem you have there is tracking who dies where Demons fork themselves and some of them talk themselves on numerous times You really have to be traced stuff to know where it ends up or you have to scan for the exact table name later The dash proc table anything that writes a ped file should have given you all the information you need because start stop demon won't work without it Not very much works. Well, it starts up. You know, it's an old idea that we still have which is as a fool, but The real idea behind the modularization of the CSV in its script systems that you can have anything as a service And that there's a script that you use to interact with that service We are probably better off if you don't break that assumption So if we need something like you described Don't automatically by the subsystem. We have to create the proper interface and use them We should not try to second-guess what the demons and scripts are doing That relies on madness. I saw We agree we come back to that that's a bit later Let's talk about the rest of them. I've already said that the rest is basically a web page or something a tail or whatever That tells you that a certain package uses in its scripts and that it would like it to Blah blah blah blah blah, which the maintaining rights so that you have a basic idea of what it needs And that the package currently starts the test run at order Fool it stops at other bar so that you can at least try to understand what people are doing Maybe we want to have so many Scripts starting at the s20 that way when they should be at s21 or s19 or something like that The problem is that that would not be automatically updated, of course, maybe LinkedIn could check it and complain if it's outdated or something like that, but It's more a tool for developers than anything else. It's not something that they need script subsystem would be using itself So that you can have an idea where should you be starting your service when you package them There is another use for it Should we have dependencies we have facilities and then we have virtual facilities which are Sort of like virtual packages and we have to write that down somewhere which facility does what? The hazard could be used for that and to the end up as Document for policy or something like that as we have now for virtual packages Then back to that question. Do you have your answer now? No let's go forward and Here we start with the highly things If we even attempt to try to run things in parallel without doing some sort of input output channel control We have a complete mess on screen. I Don't know how others are doing that, but I think they are doing the complete mess on screen way at least open best BSD does so If you try to do that right to have to somehow capture standing stand out and standard which is easy Just have them directed to pipes and manage that somehow It's not that easy, but it's not really problem problematic And of course we should not try to mess around with you need too much I mean, it's the single most important program the system not besides kind of breaks. You are really in trouble So that would be the design requirements If you have any extra requirements, I'm all ears And there are always the value-added features We feel has if we implement something like a control channel We can actually do that Green okay, and red fail thingy people seems to like so much And while at it we could do that graphically we could have a penguin dancing things work and a penguin crying if it doesn't whatever However It really gets difficult to parse they need script output I mean we have policy what people should be using and they don't use that It just won't work. We have to somehow Send control messages So that you really know that Needs scripts trying to start something and that it managed to start that or it's trying to stop and managed to stop or didn't Manage to stop or that's doing an extra action such as please why wait while I set the parameters on Schleswig device full or something like that And while we are cheap We might as well add extra output levels as important normal in the bugging and shut off the bugging so that We have less crap on screen during boot up and if you need it We have that that's been logged somewhere And I would actually be quite happy if you implement if we implemented that alone first in the first stage And then went to Dependency-based it's tough on second stage. I mean that thing is immediately useful and But log this just doesn't cut it There's that it doesn't work. I always so there are two main possibilities for a control channel In bed So they need scripts would print something like that as For I'm starting our ST for I'm stopping something like that a simple tag-based protocol that you have to add before every function Or we could have it out of bed Where you have a pipe to send those control signals But then we all know that if you send two messages down to pipes, you don't know which we will arrive first So you have synchronization problems? It's probably much faster to do it in band Wouldn't you would anybody be actually opposed to in-band control? Suppose we have set an environment variable to know when the need scripts should add in-band control and At that only when that's In-environment variable is set so that when our administrator runs the need script directly It has no tags to make output confusing Sounds like a weird deal. What do you say? Nobody yes Out of bed anyone would propose we do it out of bed Why? Yeah, I mean it's far more complicated. So you must have a good reason Is that is that yeah, I can't I can't really give you a why at this moment, but I've looked at a lot of the in-band stuff right at LSB gen 2 and I found that in certain points if you really want to go away from the semi static Setup that gen 2 for instance uses which I actually think is all right, but I could see that there are Several issues with it then what you may want is some sort of scheduler that actually worries about the The order in which the init scripts are started I'm not talking about it This control can is only about the input and output control of a single need script We this is a control can only for the input and output to screen interaction with the user We would have another control can of some sort to control the dependencies and that probably will be out of bed All right, and I am out of band 2 on this one. Okay, so in band And of course how to implement this interface Well, that's a that's a easy enough thing to discuss on mailing list but we could do something like a positional library and Document the the protocol and make it static enough that people can write their needs seeps in Perl or whatever Binary, I don't care as long as they talk the proper portable it works And if you are doing it like almost everyone else and using shell scripts you have a easy to use functions Which could we could actually make them LSB compatible for no extra gain But so that people are looking think we are nice to the house be and the dynamic dependency systems A dynamic dependency system is the most flexible solution The only thing it doesn't allow you to do well. It's a to say something like that. I want to start before someone else It's much easier to say that someone else tell please start me after Whoever if you haven't started tree, you know who will start when so you can do whatever you want the question With the dependencies, I'm working. I'm wondering how you would implement it for something like ISDN startup Which needs to be run before networking, but you can't really say networking depends on ISDN So how would that work? SQL loops will do have to break using human intelligence. There's no way around that I think so you it's a Well, the subsystem has to break loops But you can't really tell it how it should break loops. So you just break whatever whoever's Starts later and ask for something that's already waiting on itself Then you kill that and say failure and go around so that the administration can fix that later. It would be a bug for yes, then we would have to find a way to maybe break it into so that Whatever networking that needs to be set up before yes, then can start starts before then yes Then then whatever net needs to go later. We'd have more granularity so that you can introduce yes, then the middle something like that There are no real easy ways around SQL loops Well, if you had a before implementation Then then it should be fairly easy. You could just say I see a need to run before Yes, but I don't know networking setup. If you only if you just need a before action That's easy. What's not easy if you need it in the middle with your that was what I thought you were we're actually asking about But if you know only need before that means we have to use a really smart dynamic system or a Two-step system where it's dynamic But you have to to do a first pass where you ask everybody who needs a before To know that beforehand that could maybe work well You would have static a before and dynamic afters and provides and everything else Another thing I was wondering is how do you cover for third-party in its scripts Because we're now setting up a framework which requires script to conform to something For instance, I installed vmware this week which adds an in its scripts to Well, the set of inscripts But that may not have the infrastructure No, it won't have the infrastructure The idea is that if they do that through our cb compatible interfaces We know that it's trying to do that. So we put that on a different queue and We find a proper place to run the entire queue in order as if to order the original new script systems And place that queue some somewhere into the dependence chain. Okay. Thank you And then here's what we have on a dynamic system to provide which that's facility addressing those two points the first one you might be able to do a bit like the W install the menus where The Udebs know where in the packaging where in the menu order they would normally fall And that's the sort of first try at the order that the things get run And then you enforce dependencies on the packages after that So if you want something to run before other things you give a lot of static dependency But but then then you'd do it actually in the needs requires order after you've done that on the Things that are just being dropped into RC It doesn't work to run the whole queue in one place because some things need to be very early or very late Yeah, but stuff that really needs to be very early shouldn't be a from A third party if it is a local administrator would have to fix that could we not have some to have a Good glass of where to run stuff But maybe we could even have more kills. Maybe one kill for low-numbered stuff Another for middle number of stuff and another for hiding numbered stuff that runs last Could we not have a system where a default in its script would need s20 Yes, and then you could have various points in the Dynamic system that provides s20 and s5. Yes, that's about the idea of Middle middle position and queues think the depth dependency Well, there's provides With which if we run them at the very beginning of a new script might Might fix the idea of the before What I would like to see in that system is for an administrator He might want to change dependencies for example, usually the patchy web server doesn't need remote file systems But he might if the files are no more file system so should be possible for the administrator to add or change dependencies and without having the Depthconf ask him upgrades. Do you want to override this file? Well, there should be a place for out-of-band initial dependent information see for the local you would need static dependencies for that and you Also need an out-of-band storage of the dependency information If you want dynamic dependencies forget it don't work to have to be a mix of system at best and the inbound want to work with that. I mean unless we up to go there and Somehow get it out of band and then run it in bands inside an script system which sort of defeats its purpose Can you just have a file etc in a dependencies where you're at a patchy double-point column the mode file systems and the in its script checks the dynamic dependencies of patchy but also looks net file for local changes or additions Well, then we have would have a mixed system that that would work I suppose Thank you What that provides before after and test those are quite Obvious All them but before which really requires previous knowledge of the dependency or you have something really that Always fails if for if by any chance you hit a before that can't be Be acknowledged. I suppose How important is a before? That's a question we question we should answer is it that much important There's a similar arrangement it with deep package control information We have provides and enhances and at the moment we've been doing very very well with just provides so Enhances the idea there is that you can actually like extend large packages in terms of functionality Without actually having to change these large packages. So it's modularity I don't think that actually if we have a proper policy if we say that the If we have a topo topology for the available Facilities like we say there's a network facility. There's a time server facility a web server facility and so on and we won't need it if we actually need No, I actually don't I actually don't think we need it because The things that you will depend on are things like network and web server stuff and they would have to say I am the web server I need to be started before this version control system XYZ which you're going to run over web interface and it's just not gonna Yeah, and for stuff like he asked you and we could have Another pre not working set up facility that it can depends on and it's run by network network depends on pre network or something like that That would give us easy before without the trouble of actually walking downwards on the dependency Well, I have only five minutes left, so I just want I just want to add for that problem. We might have a After if they're so Network will be started after ISDN if there is ISDN, but we're not saying if there's no ISDN at all Well, we couldn't do that to using test. I think this test for well, okay It's dynamic. I mean you run it and depending on what it returns you can call a provide or an after Okay. Oh, yes, how to track in scripts ID. I have figured that by myself After a world these lights and we run it you set of environment variable with the ID Then run any scripts Everything that needs to send you data back Text that I deferred and then you know who's trying to tell you what over the control channels. I Don't think there's a much better way than this one anyone has any ideas isn't It's working here Is it before basically just the same as if the Second if the other script actually depended on the first one, so it's basically Kind of like reverse dependency. Yes. So you could handle it like that. It's nice But it's very hard to implement. So the question was can we do you about it? Can we do it without it? We've no no such functionality We had we have been doing that with package dependence for a long time as he said You can emulate it and you really want it. Well, that's all for the slides but Let's hear the ideas and jump starts One problem is things like Network disk mounts some things need your disks mounted first and And what the disks mounted to do stuff to start the network and other things what the network started so you can mount disks? so You know basically it's a circular dependency problem that More or less needs to be broken manually, but how it how it needs to be broken may Depend on your local system. Yes So we we are back to the local administration must be able to change the dependency locally somehow and that we may need to have a facility broken up into Stage one stage two stage three or something like that probably before During and after so that we keep things simple Yep Yeah, first of all, I would say that I would I really like your Effort to improve the in its system in Debian. It's been in need of Complete rewrite and improvement for a couple years now I hope you can provide the statistics on how you can speed up the process I did already does that if you have a look at their pages They have a statistics of what how much you can get out of parallel execution. Well, I've seen it But well again to seem to produce interesting graphs for a lot of things I'm not sure we can trust them Well, you can always have it optional and not enable it by default if it won't give you much of a boost That's that's not my main point. My main point is that we need to make sure the system is highly dynamic and configurable because like You can have a server which is using is it's the LDAP server and you're using LDAP to log in Yes, so we need to start LDAP server before Any of the other services that will need users and then you might have a different server which is using the LDAP The other LDAP server To provide user information, so it doesn't need to start the LDAP server. Yes, which is like the enterprise Something else LDAP server. So you have two different settings with the same packages installed and they are depending on each other Depending on the content of the LDAP server So you need to have two different dependencies for these two servers. Yes, and for Raids and LVM for example if you have the root system on raid LVM, you need to start root on LVM first Yes, and if you don't probably on later. Yes, but you'll probably do that on any tier D Anyway, or the kernel will do that for you on the rate. Well, maybe probably or maybe you are right The question about the LDAP server school actually makes a lot of sense. Yeah, and we've had that problem with With school in XR we are using LDAP and yes Automated that I mean local local overrides for administrators are fine But as as soon as we start implementing local overrides for packages to override other packages Would be better off in using before something like that really Maybe I don't know we'll have to find out so we are back to do we really need before maybe yes for that kind of thing You are suggesting Probably I Don't think there's another way We can actually implement before using a dynamic tree, but it all ends up to Somehow knowing beforehand if something we need another beforehand So you end up with two steps and it may fail depending on what you are you have instilled If something decides to provide and something else dynamically you can't know that because a conditional It's a conditional thing So if someone does a before dependency on that It will save a sorry. You cannot do that. We don't know about it Nothing is going to provide that so you either you started right now Or if you fail That's all that you work I suppose So we have before useful for Could I say highly unlikely or at least uncommon setups or not are those setups so common that we really should have defaults should have a Something tied to the packaging system that allows us to do that such just sort of we really need the services before the others. I mean in the case of Skollinux you could just add Some changes to the dependence trees that's the idea behind behind a custom statement distributions pose Yeah, I prefer it's a actually It tends up as if we can do that we should try if we cannot You have to do without I Think we have to take that the main is because my time is over So running out of time Interesting session, I hope you can continue after that a little bit more private Which means not in this room We are starting a next session will be in five minutes All the time I hear for Debian by Andreas Barton Martin sub-hellers in this room when they wake up soon Or in the small auditorium the Debian free software guidelines by Matthew Garrett Thank you Thank you very much for your participating in this bit of feeder section