 Effectivement, bonsoir tout le monde, c'est une vraie plaisir d'être vers Vissure à la meeting d'utilisateurs d'EasyBuild pour vous parler des modules, qui sont aussi appelés « Environnement modules » et même « T-Mod » par les compétiteurs. Pour vous qui ne me connaissent pas, je m'appelle Xavier de la Ruelle. J'ai pris la leadership du « Environnement modules » projet il y a deux ans. J'ai travaillé au CEA, qui est un institut de recherche en France, et plus spécifiquement dans cet institut, j'ai travaillé dans la division de l'HPC. Alors, j'aimerais faire un brief recap de l'année dernière pour les modules. Il y avait beaucoup de modules, la première, version 2, version 4.1, qui s'est aimée d'improver le test non-régressionnel de la solution. Maintenant, j'ai hâte de dire qu'on est à 99% de codes couvertes par le test, qui représente environ 8000 tests. Et durant cette nouvelle branche, c'était aussi l'opportunité d'improver la support de la solution sur plusieurs OSEs, comme Mac OSX, FreeBSD, Solaris et même Windows. Vous ne le savez pas, mais c'est l'épreuve. L'année dernière, pendant la seconde partie de l'année, version 4.2 a été élevée, et cette nouvelle version s'est focussée sur un lien meilleur de la dépendance entre les modules. Alors, je vais entrer dans ce topic. J'ai dividé cela en deux parts, première en parlant de la consistance de la dépendance et ensuite en parlant de l'automation. Alors, pour parler de la consistance de la dépendance, c'est-à-dire de la dépendance des modules. Il y a deux types de dépendance, en fait. La première, c'est des requérments. Donc, un module veut une autre module pour être élevée, et la deuxième, c'est des conflits. Ce conflit est un deuxième type de dépendance, où un module peut dire que je ne peux pas être élevée si une autre module est déjà élevée. Pour exprimer un requérment, il y a deux ways. En utilisant le commande module-file prérec, qui est juste le requérment. Mais vous pouvez aussi utiliser le commande module-load, qui est le requérment plus essayer de loader ces requérments automatiquement. Et c'est de la même manière pour les conflits. Vous pouvez juste exprimer un commentaire de conflits avec le commande module-file de conflits. Et vous pouvez utiliser le commande module-load, qui est le requérment plus essayer de loader ces conflits si c'est élevée. Donc, toutes ces dépendances, elles sont élevées dans une version prévueuse. C'est pas mal, en fait. Parce que, quand vous éleviez un file module, toutes ces dépendances se déclare, elles sont complètement élevées. Le commande module-load n'est pas réunis sur ce qu'était le requérment ou les conflits que vous avez élevés. Donc, ici dans ce petit exemple, on voit qu'on a un app X. On veut le loader, mais il faut le loader avec le requérment A. Donc, on load le load A avec un app X. Mais si, après, on ne load le load A, il n'y a rien à dire, mais on l'a élevée avec un environnement inconsistent. Donc, c'était le cas dans la version prévueuse de modules 4.2, mais c'était aussi le cas dans toute l'existe module d'implémentation. Donc, ce qui a été élevé dans la version 4.2, c'est de traiter ces requérments en utilisant des variables environnementales, c'est-à-dire les modules n'ont pas été checkés quand vous essayez de loader ou de loader sur les modules. Donc, ici, dans ce petit exemple prévueuse, on peut voir ces variables de traquage. C'est tout le nom de l'app et les requérments que l'app déclaire. Donc, maintenant, quand vous essayez de loader le load A, vous avez un erreur, parce que vous ne pouvez pas faire ça, vous devez retirer l'app X avant. Et la même chose pour les conflits. Ils n'ont pas de traquage avec une variable environnementale. Et cette information est checkée quand vous essayez de loader les nouveaux modules. Donc, ici, on dit que le load A conflite avec une autre version de load A, mais le load B. Donc, si après, on essaye de loader le load B, on a eu l'erreur, parce que le load A est loaded. Et on a eu l'erreur, parce que même si le load B n'est pas un conflit contre le load A. Donc, bien sûr, avec ces nouvelles features de consistance, il y a une feature pour bypasser cette consistance pour la convenience des gens. Donc, le dash-dash-force command line switch a été introduit pour bypasser toutes ces requérments ou conflites. Mais vous endez dans un environnement inconsistent. Donc, les modules d'après-midi, après-midi, ils expliquent des règles de dépendance qui ne sont pas satisfaites. Donc, ici, si on essaie de loader le load A, il y a un load B, on peut forcer le loader. Donc, il n'y a pas d'erreur d'en forcer le loader. Vous avez juste un warning et vous endez avec ce que vous voulez. Mais c'est inconsistent. La bonne chose pour mentionner c'est que si vous expliquez les requérments avec Prerec vous destructez de la même façon. Et la même chose pour les conflites. Moving to the other topic regarding dependency management, automation. Automated module handling mode a been added in 4.2. To automatically manage to automatically handle this consistency for you. For users not to bother with the dependency, they may not be aware of. So this is a set of mechanisms that are automatically triggered when you try to load or unload module files. So in the next example this automated module handling mode will be automatically enabled to see what happens when you have this set. But first I would like to state that the order you load modules, it matters because you should have a requirement loaded prior a module which depends on it. Because this dependent module it can adapt its evaluation compared to the requirements loaded for it. And here a small example where we have 2 module files loaded, a toolchain module file and a software module file. And if we look at the location of the binary of this software it has adapted the installation path based on the toolchain. So if loading order matters unloading an order matters too because to get a consistent unloaded you should unload it you should unload a dependent module prior its requirement. Now let's dive in the automatic mechanisms. First when you try to load a module we have a first mechanism that at first step will unload dependent module and at last steps will reload this dependent the second mechanism will load automatically the requirement and then at first step you load the actually asked module by the user. So if we dive into the dependent reload mechanism the idea is to check what are the loaded modules that depends on the new module you try to load to first unload them and in the end load them again because they can benefit from the load of their new requirement which is now there. So if we take back our example where we first unload of the apply without its libby requirement if afterward we load libby it will automatically reload appy because no appy may benefit may do a proper evaluation based on the requirement that are now there. So by doing this we end with libby the requirement in first position and in second position we have appy so that's dependent reload. Requirement load this is what you get already when you put module load order within your module files. So this is a step where you load all the requirements for your module prior this module load. So here if we try to load appy without nothing loaded we automatically load libby. If we focus now unload of module files there are other mechanisms automatically triggered again dependent reload which acts at first stage and last stage then the second step are dependent unload you unload the module that are dependent of the module file you actually try to unload which is the first step unloading this asked module file and the fourth step is to remove the useless requirement. So dependent unload quite easy to figure it's to remove when you try to unload a module remove all the module files which depends on it and of course the module files which depends on with recursive effect. So here still with our libby appy example if we unload libby automatically it will remove appy. Useless requirement unload when you automatically load requirements there are no tract with an environment variable which helps to know what the users actually asked for. So here in our previous example we have libby and appy libby was automatically loaded so it is tracked there in this lm not user asked variable so then when you unload appy automatically it will remove libby because libby is not required by anything else and it has been automatically loaded so it is not required anymore. This is if you know the depends on mechanism of lmode it is quite similar and dependent reload again a concept triggered at load but also unload time and one additional things that is done at unload time is that it track conflicts so if you have a conflicting environment if you try to unload a module file with conflict with others the others module file will be part of the dependent reload mechanism because no there is no conflict anymore it can benefit from being reloaded to perform a correct load evaluation so here we have an inconsistent environment first we load libby version 1.4 but then we wanted to load appy but appy it requires libby 1.10 so we have 2 versions of libby which is inconsistent so when you try to unload libby 1.4 you solve the conflicting situation so to restore something sane libby is reloaded but as appy depends on libby it is also part of the mechanisms so here are simple examples but you imagine all of this can mix you can have a bigger situation that are quite well unloaded currently all these systems there is a 1000 non regression tests covering it so it is well well tested this is also a redo of a proof of concept we add in production at CA since 5 years so this is something we manipulate since a long time but there are stuff that could be improved there is a missing mechanism automatic mechanism currently the first one is an automatic way to unload conflict that I call conflict unload to automatically under some situations and a second mechanism is to automatically reload loaded module when you ask for loading a module which is already loaded on so two mechanisms to what will be added later on it will be also good to evaluate all the dependencies prior to actually evaluate module file in a load mode because currently you load a module file you see there is a requirement so you try to load it and so on but in the end if you fall on a conflict or if the ending module file broke you are lost so you need to try back and to try another solution so it can be better un lid if you first do a pass to only evaluate module load, module unload, conflict and prior statements in the module files which are of interest what you need to do that will correctly perform so this is a specific evaluation mode of module files that will be introduced and the topping of the cake is to use for all of that satisfiability algorithm SAT which is the way to go to do all these things very performant so this is big change and I would like to state module 4 will I will try to make it last longer first because as it is introduced in rail 8 it will be interesting maybe to get some updates of this version 4 in rail 8 but I don't know I don't work at Red Hat unfortunately so each new features that change existing behaviors they are disabled by default but you can enable them at configure time with an environment variable or on a case by case basis with some command line switches and even in the next feature release 4.3 there will be a new config sub command which will help you to see what is currently configured but also which will help you to configure the properties you want to enable so you can also put that in your default init script for modules at your site so what can we do with all of this so I think we can introduce a new way to write module files and to organize them so with all this mechanism a new naming scheme could be introduced where one module file match one software version but not one build match one module file so just have one module file per software version and not n module file per software version for each tool chain you have on your system you can do so by describing the dependencies of the module file the tool chain which enables this software version within this second module file so for instance here we say we have a prec on two tool chain where this software is available and then later in the module file script you can check what is the tool chain that has been loaded and you can adapt accurately the installation path so currently with a hierarchy setup you have of course one module file per tool chain and then you have one software version module file per tool chain in which this software is available so if you have free tool chain this software is available in the free tool chain you have free module files and in a dependency setup of course you still have one module file per tool chain one module file per software version so it it can reduce a lot the number of module files you have over advantages that you are free to organize the module files the way you want it also opens the path for multi hierarchy and a nice thing for the users that they are able to see everything from the start so here a quick example we have a tool chain hierarchy and a build variant hierarchy so we have loaded fv for first 18b and the optimized build but now if we want to switch the build variant to use the debugging version of the app we see the switch automatically reload the dependent app module file and in the end the location of the bin has been updated accordingly you have the first the tool chain here and then as a suffix of the version the build variant so ok but there are things to adapt of course especially on tools that generates module files it shift quite the way it works currently because from my understanding today one build correspond to one module file and in this proposed scheme one build update may be an existing module file so you need to know for what other tool chain combination the module file has already been built for so you need to update existing stuff or remove stuff so I don't know the details how this is done in easy build currently but from far perspective there may be things to adapt and also to not disappoint too much users following this philosophy there are other things to add on the module side for a first point to add it's an option can I just ask you a question yeah yeah of course I mean the one thing I see I can see how you could make it work for a person on that module file which is not necessarily the case so say in one way we manage things multiple people can install software in a certain direction which means they might not have permission to actually change that core module file like logistically I can see how we could manage it in one of our naming schemes but then there's this permission restriction I mean I guess you just have to logic around that you won't have permission to actually install this good to know there need to be a group permission or all the people who do the job in the same unix group stuff like that good to know thanks it's also dangerous to do on a live system if someone happens to be loading the module at the moment that you're writing it if you install a new module when you install new software then you're sure nobody's using that software at that moment because it's new but if you change something on a running on a live system there is always a very small risk or even I wonder what would happen if a program loads it twice once after another I kind of expect the same behavior but could get different behavior you also have the effect when you spread your module files per toolchain like in a software hierarchy scheme if you introduce a new module file it may be loaded yes but if it's a new module file with a different name no body is using it so you don't harm anybody but in the module perspective this is the same name you already have that problem when people load modules without specifying a version if you install a new version and they do module load python again yes so do we but nobody listens so good thing to know in this system but when you load a module file as you register in the current environment the dependencies even if afterward you change the module file you have a footprint a snapshot of the dependencies so this is what will be done when you try to unload it so it's a bit it's quite consistent in this way so in this case the change that would happen is you'd be adding a new toolchain so the only reason to rewrite that file is to rewrite the software is because you've added a toolchain do you know what I mean so that's the only change that would be happening so unless they actually loaded the new toolchain it wouldn't have any impact exactly yeah so what's the so if we have like suffix versions right so you have gromax for instance gromax with python gromax with python would be then another module right wouldn't be the same module with a dependency or anything like that could be the same in fact it's another kind of hierarchy so you have here a first hierarchy describing toolchain but then you can have second hierarchy describing flavors that are consistent between each other but then I would have to the one that's non python I would have to actually say non python I would have to like pre-wreck non python something like that but then I have to give a name to the non python version because the pre-wreck would be python as soon as you build it for python or non python in the software version you say ok it's the python version or the non python version yeah but you you say no you have two ways currently you just say no python no python so you have two modules that are opposites and what I would like also to do is to have optional requirement so python may be an optional requirement so by default it is not there you do not try to load it but as soon as you load this optional requirement it reload the stuff I understand yeah good solution yeah optional is not there currently but I like to take up on this issue on the previous slide you showed for instance like three modules FOSS tool chains and then one application and a user will always expect that these are compatible and of course if you don't compile it against a certain soft tool chain it just won't work if it's another tool chain perhaps in a complete intricate setup with different applications where one is supported by tool chain A but the second not and the user will expect because the tool chains are present the application is present that we actually did the compilation against the tool chain A and if it's not present you cannot load it or you need to change the tool chain yeah so there are specific things to look at in fact here it's a generic dependency system you can see it this way and then we need to see the specific cases related to software installation I will move on if you will not mind because I am not finished at all unfortunately no problem, thank you for your question so first thing to refine on the module site to do a good job with such a new philosophy is to be able on the avail subcomment to filter out which is not compatible with what you have currently loaded so for instance here we have two software related modules the same software two versions and if we load a given tool chain a user may wish to only see with some options what are available modules for compatible with this version and another aspect that should be very nice to add it's a new automatic mechanism which I could load compatible with which would bypass the current default loading mechanism currently when you try to load a module using the bare name it will load its default version whether this default has been explicitly set or implicitly set when it's implicit it's the latest version but currently you may move from one tool chain to another but do not have the same set of application built for the tool chain so to avoid some flip flop between the tool chain because users I think they prefer to build step by step their loaded modules so they do not want to flush everything automatically a good thing is to once you load one tool chain to then you just use the bare the bare name of the software load the version compatible with what you have already loaded so here in this example we see we have two version of app and 4.1 is the default so with the current mechanism it will not work because if you try to load app y it will load 4.1 so it will flush out force 18B so it may not be what the user want so something here to also think and implement I think so it's nice to do some new design but we need to face the current situation and currently the trend is software hierarchy we have seen that I think all the site presentation 2 days ago so software hierarchy it's a concept that has been introduced and popularized by Elmod and you know it it's to play with path and automatically adapt the visibility of the module files, the verbal module files depending on the path the path enabled and today software hierarchy is the best known way by people but also the best supported way by tools like EasyBuild so I will go for software hierarchy support in modules because first this is what currently people are asking for this is what they are used to I think and second point it will be also a better way possible with a tool like EasyBuild with what this tool currently provide and how to do the support hierarchy this software hierarchy support in modules quite easily in fact you just need to track module use to say state that module use it's a requirement it's some kind of a requirement and then as soon as you track it the mechanism will handle that automatically like dependent unload or dependent reload mechanisms so here I just would like to show you because I know that you better know lmod currently and this is what you use so we have built up a small table to put side by side the features that are unique to lmod and the features that are unique to modules I say oui because I have shared that with Robert Robert McClays developer of lmod and he was able also to put his own perspective in this slide so to get some neutral point of view so I will not go in the detail of each features unique to each tools because I will need one hour one hour talk just to state that of course lmod has more features currently but they start to be quite nice things also in modules so quick roadmap to finish version 4.3 sorry sorry to interrupt again there was just a comment on the chat they said that lmod has forgotten a feature here which is hooks that you can hook into it you can influence it or change behavior that's possible too and we also have this one so that's why there are a lot of stuff for instance collection we have also features that are common we are not in the slide just to show you what are the differences so if it is not there for lmod it means that it is already there for modules sorry no no no no the spider is included in software hierarchy just to sum up things so the next step version 4.3 in one or two months so on march it will be released just to focus on one point I will add a case insensitiveness for avail but also for load because users they move from one site to another and depending on site there may be different spelling of the same tool different cases for the same tool ever they use the lower naming scheme or the up lower naming scheme yes can this that will be configurable as well yes of course is that going to be separately configurable for avail and load it should be don't know it should be that's your recommendation because lmod already has support to make avail case insensitive but not for load for load Robert says it doesn't make sense so if you you can allow it for both but do it separately that way you can make it compatible with lmod or not it's up to you if you only have one configuration like case insensitive and it's for everything then you're like you want something in between sometimes ok good to know searching is easier if it's insensitive of course but where you were at the bird of fever during supercomputing it was mentioned it would be a nice thing to have case insensitiveness also for load in the case you have a tier zero project for instance at Ulish you use uppercases software and then we move to CA or Switzerland for the year after and then we use lower cases only so you need to redo your batch scripts for such stuff like but ok 2 options no problem right now I don't think lmod has a way to configure it to make it case insensitive for load I could be wrong because Robert says no it's a bad idea yeah I remember that but we are use case 4.4 a quarter letter 4.3 and of course after 4.4 there are many funny stuff to do caching expiring modules files also I'm also thinking of supporting modules files written in python as of today this is the scripting language people know and it will be nice also to play with stash features like git to throw everything into a collection work on something else and then pop it again some ideas thank you very much any questions for Xavier we already had several questions so I guess we already got them perfect thank you thanks