 So, welcome everybody to this tutorial about EasyBuild usage of on-cray systems, which is meant as a part of the EasyBuild tutorial for the Lumia user support team. I would like to start before going to the actual tutorial, as I said, with a quick overview of EasyBuild at CSES, which will show you some custom builds that we have and some pipelines, and it will be a short introduction to the tutorial, giving you the context of the use of EasyBuild at CSES as of now, and it will make also clear some of the information that I provided in the tutorial. Now, after the quick introduction, we will go to the tutorial, the EasyBuild on-cray systems that you see on the EasyBuild tutorial page online. And then I prepared just a slide with a starting point for discussion with question and answers based on the Confluence page that you have for the Lumia user support team on Confluence. So I don't have access directly to the page, but could kindly send the content. So I adapted that in a slide with the most relevant questions, and then more questions might come. Most of the questions will be answered during the tutorial, but still it's good to go through them at the end, just in case you have more doubts. And one more thing, now I will show just a few slides, but later I'll switch to a split view of the screen so I can have a shell ready. And I also advise you to do the same, maybe, to have a shell open on the system. You are users of the system, so you might try immediately things as we go through the tutorial, and if you have troubles, just raise your hand, just raise a question. Okay, so let's go down. So this is the quick introduction that I wanted to give you, and that's something that I've already presented more or less at the EasyBuild user meeting in January, last January. It's updated to the recent improvements of the system, but still more or less it's sink on. Now, this is the timeline of EasyBuild CSCS, just to give you an idea that we started already, now more than six years ago with EasyBuild when Guillermo and Kenneth and Peter started developing the trade tool chains on the system that we had at the time that is still the current one. It's called PITSdient, initially the support was experimental, but then it moved very quickly thanks to the support of Kenneth and Peter for creating the tool chains, and we could also deploy the EasyBuild trade tool chains on a Meteo3 CSStorm cluster that was in service until last year. And there was also six years ago an EasyBuild hackathon organized to provide trade support, and that was, let's say, a kickstart for the use, a widespread use of EasyBuild on trade systems. Then over the years we have been improving the use of EasyBuild and integrating that with Jenkins pipelines and our GitHub production repository in order to make the installation of EasyBuild recipes automatic after new pull requests and new items are pushed onto our repository. And then we arrive through many upgrades and installation of new systems up to this year, 2021, where we have the Cray EX system available at CSES. So it's called Alps and the partition for production name is Iger. You are all users of the system, so you might have already had a look at the Alps user guide that we have on the conference page restricted to CSES users. You know already a bit about the system. Now just to give you then another view about the other system that I mentioned and to let you understand that EasyBuild can run smoothly, can build smoothly on different systems that we have. So Bitstein is our flagship system still. It is a Cray XC50 hybrid with XC40, featuring on the XC50 P100 GPUs with Haswell CPUs and then on the XC40 partition, Broadwell Intel processors. The Alps system that I already mentioned was early user access until recently. It will be it's more or less general access since mid-May and it will be let's say even replacing for some users at CSES their quote on Bitstein as of July 1st. So for us we consider that production and full general availability. It features AMD ROM CPU processors on the nodes and it's CPU only for the moment. Then last but not least the new Mateo 3 systems which are called Arola and Sa. And they feature Intel Skylake processors with GPUs with a VRV100 GPUs 8 per node. And I wanted to show you the Mateo 3 production system just to show you different approach that we have on the different system because on the three different systems the EasyBuild installation provides a slightly different environment. On the Intel Skylake for instance here we have the EBSOFT2Stack available since January 2020 last year. And we use a custom module naming scheme. It's lowercase as requested by the customer Mateo 3s with few exceptions. So the exception are mentioned here. So the EasyBuild custom module that is the CSES EasyBuild module providing setting the environment. And then two metamodules Program Glue and Program PGI which are based on what Cray provides in general on the system. So we kept the case of these module names. These metamodules provide a kind of hierarchical environment. Not really a full one because once you load one of the two metamodules, either the GNU or the PGI ones, you have a flat module naming scheme. But then before loading that you just see some generic modules that are not built with the specified compiler. So that was meant to organize a bit better the modules seen by the users. Although not in a fully hierarchical module naming scheme. Alps instead is, as I mentioned, the Cray EX supercomputing system providing for each node two sockets with two AMD EPIC processors, AMD ROM in Dragon. Then the current EasyBuild on the system features custom tool chains. For the CPE, the Cray Programming Environment. This custom tool chains are addressing the four different compiling environment that we have available on the system. So the AMD, the Cray, the GNU and the Intel ones. It features a mod modules with Lua syntax. So the modules are based on the Lua version 831. And we employ the fully hierarchical module naming scheme as implemented by EasyBuild in agreement with the hierarchical module naming scheme. We have the tool chains modules used as meta modules, as usual. So you build stuff with the custom tool chain that I mentioned here on top. And then in order to disclose the modules that we built with each one, you have to load the tool chain module. Just to give you an idea of what we built there, the scientific software is quite... It's a bit less than what we have on the flagship system, but still we basically built the most used applications at CSES and libraries. And we have also provided some tools for building containers, running runtime, not yet runtime, but building containers with Builder. Then CMake, Greasy as a Metascadular and so on and so forth. And last but not least, Pitstein, the current flagship system. It's a Cray XC50, XC40 as I said. And the main information here regarding with EasyBuild is different with respect to the previous two in the sense. So here we had a long experience with the software stack and from the starting in 2015, and then a new software stack in production with EasyBuild with 2016, we have provided also here a kind of hierarchical, half hierarchical modeling scheme because the two nodes, the two different architectures are targeted by two metamodules that are called Dane GPU, Dane MC. So the users can decide which metamodule to load that will load all the software that has been built targeting in the cross compilation that architecture. So once you select the architecture, the module name scheme is flat. So also here we use TCL modules. And so the hierarchy is quite limited, but still it was a starting point. Now, one thing that is not really directly related to the use of EasyBuild, but I think it's good to know in case you have to maintain your software stack on Lumi and you want to use some of the experience that we have with EasyBuild in maintaining automatically a software stack. We have, I show you here briefly how we integrated EasyBuild with Jenkins. So we have been using in the past years the Jenkins interface in a Jenkins service inside CSS for continuous integration. So we deploy software packages on the system in production using a Jenkins project. We have a separate project for testing new EasyConfig files that are submitting input requests on our GitHub production repository. And we also have two more projects, one that checks regressions of EasyConfig files and one that updates the production recipes when we have system upgrades. So the names of the projects that are available as pipelines on the GitHub or CSS production repository are production EB, the one that builds EasyConfig in production, testing EB, the one that is triggered when we submit a new pull request on our GitHub production repository. Rebuild EB runs EasyBuild from scratch and it makes sure that we can reproduce a build with a certain version of EasyBuild and a certain version of the Cray programming environment. And last, hopefully, it's update EB that runs EasyBuild to update the recipes for a new CrayP when a new CrayP is installed on the system. So as I said, we use pipelines that are stored, the scripts of each pipeline is stored on our GitHub production repository. The use of pipeline enhance the flexibility of the actions that we can perform with our Jenkins projects. And also it makes sure that we have a Jenkins file script that is version controlled on our GitHub repository. Furthermore, with pipelines, you can run in parallel on different system as a same project. So the CI can actually use more efficiently the available resources on each system. And just to introduce you to our EasyBuild production repository on GitHub, I paste here what you can find also at the link below. The github.com ethcss slash production. So you can submit, users can also submit pull requests. They just need to clone, to fork the production project and start from a new branch. And then they are actually encouraged to contribute if they want. In order to create a pull request, there are some rules like matching system names and so on and so forth. But apart from that, it's a standard GitHub pull request. After they submit the pull request, this will trigger the Jenkins project testing a bit that will start testing the pull request on the selected systems. And the Jenkins pipeline script that I mentioned so far are available also here at this link on the production repository. So this is to give you a feeling of what will happen. So this is the production EV pipeline. After the initialization, it will start deploying on the different systems. Here you see the name DOM, which is a TDS and Iger. And it is triggered when you start the GitHub pull request. So you put on the top of the pull request the names of the system where you want to deploy your new recipe. And then the pipeline will trigger the EasyBuild run on each of the selected systems and the selected architectures as well. So as I said, with the pipelines, you can run on multiple systems and multiple architecture in parallel. So it's quite convenient. I mentioned also a regression test. So it's called the Rebuild EB pipeline, which rebuilds all the easy conflicts that we have on a certain system. Also downloading from scratch the source is only to verify possible regressions, changes of links and so on and so forth. Since we have now at CSCS Atlas in Gira is the main tool for ticketing system and documentation with Confluence. We also integrated our Jenkins project with Gira with the Jenkins plugin Gira steps, which also allowed the creation and update of Gira tickets automatically. So let's say we have some failure in the rebuild in the regression test, then in a certain recipe, then automatically a Gira ticket is created and it's followed. So if somebody actually takes action, then it can re-trigger again the pipeline. And then if the fix has been successful, the Gira issue can be closed automatically by the pipeline. And last, the update pipeline. So as I mentioned, we have also this pipeline in order to update automatically the easy config files and we have an update of the CrayPE on the system. So the pipeline allows you to select which target CrayPE do you want to address. And then it also goes through the Gira stage in order to submit the Gira ticket in case of a failure like in this case, if you see right here. And then if everything is successful, it submits automatically a pull request on our GitHub production repository in the GitHub stage of the pipeline. And for users, we have on our CSS user portal, user.css.ch, some basic instructions on how to use it and how to load it. As I said, we have a custom easy build module that is meant just to set the environment for users. And it can be adjusted with custom variables if the user want to have their own specific environment. And the main most important point is this EB custom repository variable that points to the local clone of the production repository that we have on the GitHub. And it also sets the path like the XDG config tiers that easy build uses for the configuration files. So these are some useful links that you might want to keep in mind. So mainly, as I mentioned, the user portal and the GitHub production repository. We have the GitHub production repository mirrored under the EasyBuilders GitHub repository. So there is a folder on the EasyBuilders called CSCS where you can find all the production recipes, the pipelines, the specific module naming scheme, the tool chains and the custom easy blocks that we use at CSCS for reference. So thank you for the moment for your attention. I will make maybe a break just to see if you have questions around. The only question I have is whether you can share these slides when I'm sure that's not a problem. Yeah, yeah, sure. These slides were basically the EasyBuilder user meeting slides. So yeah. Okay. It's completely public. Yeah, and it's actually quite interesting at the moment because you're also starting a discussion about automating some things. Yeah. Just one or two days ago, there was a discussion started about also using things like Jenkins or other CI systems. Yeah, yeah, exactly. So there are different possibilities that the moment we are using Jenkins is the CI production, but we are also experimenting with GitHub pipelines so you can have different tools. I see that Johan has a question. Now I can see the raise hands, please. Yes, several questions. So the Jenkins CI, so is it running then from GitHub on the CSCS systems actually or? Well, so the Jenkins server is running the project. The Jenkins server is polling GitHub production repository frequently. So if you have a change in GitHub, then Jenkins start triggering. The Jenkins server itself is not public, is under the CSCS network, but yes. So GitHub triggers Jenkins and Jenkins starts to run on the systems. There is only a minor intervention needed by CSCS staff to trigger because once you start a pull request on GitHub, if you are not a member of the project, then you need a member of the project to get you to allow you to run in the sense, you need just to comment like okay to test for the pull request. This is just to avoid the weird, let's say pull request with hundreds of new recipes that will maybe take too much time in order to test. But other than that, if we get requests with a single or a few recipes, that's fine. Okay, okay, interesting. Yep, no problem. Perhaps we should not go too deep into Jenkins and Automatization, but I was just curious what you would say because I guess you spend a lot of time setting up this. So in the end, what were the main benefits? What do you say? What things can you do now that you didn't do before or what changed in practice with the software installations and maintenance and so on? Well, I think that there are advantages in the sense, for instance, the automatic deploy of recipes, of new recipes on the system that you don't have to follow that manually after new recipes are added in your repository and put in production. We have actually, you can add something in the repository so a new easy config file, but then it doesn't automatically, it's not automatically deployed if the recipe is not put in a production list, what we call, because of course, not everything that we have available in EasyBuild is deployed because sometimes users at CSCS ask for specific software that we don't support, but instead of giving them Hadoq manual, let's say installation rules, we tend to give them an EasyBuild config file, an EasyConfig. In this way, also the user can build the module that can stay in the user space, plus it can retrieve the same build after an upgrade with just a try tool chain of EasyBuild. So I think using then an automated way of managing also users' recipes that can stay in our repository, it's a good advantage. Then second thing I think as an advantage, not only deploying in production but also regression testing, because as you know, you might have software that changes like broken links or configuration instructions that might change with the version. So having the possibility to have a Jenkins project that periodically rebuilds everything from scratch and checks possible mistakes, it's also a good idea because you don't have to think about that. And then you automatically get also the integration that we have with our ticketing system, the Gila-classion one, you can also get an immediate issue that you can assign to stuff. I see a question by Thomas, please. Yeah, regression testing, you're doing this reframe, I guess? Yes, but here I'm talking about regression testing for the EasyBuild recipes. So it's a regression testing that is just limited to building stuff, not running runtime like an application. Yeah, we also use a reframe that is developed at CSCS for testing the sanity and performance of the applications that we build. But we don't use it for building the applications, like regression way. Okay, thanks. So I hope I've answered Peter's question also about the advantages because if you do things manually, of course you might have your own way in the sense you can still implement something. I think nobody does things manually nowadays, in the sense everybody has some kind of automation process. The fact that we rely on Jenkins, EasyBuild, GitHub, we might use GitLab pipelines instead of Jenkins pipelines does not matter too much, but using standard tools also reduces the need of stuff going through the tools and fixing the tools because there is a large community behind the tools and you can spend time instead directly on the specific items that you have to customize on your environment for your users. Yeah, thanks. So I mean, the reason why I'm asking is that you can have different goals, I guess with a project like this. So one goal with the automatization could be let's say that we want to increase the number of software packages available. So we will use this to automatically install everything, for example, I'm just simplifying now, or you could also think that there would have been a different goal, let's say that we do not focus on providing more software but increasing the quality. And I think from what you're saying, I think it was more the second type of goal here that you wanted to make sure that the software that you have is brought up to date and it works and so on. Yeah, maybe more than the first one, yeah. Yeah, that's correct because we actually, before increasing the number of software that we maintain, we need to have stuff that will support also requests for the software coming in. So let's say if we add a new piece of software that nobody is actually able to run or to check if there are issues related to runtime production of users, then it's a problem for us because of course, one thing is deploying something on the system and making it available. Another thing is receiving requests saying, this doesn't work with this input file and so on and so forth. So we tend to maintain a restricted set of applications and libraries that are published on our user portal, user.cs.ch and we tend to avoid increasing this list until we have, let's say, many requests coming from users. So if we see that several users need a new package, then we try to accommodate for that. But other than that, if it is a single user request or a single group's request, then we provide the easy build recipe but we don't publish that and in the sense on the user portal or so on the documentation and we just provide it as it is in the sense we leave the user, the easy build recipe and the user can build its own module on the user space. I've seen a question by Alfio. Yes. So I might have a question. I see that you have your production GitHub repo. And my question is, well, you have a lot of easy configs, easy configuration file that you are customizing for the Craig EX. So I'm wondering how long takes for you then to push to the standard easy build, to official easy build? Yeah, we are a bit lazy there. So we don't push because actually we have the CSS repo that is mirrored under easy builders. So since some of the easy configs that we use for some applications are rather different with respect to the standard easy build way to build the code, we actually don't push directly to the main easy build database, let's say the repository, like the easy config repository. This depends also on, so we advertise the mirror, CSS repository. So users that are interested in Craig recipes because we mainly push Craig recipes that are for the Craig system on the repo. So users can go and browse the mirror in general or if they have access to our system they can find immediately the robot path will find it immediately. So as a user then that I have, I want to make my own, you say that maybe people can make it their own because they have their own application. You suggest to push, to make a pull request to your production and then you take care to push to the official one somehow that's my understanding. Yeah, we don't push to the official one but we have a project that automatically updates the mirrors under the official one. Yeah, great. Okay, thank you. And this kind of a project actually makes sense. So not being part of the central easy build repositories is not a problem because easy build gives you lots of flexibility, easy config files, you can pick them up from anywhere. So you'll get a repository there, it makes sense. This is true. And also for easy block send stuff, it's easy to plug in a new easy block or a different easy block. And not only CSS is doing this, mostly because they work on Cray and I guess they're maybe the only or certainly the main site using easy build on Cray. But the Uleak Supercomputing Center has a very similar approach. They have their own repository as well, even though they don't have Cray systems. So they manage their own software stack with easy build and they steal some stuff from the central easy build repository but they mostly build their own set of easy config files. Yeah, for easy config, for easy config, it's maybe not a problem. I'm thinking more about easy blocks where you just need to probably make an extension. So let's say adding a new compiler, right? Cray compiler for instance, that this is the problem I'm facing now. So, okay, everything clear now, no problem. So yeah, there is a lot of flexibility for easy blocks, you have to be a bit careful. If you don't agree for whatever reason what easy build has centrally, you can replace it. If you wanna have an updated version, then you need to maintain that updated version as well because we will make changes centrally and it's up to you to keep that in sync, of course. Yeah, yeah, yeah. Okay, thank you. Yeah, the same happens. Let's say it is very good that with easy build you can have your custom easy blocks. You can have your custom module naming schemes. You can have your custom tool chains and this will be part of your local easy build installation but then as Kenneth says, if the standard easy build changes you need to adapt also your custom files. So for, again, from my understanding, so for Lumi, this is again a Cray that you are, you see that the possibility to use the CSS stuff. So the two sites will be like in sync. This is the way you see it because both are Cray and maybe they can share most of the stuff. So, you know, it must be some common files. So let me say. I guess it depends. CSS is now used to playing in their own repository. So changing that is gonna be some effort from their side. There's definitely stuff you could share if you want to work closely together or the Lumi team could use what CSS has as a starting point and look at how they organize their tool chains and easy conflicts but maybe they wanna make different design decisions or have a different focus. Yeah. Yeah, sure, sure. It could make sense to have a bigger collaboration, something more central for EasyBuild on Cray, which could live in the central EasyBuild repository and then the management of that repository would be up to the Cray sites. That's certainly possible. Yeah, that's the way I see it because otherwise it's duplication of effort sometimes, you know? What I'm doing now because I'm actually doing most of the work with EasyBuild at the moment in the Lumi team is I'm heavily inspired by what CSS is doing. I even have a similar structure for our repository but I have, for instance, my own EasyBuild custom equivalent module because our system is set up slightly differently. We have to deal with all the partitions. My tool chains are essentially the same though I have some suggestions for changes in the EasyBlock and then we can see whether CSS takes them over or not but even with those changes they should behave in a fully compatible way so that at the level of recipes we can again share but since Lumi is a very different system at least if you look at the GPU part I think there will also be a lot of things that don't make sense sharing initially or as sharing, putting in a common repository may just cause too much burden to manage it so we will have to see how it proceeds but Switzerland is part of the Lumi consortium so I'm sure there will be very good connections. It depends a bit also how the repository will be organized then so you could have directories per site or something and then a common directory where you share common stuff and you could have a policy that says anything specific to Lumi we will manage and we don't expect CSS to be involved in that and the other way around and maybe other sites like I understand PDC is maybe interested as well but it will cause a bit of organization and I can tell from experience with managing the EasyBuild site or the EasyBuild central repositories for close to 10 years now it's also a bit of work there's a big return in doing that work but still you have less control in some sense and sometimes it slows things down but working together will have big benefits as well. Sorry, the problem is that you also need a good overview of all issues before you can even start making that discussion. My question was from again from let's say the workflow that you see, right? So, you know, from my HP guy, right? So I have to push something and I'm using CSS stuff but you know, so my question, yeah I'm doing an EasyBlock specifically for Cray now compiler that's my question was should I push to CSS repo or to the official one? So I know now that I will push to CSS so expect my PR then. Yeah, okay. Right now there's no difference so there's only the CSS one? I see, I see, but you know, it's more than general question than if someone else will do the sync for me or should I have to, right? So that's fine. Yeah, it's probably easy if you push to the CSS repo if you are actually developing your recipe on CSS because as I said, some easy, we have some custom easy blocks and things might change if you try to push on the standard EasyBuild repo. Okay. Yeah, of course for Lumi, as mentioned by court, we have, we are part of the Lumi consortium and we have a colleague, Rafael who is part of the Lumi user support team. So we will have to support users as well. And so I guess that we will also have to maintain recipes for EasyBuild on Lumi. So I don't know yet how this will be done because this is a decision of the Lumi user support team. But then of course we will try to cope with that. It's one of the many things we need to discuss after these sessions. Yeah. But we can make up our mind how we will use or how we intend to use EasyBuild. Because there's not only CSS, we also have to take into account that other countries will hopefully want to help in contributing to EasyBuild recipes. I think that the most important thing I guess is that we find some way so that we can access each other's recipes. I think that's the main thing. And then long term we can think about whether it makes sense to have a common repository. But I mean, right now in Lumi, we are looking at the CSS repository. So we see what's going on there. So it makes sense to put things there. And then most likely we will make our own probably stuff for the GPU part. But the important thing is that other sites also can access this if they need it. I think that's that. Because it's rather easy to, if you know that you can go somewhere and copy some easy configs or config blocks, that helps a lot I think. Okay, so I guess we can transition towards the actual tutorial for EasyBuild on Quay if you don't have more questions. So I think that's the starting page, part four, EasyBuild on Quay Systems. And here I have on this side a shell already open on the login node. It's called user access node, as you know from the documentation on conference of the ALP system with the IGAP partition. I might be cheating a bit and I might use a TDS system that you don't have access to. It's not so different in the sense it's more or less the same. We just have an additional Quay PE installed on that one because on the current production system we just have a single Quay programming environment version. It is the version of April. So some of the features that I will talk about are not really meaningful for with a single Quay programming environment installed on the system. Like I mentioned, the CPE module file. But still we plan to have new programming environments, Quay programming environments installed on the system so it will make sense. And there is also one disclaimer that I have to give you at the beginning in the sense since the system is new there are still some bugs in the programming environment. And we have open Quay cases so that I cannot disclose because this is actually confidential. And we still have something that needs to be fixed. So if you use the system you might find actually issues. And sometimes the issues are not your own fault but it's the environment. So please just I ask your patients about that and in case you find the issues today just raise your hand or just talk and I can tell you if the issue is due to your specific easy config files or it is something in the system. In particular, there is an issue with a module, the CPE module and sometimes you might have your environment not finding some MPI library if you change some modules. It's just to give you an information in case you find an error like MPI not found or package config for MPI not found. This is a kind of error that you might expect sometimes depending on the order you load some modules. This is something that will be fixed in the programming environment 21.04, the May one which will hopefully be installed in production in the production partition soon. Just this is an initial disclaimer just because if testing you find that then you don't get upset. The tutorial is meant to be not so long in the sense the three hours initially planned were of course too long for this tutorial so it would be quick and it's mainly meant to be interactive. So I put hands on for each section because as also Kenneth suggested it's better to show immediately things as we go through them in the sense that you can also maybe try to reproduce the same thing in your own shell with your own environment and raise questions in case something is not clear. Then let's start before going to the actual system to the actual easy build implementation and start with the introduction to the Cray programming environment. You might already know something but still it's good to have a quick look. Now the information provided here is partly available on the craypubs.cray.com and as you are a user of the system you have access to much more of this in the MAM pages on the system as well. So the Cray programming environment on the XEX system is called CPE and it provides tools designed to maximize the productivity, the scalability of your applications, the code performance. So these include compilers, analyzer of performance, optimized libraries and the burgers. Now the CPE components that we mainly use with easy build are a subset of these components. This feature the Cray compiling environment called CCE which provides the Cray compiler that performs also code analysis during compilation. The Cray scientific and mathematical libraries. So it's a set of high performance libraries that provide portability for applications and in particular it provides the routines for linear algebra. So BLAS, LAPAC, SCALAPAC and also it provides a fast Fourier transform module for Cray FFTW. Then the Cray message passing tool kit, Cray Ampage. So this is not only Cray Ampage but also collections of other software libraries that are able, are used to perform data transfer between nodes as usual in message passing. And then there are additional libraries that I put here under Cray environment setup and compiling support that help you in support your code compilation and setting up the environment. Part of this are the so-called Cray compiler wrappers. These are wrapper commands that call the selected compiler plus they put in your command line include folders, library folders for linking the correct libraries and so on and so forth. At the bottom you see Cray performance and analysis tools and Cray debugging support tools. They are not less important but we are not using them in our easy build implementation on Cray. So I put them just for completeness to describe the different CP components but they are not really relevant for this tutorial. So we can skip them quickly. Although you might want to know that we have any way some specific debugging support for the HPC community on the Cray like GDB for HPC and Valgrind for HPC which I mentioned at the bottom. Now let's discuss modules. So the Cray programming environment is organizing the different components in modules. Now you are probably most more experienced than me in the module environment. So I would just go through briefly because you already know most of the thing but of course the module command is the main tool that you use in order to see the modules available on the system and browsing through the environment modules. Now you can have the modules in the tool command language, TCL with the TCL code syntax for the module files or Lmod. Now since recently so in the EX system Cray sites can also choose Lmod as a modules management tool. And that's what we have at the moment on our system. So if you type eco Lmod command you will find that and also Lmod version. We have version 831 that I already mentioned in the slides of EZB and CSE. So Lmod as you know is a system based on Lua syntax for the module files. So it's rather different with respect to the tool command language. In some cases it's, I think it's a bit more modern and it helps you dealing with issues in a more efficient way sometimes, at least to my experience. Now one note, environment modules, so TCL modules and Lmod are mutually exclusive and they cannot both run on the same system. That's just to avoid some strange mixing that you might have in mind. And also one more important thing, there are some configuration files provided by Craybot for the Bash and the C shell that allow each site to customize the default environment. So once you log in on a Cray system you will find already with the ML command some modules loaded by default. Every site can override this, but usually we keep them because there's a, of course it makes sense to have already some modules loaded. I will mention the login environment and the modules loaded already at login later. And these are crucial because they define some environment variables that are really used by our easy build implementation. So if you are one of the typical hacking person purging the modules that you have loaded initially and loading by yourself, this might be tricky on the Cray environment. So I would suggest not to start with a purge but just to try to have some, let's say experience first. Then more information can be found of course in the man pages. So module and module file man pages. If you type the man module, you should get not sorry. That's something that in fact doesn't work as always, but anyway, nevermind. The documentation maybe is not as complete as I meant. But anyway, module dash H will provide you some information on the main comments that you can have with the modules with the mod, which are more than what you have in general with the environment modules. So if you are not experienced with the module might take a while in order to go through this information. So that's for the generic introduction to modules and a module in particular on the system. Let's go now to the specific modules of the Cray programming environment. Now, programming environment are organized in meta modules where meta module means a module that is actually loading some parts and some other modules that are meant to create the programming environment, the compiling environment that you want to target. For instance, we have on the Cray system four main meta modules, program Cray, program GNU, program AOCC and program Intel. So each one is targeting the Cray, the GNU, the AMD and the Intel environment, loading the correct compiler, adjusting the path to the MPI library and the Cray mathematical library and so on and so forth. And also loading the module that provides the Cray compiling wrappers. So these wrappers are CC and FTN and capital CC for C++. So if you type on the system CC, then dash help will give you some information on the wrapper and on the flags that you can use. So since there is in general a lot of information sometimes you just go to the relevant main page. Now the main feature provided by the wrappers is the fact that they call the correct compiler and the appropriate options to build and link the applications. They also link the relevant libraries. So you have some libraries already loaded at the beginning like as I mentioned, the compiler, the network library and so on and so forth. So these are all fed automatically by the compiler inside your command lines. So you don't have to worry about the include path for MPI, the link path for MPI and the mathematical library, the FFTW and so on and so forth. As far as you load the correct module. So for instance here, if you load Cray FFTW, you will have it on the system and it will define some variables. And then once you will call your compiler, it will feed automatically in the command line, the include path to Cray FFTW. And once you link also the path to the library. So I think you can use also the Cray verbose in order to see, okay, with some profiles, of course, in order to see, which is the actual command line. VP, just let me show you. Okay, now I would just have to switch compiler. Forgot about that. So I have now loaded Cray Bitsai. I have Cray FFTW, although it's not really used in this small example. But then if you compile with Cray verbose, you will see that after the initial command, so I use the Cray compiler wrapper CC for C code. Yeah, I'm calling that on a FORTRAN code, of course. Yeah, that's why. It will show you the beginning, which command line is actually used by the Cray wrapper. So that's useful to understand what is involved. Don't know why Cray Ampage got unloaded. Now we have it. So the Cray P verbose option is one of the most useful ones if you want to really have a feeling of what the compiler wrapper is putting inside the command line. And it's one of the options that I use most mostly. Now, let's go to Lmod. As I said now, in addition to the tool command language environment modules, you have the possibility to use Lmod in order to manage the modules on the system. And that's what we have now at the moment on the system at CSES. The CP implementation of Lmod that we have is hierarchical and it manages module dependencies in a way that are consistent with the compilers that have been built in the module. So for instance, one specific example that some user actually realize when they look for, when they come on the system from a flat module environment scheme, if you want to load, like if you use on a Cray system to load Cray net CDF, let's say. So Cray net CDF in general cannot be loaded immediately and cannot be found even immediately because it's in the current implementation with the hierarchical way of the modules on the system, you can see Cray net CDF only after Cray HDF5 is loaded. So on with Lmod, you just do a module ML spider that's the command to let you see all the modules on the system, even if they cannot be loaded immediately with your current module path. So with our current module path, we cannot find Cray net CDF with a common ML spider Cray net CDF, we go through the different module paths that Lmod can load and we can see that we have different versions of net Cray net CDF, different module matches and in order to see which command you have to use in order to load it, you just need to choose the version. So like module spider Cray net CDF 4744, we'll give you exactly the command that you have to use and Lmod tells you here that you need to load one of these combination of modules, one per line. If you want to load Cray net CDF, you can use the AOCC compiler with Cray HDF5 or the GCC, the CC compiler. You don't see here, for instance, Intel because it's not provided for the Intel environment. That's an example of how to deal with the hierarchical module name scheme. Now here to ensure optimal use of Lmods, then you need to keep that in mind. So Lmod uses families of modules. So modules are displayed using this constraint and Lmod automatically loads a default set of modules for each family in the sense compiler, network, CPU, just to have an MPI modules that are related to the family of modules that is available. So for the moment, this is for instance, in this case, I have the GCC compiler loaded in the programming environment group and then all the modules that are belonging to the group family will be disclosed. You have then some links that you might want to consult the user guide for Lmod, if you're not an Lmod user, an experienced Lmod user. And then in terms of documentation for the Cray programming environment, Cray pubs is the main source of information for the HP Cray programming environment. So you can find a documentation specific to the PEs on the PE tile on the Cray pubs website. So this is the PE tile with the recent announcements. So the recent PE, so you probably know about anyway, Cray delivers a different PE almost every month. So you have two digits for the month, two digits for the year. So the 20.03 is the March 2020 and the Cray compiling environment and other stuff. There are also white papers and a Gita project. So the Gita project PE Cray provides additional documentation in particular I found there are some white papers that you might be interested in. If you don't have questions, I will go through them to the next with external modules. So having a quick view of what are these external modules that we will use in our easy build recipes targeting the Cray programming environment. So easy build supports the use of modules and we talk here about the external modules. So modules that are not installed when with your easy build recipes. So modules that are called with the easy build flag external module in the dependency list. So this feature of calling an external module is used extensively on Cray systems because we already have a full feature a fully feature programming environment. So it makes no sense to build from scratch a lot of things which are already available and are optimized for the system. Here in this dependency line you see the example if you want to use the Cray FFTW module that is already available on the system then you just call Cray FFTW and external module. Here I don't put any version and for this dependency easy build will load the module and it will load the default module that it will find on the system with the Cray FFTW name. Now if the specified module is not available easy build will give you an error message the dependency in fact cannot be resolved and now I'll go through the main external modules that are used as dependencies in the Cray programming environment. Please note that the library specific man pages that I mentioned here will be available only if you load the corresponding module. So for compilers I already mentioned that the CPE supports multiple compilers and users can access them loading the corresponding meta module create new Intel or AOCC and the corresponding compilers with their dependencies will be then available. I mentioned already the dash help command to invoke the information on each compiler wrapper and the compiler wrapper will call then the correct compiler as I already mentioned before. So in order to write an effective easy build recipe in general we suggest to define the compiler wrapper the Cray tool chains are doing that already for you. So you don't really usually need to define the environment variable like cc equals small cc or cxx equals capital cc or the fc compiler variable that you usually define for FORTRAN compilation. Still in some cases you might need to know that some environment might not be taken properly. For instance, often it happens with CMake not taking correctly the variable and then you might need to tweak a bit the configuration in your easy config. So it's good to know that you need to invoke the compiler wrappers if you want to have the compiler, the command line and the link line completed with the modules that you have loaded. One more note on the system or the dynamic link it is supported with compiler wrappers so you might take this into account if you have specific needs. I will then go briefly through the different compiling environments with the main modules just to understand what you can expect to find. So the module program Cray it's the one that loads the Cray compiler environment, cc. The Cray compiler environment provides FORTRAN c and c++ compiler. We have now the cc implementation based on clang as default because the Cray classic compiler is no more supported. For more information, you can still look at the man pages. Now I guess if I do, I have the gcc loaded. So if I switch to program of Cray the module replaces that. So I have now the cc module loaded. So if I type mancray cc I should find the information on the Cray compiler which is as I mentioned based on clang. So it's the Cray clang c compiler now with a lot of information about that. So I will skip this additional stuff and go directly to third party compilers. So AOC is the AMD compiler that you find in the module program AOCC. I put here the link to the documentation the online documentation and again some information can also be found locally with the demand pages on the system. There are also some bundled packages supported by the Cray programming environment for AOCC that you can find for instance if you load program AOCC. Again, the MPI, the message processing library is adopted and also the Cray libside with the mathematical functions adopted for that. So once you switch, all the modules are corrected to the point to the correct path with the correct libraries for the compiler environment that you choose. Same for GNU and for Intel. Now for Intel, we will have recently the Intel one API at the moment on the system, you don't have it yet. You see probably that here because if I load more programming Intel, I'm on the test list, no, it still goes to the Intel 19. Yeah, so it's not yet the one API but I think MKL is one of the most recent ones. So it's anyway good to have Intel sometimes because of course Cray provides the Cray libside with the mathematical libraries, mathematical functions but sometimes in order to cross check some results you might want to test also other mathematical libraries. At the moment we have the Intel one on the system as an alternative. We don't have an alternative for the GNU compiler environment except for Cray libside. So users in general, they build themselves additional mathematical libraries from scratch if they need them. Now the Cray scientific and math library as I mentioned is called Cray libside. That's the module that I have here already loaded and in general it's loaded at login. So you see that the module defines some variables that are used generally by the the easy build implementation that we have. Currently the default version on the system is the 2104 because we have the P2104. There are usually not every P comes with a different version of mathematical libraries but recently I've seen improvements and new P is like 2105 and 2106. I've seen defining a new version of the Cray libside. As I already said, this is a collection of numerical routines that are optimized for the Cray system. And they are commonly used by our users on the system together with other additional third party libraries but still this is the core. In general Cray libside and Cray FFTW are the main libraries that you want to have in your compiler environment. Now the scientific libraries that are provided by Cray libside are listed here. So these are mainly the linear algebra packages. Plus there is also some external routines for interfacing like LAPAC interface, C interfaces, LAPAC routines with C interfaces and libside underscore ACC, libside AC that is also a set of K optimized linear algebra routines for accelerators. Next in the list, one of the most important components is the Cray Ampitch. So the Cray message passing interface based on the Argon Ampitch implementation that implements the standard 3.1 of MPI. Also here for each programming environment you have a specific MPI library that is adjusted immediately by the program. So we have here program Intel loaded then if you show Cray Ampitch you will find that the parts that are loaded and defined here are adjusted for the Intel environment. So for instance here you see the prefix that is pointed to by this Cray Ampitch prefix environment variable has been adjusted to point to the Intel version of the library. So one module that is also relevant is a DSMML module, Cray DSMML which comes together in the system. I think it's already loaded. No, usually it's loaded at login and it's managing the memory management. It's a memory management library so it manages the memory in the message passing interface. So it's an important module and you can have more information also here with the intro DSMML page. Recently we had submitted a bug for that and it has been fixed in an upcoming programming environment. So still it doesn't affect at the moment the runtime is correct. So you might be confident that if you manage to build your application then it will be fine at runtime. Now after this short overview of the components I would like to talk about the easy build metadata because the external modules are defined in the metadata that can be supplied to easy build. Now in order to do that you can supply a configuration option to the common line which is dash dash external modules metadata or you can have the location of the metadata file defined in an environment variable. So the file, the metadata file is in any format with a section per module name and a key value assignment. So on CSE system we define the external module metadata with an environment variable that is available once you load the easy build custom module file. So it's called easy build external modules metadata because as you know with easy build you have the same name for environment variables the same model naming scheme as for the common line option. So you can find an example in the CSCS guitar production repository. For instance here I show you an excerpt of the metadata file for 2104, the P20 of April for KFFTW well you have the name of the module file you have then the name that easy build will look for in order to define the EBROOT variable and then the prefix and the version that easy build will use in order to expand the EBROOT environment variable that you might use in your scripts. Now just to have a look if you click you can see that mainly so this is again the CSCS guitar repository and you can find the different external modules that we usually employ in our recipes. So you can define more of them but in general we try to recently we try to limit that to the essentials. Now one more important external module is the CPE meta module. So if you type module level CPE on the system in production you will find just the CPE for 2104. I have here on this test login node also CPE 2105. So the CPE corresponding to the May Cray programming environment release. Now this is a new module in the EX system that provides a correct environment for all the defining all the defaults for the selected version of the CPE. So it has more or less the same purpose of the CDT and CDT CUDA meta modules available on the Cray XC systems if you are familiar with them. If you type module show CPE you will find this information here that I display so it just sets environment and loads the correct modules depending on the Lua modular C file that I list here. So this defines the version of the defaults. So for instance now here for the 2104 you will find the defaults for that are compatible with the Cray P 2104. Of course the meta module can be overridden in the sense a site can decide to have his own defaults. In this case you just need to provide your own module ARTHC.Lua file to put inside your custom CPE module file and you need to make sure that in the module path the location of your CPE custom comes first. In this way every user that will load the CPE module will load your own custom the site custom CPE module file. This has been useful in the past for us in order to override some defaults for instance GCC defaults, GCC compile defaults or similar stuff. So if you don't have questions I will then skip to the next. I actually have remarked about this particular module since one of Cray or HPE is present using set ENV to set LMOP module ARTHC file is very asocial because it overrides any module ARTHC file that the site may already be using. So it would probably be better to use prepend path since you can build a path of multiple module ARTHC files. Okay. I actually wanted to use a module ARTHC file also on LUMI for some LUMI specific modules to set defaults there. So they nicely got overwritten as soon as I loaded the CPE module. So for LUMI we will need a customized one just for that reason. Yeah, you can actually create your own CPE file as I said and prepend it to the module path. Yeah, but it could live together in a better way since LMOP module ARTHC file is actually also a path style variable. You can specify multiple module ARTHC files and they are already in order and separate them with a column. So that's why I suggest that prepend path would actually be a better option than set ENV. Then it would happily live together with whatever site is already doing since it would just add to the number of module ARTHC files that are being used while now we just hijacks it. Yeah, of course. I mean, you can always ask, you can submit a request for an enhancement to CRE and they usually take them into account. Although, yeah, it might take some time. It depends on the request in general. That's why I say that you can also create your custom version. Sometimes it's quicker, so that's a quick solution. But yeah, other than that, in general, CRE responds in the sense that we suggested, for instance, to add some environment variables to be consistent with our local implementation and they did that. So you just need some time in some cases. Okay, so let's go now to the custom CRE tool chains. Now, when we have easy build on CRE, we are already talking about some kind of custom tool chains because the CRE tool chains have been created as custom tool chains in order to address the CRE programming environment. Now, here I mentioned something that is classical CRE tool chains that are the CRE tool chains that you find on the main easy build repository. While recently we have implemented on the EX system, what we call CPE tool chains that are targeting specifically the CPE environment, the CRE programming environment on this system, formerly known as Shasta. Initially this was needed, so historically, because there was a difference at the beginning between the CPE environment, the CRE programming environment on the EX system and the standard CRE programming environment, like on the EXE system. For instance, we didn't have the metamodules initially, like program GNU, program Intel, CRE and AOCC, they were just defined as module collections at the beginning. So that was not matching with our implementation of easy build. And apart from that also, there are some differences. So even if now, as of the CPE 2104, the difference between the two programming environments is limited, so CPE, the custom tool chain might seem not so relevant any longer. Still, the programming environments are released with different release nodes. And you can have the same programming environment like 2105 with a different set of default modules for different CREPE components. So for instance, I show you here in our production, GitHub repository, the external modules metadata that we have for the generic, the CRE classical tool chains, as I defined here, CRE external modules metadata, and we have the CPE ones, so for the custom CPE tool chains. And if you look at the two files, you can find here under the easy build folder of our GitHub production repo, you will see that there are some little differences in the versions of some modules like CRE Python. For instance, it's one of the differences or I think also, well, I think there are also a few other differences. In general, shouldn't be an issue, but if you want to be really strict and you want to have full, you are concerned about fully reproducing a certain build, you might want to have a way to retrieve the correct version of a certain external module and then you might still need to have two different set of external modules, so two different modules metadata files. And the same, in the same way, you might need to have two different easy config files for a certain package that you install on the two programming environments because they rely on different versions for the external modules. In this sense, having the custom tool chain separating from the classical ones, I think it might be still significant, although one could probably arrange that in a different way at the moment. So I'm talking in perspective for your looming machine, for instance, you can also decide to have the classical CRE tool chain, but you can have custom recipes. At that point, you will need a version suffix, so it depends on what you want to give users on your system. If you want to give them full flexibility of deciding to customize the versions and the easy config files, or if you want to have your own recipes correctly reproducing the environment and manage that through easy configs instead of tool chains. So it's a maintainer's decision in the end. Now, let's go to the tool chains. So as of version 270, EasyBuild support CRE systems, as I've shown in my slides before, in the timeline it started in 2015, the CRE support in EasyBuild. Now, if you go to the GitHub repository, then you can find in the standard, the classical CRE tool chains support for this tool chains, CRE-CC, CRE-NUCRE, Intel and CRE-PGI. Each classical tool chain comes with a version matching the corresponding CDT module, so the CRE development toolkit on the CRE-XC system where they were initially defined. So the tool chain then will load the corresponding CRE programming environment, will load the corresponding compiler, the Ampitch library and so on and so forth, matching the definition of the CDT module file. The tool chain will also follow the naming convention of the CRE programming environment. So as it is released on a monthly basis, also the tool chains will have a version that is in the form of year, two digits, dot months, two digits. So that's more or less similar for the CPE tool chains. Also the CPE tool chains that we have created as custom tool chains for the CRE-X system come with names that are for the CPE. They were based on the CPE versions up to 2103. So initially we created these tool chains to address some modules, CPE CRE, CPE AOC-C, CPE GNU and CPE Intel. But as of CPE 2104, CRE reverted back to the metamodules. So we now have a gain-programmed metamodules. So there is now, if you look on the GitHub repository, the current custom tool chains are defining now again the program metamodules in the custom easy block CPE tool chain.py. So you will see again there, as you see in the classical tool chains, the known programs defined as the metamodules as you have in the classical tool chains. The only difference here is that for the moment we don't have a PGI or NVIDIA tool chain on the system because we don't have GPUs. And on top of the classical tool chain, we have the program AOC-C that is not present in the classical ones. This file, the custom easy block CPE tool chain.py is available in our production repo. So if you click here, you should find it. Yes, it's in our production. We follow more or less the structure. So we have an easy build folder. And under the easy build folder, we have the same structure that you find in the easy builders repository. So we have an easy blocks folder with a custom easy blocks. And in particular, the CPE tool chain.py is the custom easy block that you find here. It's practically the same as the standard CPE tool chain for the classical CPE tool chains, apart from the fact that now, as I said, you have this AOC-C and you don't have the NVIDIA. As I mentioned, it's also mirrored under the easy builders GitHub project. So you will find that there as well at the link that I provide here. Each custom CPE tool chain relies on specific files that are available in our local easy build framework. These files are listed here. CPE-MD, CPE-CRE, ACP-GNU and CPE-INTEL. CPE-MD, in particular, we look for the custom compiler AOC-C.py. While for the others, there is no difference with respect to the standard easy build repository because the corresponding CRE tool chains will use the same CRE-GNU and Intel compiler definition. So there is a CCE.py and so on and so forth. As well as the custom easy block CPE tool chain.py, also these custom files are available in the tool chain folders and the compiler folder of the CSE production repo. So if you click here, you go to the tool chain folder. So you see you have here under again, production easy build tool chains. You have the CPE tool chain scripts and you have also the compiler. So the CPE.py and AOC-C.py. So CPE.py is the equivalent of CRE-P.py. That is the file relevant for the classical CRE tool chains and that defines the CRE compiler wrappers. Yes, so that's exactly what I've written here. So the custom tool chain will look for this CPE.py. It can be found in the compiler folder. It is supporting the CRE programming environment compiler drivers that are defined by the CRE-P module file. If you look here, module show CRE-P. This is the module that's actually defining your compiler wrappers and more than that. So, and this is the relevant module that is referenced in the CPE.py custom script. This is so far for the tool chains. Now for the target architecture. So as you know, in EasyBuild, you can target specific architecture in order also to cross compile correctly. The module that is used by EasyBuild in order to target to match the target architecture is called CRE-PE-TARGET and where target is your target. So in our case, on the system, you find CRE-PE-TARGET, I think it's called AMTX86RAW, yeah. So that's the module that is already loaded in general at login since it is the single architecture that we have now and if you inspect this module, you will find that this module defines some environment variables in particular the CRE-PE-TARGET environment variable, that's X86RAW. And this variable is important for our EasyBuild custom implementation because we use that in order to set the opt-arch flag correctly in the EasyBuild builds during the builds. And now an important note, please don't unload this module unless you know what you're doing because otherwise you will mess up targeting the correct architecture on the system. So, Hendrik, are you, are you, are you, are you, we hear you typing, do you, okay, do you want to ask questions? Oh, sorry, I just crashed my Zoom and forgot that. Oh, okay, okay, I thought it was on YouTube. Okay, no problem. I thought it was because you had questions, yeah. No, sorry for that. No problem. Feel free to talk and to interrupt me in case you have questions. Also, as we go through, if you find differences or what I'm doing with respect to what I'm showing in the share with respect to what you do, it might be interesting that you raise your hand or that you talk immediately. So, this note about the target architecture might seem a bit convoluted but because I'm showing you that we actually depend on an environment variable, it is defined by a module that needs to be loaded by EasyBuild. Still, since this is already defined at login, you have already the environment variable defined. So, when you load the EasyBuild module file and then the correct module will be targeted in the, in your builds. That's why I'm saying to avoid purging the modules when you log in unless you really know what you're doing. So, I don't suggest to purge. Now, we have seen the custom tool chains and now we want to see how to use these custom tool chains. So, in order to use them, you need to provide an easy config file for the custom tool chains. Now, I put here an information about the historical reason to have some custom tool chains using modules footer because we had some improvement in EasyBuild release 4.4.0 that actually makes using footer no more relevant. So, the current easy config files can be much shorter than what you can find on our repository. If you click, for instance, the current version of the, of an easy config file, I'm showing you the CPE GNU 2104 easy config file. It was defined loading as a dependency program GNU and then in the footer, loading a lot of other default modules that I usually loaded that log in plus at the end the CPE module file and preventing also some path. Now, this was needed before EasyBuild 4.0. Now that we have EasyBuild 4.0, this is no more needed. And for instance, the CPE custom tool chain could be simply as short as the one that I'm showing here. So, it just loads as a dependency, the CPE module file with the correct version. It is the same version of the CPE custom tool chain and the program GNU external module. Now, the CPE module file is loaded because in a system with multiple CPEs, if you want to switch between different programming environment versions, you want to make sure that the defaults are loaded consistently. If you have a single CPEs, of course, you don't need it as a dependency. You will just need a program GNU. So, the advantage of this approach, loading the CPE module is that you avoid pinning directly the version of the CPE components in your module file. So, why I see that as an advantage, that's because the set, let's say, the pipelines to update automatically the easy config recipes that we have in place would actually not work so smoothly if we pin modules versions. Because at the moment, we don't have a configuration file with all the versions, with all defaults, of course, you can take that from the module RC files used by CPE, but since the CPE module file is provided, it's straightforward to use it as a dependency. Of course, the maintainers of the recipes can decide otherwise, and instead of having a very minimal definition for the easy config, for the custom tool chains, they can decide to have a list of dependencies with all the components. As I mentioned here in the module list, you have 11 different modules that are usually loaded that login, and then you can simply pin the versions of each of these modules, and this will be self-consistent because this version will be on the system if you have the corresponding CPE version. So, that's just a decision on the side of the person managing the installation. Luca, we have a question by Mashei. Yeah, please. Thanks, Nico. So, I'm not quite sure if it's a good point for asking this question as it might be quite disruptive, but I will try. So, I feel I can follow the idea and it's very nice. I mean, combining tool chains in EasyBuild and Cray CPE modules, but it's quite, let's say, consistent as you have only one architecture with CPU-only partition. But I try to imagine how it will or may work if you have a GPU nodes and you would like to produce code, for instance, that will run on CPU with OpenMP offload to GPUs. So, you need to have environment which will combine x86 compilers together with a compiler that will run on CPU-only. It will support OpenMP offload with a correct, let's say, GPU target at one moment. So, I try to imagine how it will look to have in a one moment, let's say, CPU AMD target module loaded and GPU target module loaded. Could you have some comment on this? Yeah, yeah, sure. I mean, here I'm mainly referring to the EX Cray system where we don't have GPUs, but we have experience with the system with GPUs since as I showed in my slides before, our flagship system, Pitsdient, is a hybrid system. So, we have nodes with GPUs and nodes with just multi-core processors. So, in this case, as I briefly mentioned, we have two metamodules that are custom CSCS metamodules that are targeting each one the different architecture. We have DaintGPU and DaintMC. These two metamodules will load the appropriate modules and will define the appropriate environment variables for targeting the compilation on each architecture. So, the two are mutually exclusive, of course, since they conflict with each other. And if you load one of them, then your easy build implementation will be consistent just with a single architecture. This is how we manage things on the flagship system. In view of having a similar management system with L-Modern hierarchical modules, I think the simple thing would be to create additional custom tool chains. So, you could have a tool chain CPE GNU-GPU or CUDA, or depending on, it might not be CUDA, your GPU code anyway, you can have a custom GPU tool chain. And in this custom tool chain, you might load the relevant, you might define the relevant modules. For instance, as I shown here in this easy config file, I'm pretty minimalistic in the sense I just load program GNU and CPE, but you might also load, for instance, a CUDA toolkit module if you wish. You might also load additional modules or define additional environment variables if that's what you intend to do. So, I think having a custom tool chain will manage also the software that you build hierarchically in a much easier way because you can separate clearly modules and software built for GPUs with respect to modules and software built for the CPU only architecture because if you see here module level CPE, it will show you that with this hierarchical framework that you have, once you build software that is relevant for this tool chain, all the software will be installed under the corresponding tool chain module path. So for instance here, CPE GNU 2104 will show you the software that has been installed for that particular, with that particular tool chain and then it will not conflict at all with what you have installed for other. So you see here, once you load the corresponding module file that has been created using this easy config file, you will see the list of software that has been installed using this custom tool chain. If you create a custom tool chain for the GPU instead like CPE GNU GPU, which doesn't exist at the moment but you can create that, it will be completely independent from this list. So it will not be possible to have any conflict and it will load the corresponding modules and define the corresponding environment variables consistently. I hope I answered the question. So may I, is it correct that we rather should think of two different, let's say paths, I mean for AMD, sorry, AMD CPU only software and the other path for AMD CPU and GPU in the, let's say in a single source, right? Yeah, that's what we do at the moment with the hybrid system, it's done, yes. And for the moment it has worked, I mean, so far. You might think about with a hierarchical module naming scheme is even easier in the sense you have some software that is not really depending on the tool chain. So easy build as the system tool chain for that, which used to be the dummy tool chain before some time ago. In this case, it is managed quite smoothly by the hierarchical module naming scheme since it will put in the core file, whatever it's not specific to a tool chain, like here you have these modules, these are just independent of any tool chain, they are put in a certain path that is the core path and then you have the specific paths, like as you said, CPU plus GPU, you will have a specific module path for that, you will have a CPU only module path, you can have multiple GPU architectures. For instance, you might have nodes with different GPUs and then you can have, also you can differentiate the tool chains for that. I think it is managed quite smoothly with both, let's say custom tool chains plus hierarchical module naming scheme in my opinion, at least for my experience. Right. I think I'm gonna interrupt here because that's something that we should discuss in our own software meeting. I have a setup and I have actually already presented it, but I don't know if my chair was present on that meeting. I have a setup that does something similar to that to take care of those issues so that I can have different targeting modules for the GPU nodes and for the CPU nodes so that in both cases, the Cray environment reacts appropriately and it is partly based on a hierarchical scheme. So I do have separate modules and then I do have a separate module three for Lumi-G and for Lumi-C for instance. Okay, thanks. Okay, no problem. So, so far I've talked about the tool chains but what can you build then with the tool chains? Now, here I provide a list of the maintained software on the system, on the EX system with the current default CP, which is 2104. On our way of deployment modules, we have in our production repository a symbolic link, the points, a symbolic link with a system name, the points to a list of software that needs to be installed. This is the list for the moment for the EX system. So if you click here, I think you find the same list more or less because it was updated, yes, yes. So, easy build comes with a default module flag that is useful if you want to set a module as default and you see on top things that are installed using the system tool chains and then in the bottom, you see everything installed with a different tool chains like CPE AMD, CPE CRE, CPE GNU and I can probably show you one of these recipes, maybe GSL is in general very easy. So, let me just, oops. I have a clone locally of the production repository. What's that? So, GSL is a software that we usually provide because it's used as a dependency by several projects and this is the default that we have on the system. So, this is built on top of the, well, GSL is easily built with every programming environment. So, we provide that as you can see here in this list for any tool chain AMD, CRE, we have then GNU and also Intel, I think, yes, Intel here. So, it's quite simple, doesn't have many options, just with position independent code and that's what you would do for a configure make build with easy build. So, this is not using any custom easy block, the easy block is the stock configure make provided by easy build and as you see here in the tool chain ops for this tool chain, this is the custom tool chain but it uses the same tool chain ops, the classical CRE tool chains, you can set the optimization on, which is always true because we want to target the opt arch. So, opt arch is true as well. You have unroll and you have position independent code which I also put here just to make sure that it's taken, maybe this is not really needed. So, that's quite simple actually. So, most of the configure make recipes will look like this based on the custom tool chain. So, I think since you all went through the easy build tutorial, you already know about how to write an easy config. And the reason to show you that is just to see that you can have quite smoothly all these different software built with different tool chains and managed by the hierarchical model naming scheme. So, if I type model spider GSL, you will find that GSL to the six is available and you can load it using one of this, one of this module, CPMD 2104, C2105, CPE GNU for some strange reason doesn't find the Intel CC at the moment, but maybe I can just purge my environment and so just logging out and logging in again and see some modules by the GSL. No, that's weird. Might be that on this, I didn't build everything, not just by the GSL. Or maybe it's because you have a local L-mode. Yes. No, yeah, I think on the TDS login node I didn't build everything as I did on the standard one. So I'm logged in now on the standard login node that you have also access to. If you type module spider GSL, you'll find correctly the four different modules that has been installed. So it was just because of the, I was using a TDS node. So I think in my personal experience, the hierarchical model naming scheme can do something smoothly for you instead of you writing your own metamodule files in order to manage different modules. So at least that's my experience. And also I put here an example for automated updates of supported software with new CPE. So when a new CPE comes in, a new version of the CPE, we have this Jenkins pipelines that I've briefly mentioned in my presentation in the beginning with the project update to be in the Jenkins interface that will use the try option of EasyBuild in this case try toolchain version in order to update existing easy config. So in this case, for instance, I show here the update of the Amber recipe, which we provide already for 2104, but when you have 2105 installed on the system, you just do try toolchain version 2105 and this is with CPE Intel and EasyBuild goes through that, changes 2104 to 2105. And then in roughly two hours, you have your new module. So I won't show you it here because two hours, it's too long, but still this gives you the feeling of what can be easily done using the simple easy config file that I've shown here. So in case you have pin versions, of course you need to make sure in your pipeline script to have an RC, a modular C file where you match the version, the default versions of the new P instead of the with respect to the old P just to avoid problems. And then you have to parse your easy config file for the toolchain in order to substitute the recipes versions, the external modules version. This is something that is at the moment easily done with this CPE module file kindly provided by CPE. So I think it's pretty convenient to have it. So I think we should go to EasyBuild at CSS just to show you then how we really built things on the system with our local implementation of EasyBuild. So as I've showed at the beginning, we have been using EasyBuild since 2015 on multiple systems, both Cray and standard toolchains. And we have integrated EasyBuild with our automated pipelines to maintain supported applications and libraries for our users. And the pipelines are stored as shown in our repository. So you have here the relevant links for the CSS production repo, the Jenkins pipelines. And we have also a page as I briefly mentioned how to use EasyBuild basic instructions at CSS. So here the main module that we have is the so-called EasyBuild custom module. This module provides the EasyBuild framework that is implemented locally at CSS to the users. This module defines the location of the EasyBuild configuration files, the easy configs files, the easy blocks, custom easy blocks, custom toolchains, and so on and so forth. So you load the custom EasyBuild module, so like that, what is the custom. And then you can, at the moment, the EasyBuild version that we have as default is still 434, I think, but we have the latest available. So you can switch to the latest in order to have the latest features available, sorry. In this way, you can check then the short configuration. Now, if you don't specify anything different from just loading EasyBuild custom, there are some defaults that are assumed by the custom module file. In particular, the installation folder of your software projects will be under your home EasyBuild. So it's the prefix, I think, yes. So the prefix is set as you see as my home folder users, look-amar and then EasyBuild. Then there are, of course, many other things that are set by this module because if you see in the configuration, things that are non-default are shown here. Now, we will go through them one by one. The system name here that is used, of course, in the prefix, if you don't specify it, otherwise it is the name of the system return. In your case, on the Iger system, it will be Iger here, I'm on the TDS, so it's called Pilatus and it shows up here in the prefix, so, yeah, prefix, it's this one. So EasyBuild Pilatus. Now, you can, of course, override a lot of things, customizing that, and you can also change things afterwards. Let's see what. Now, first of all, this custom module file is also available in our CSCS production repo and it's available in both TCL, TCL, and Lua syntax. Technically, it should need only the TCL one because Lmod converts with the TCL to Lua script, TCL scripts to Lua, but I was, for some reason, not having that converted correctly, so some variable was not exported correctly, so I created a Lua one. And you can find that here. So that's under production, EasyBuild module. EasyBuild custom, you have the CSCS standard TCL script module file that sets all the environment and then you have the Lua one that does more or less the same and the good thing about that is that on a system where you have TCL environment modules, the Lua module file will be ignored and on the system with Lmod and Lua, the TCL file will be ignored, so they can coexist. Now, here I also show this EasyBuild custom module file. What is actually defined? So this is a module show that will define the most important variables for EasyBuild, in particular the XDG config years and the robot parts for finding the easy config files. On top of that, you have the include EasyBlocks, include the module naming schemes and include tool chains, the points to our custom definitions of EasyBlocks, naming schemes and tool chains, as I've shown with the custom CPE tool chains, plus then a build path, TMP here, the usual things that you want to define. Everything can be of course modified by hand by the user if you don't want that. Something that is relevant just for the system, the EX system is the module syntax that is Lua, the module's tool, Lmod, on other CRE systems at CSES, we don't have that. Opt-arch, as I said, is the AMD ROM. Then we don't use recursive module unload because this will unload completely everything and will basically almost purge your environment. So I don't suggest to use that for the moment. And then the prefix. I didn't set myself a prefix before loading this custom module file. So I got the default that is implemented by the custom module file. And that is the easy build configuration that is coming out. So it's more or less what I've shown here, just a better format. Now, I said that you can configure most of the things. So the configuration options come with environment variables. In general, with easy build, you can define things either with a common line or with a configuration file or with environment variables. So we usually define things with environment variables, except for few things, let's say hidden modules mainly. So you see in your EB Show config, the E after each name of a certain item, like include easy blocks, saying that the include easy blocks has been defined with environment variable. You probably already know that, so won't spend much time on that, but then you can check it if you want to make sure that everything is set. So easy building, include easy blocks, then you will find it. Easy blocks, sorry. So you will find the path that is actually used. Because the include easy blocks as you probably know expands all the Python files available. So we have a lot of, well, not too many, but still quite a number of custom easy blocks. Among those, you can find also the CPE tool chain.py. Same goes for the other ones. So you have also, as I mentioned, the include module naming schemes and include tool chains with their custom tool chains. If you want to change anything of this, you can either export to the variable or use the common line in order to change that. Please note that the prefix might be different from the install path in general. So here, I don't have anything in my environment before loading the easy build module. So I didn't export a specific prefix or a specific install path. So I got what the default is set in the easy build custom in particular. The prefix, as I said, it's going to my home folder and the install path is set accordingly. So it's exactly the same as the prefix. You might be needing to specify different install paths in case you are a user and you don't have right access to something like a prefix that you might need to export to find the installed modules of easy build locally. Because in a flat module naming scheme, you can find all the modules at once. In a hierarchical one, you do not. So in case you want to build on top of existing modules, you might need to specify a different set of a combination of prefix and install path, which you can do choosing your custom folders as shown here. So you can override the default installation and in general, we suggest to do that. Easy build prefix can be defined wherever you want. And EB custom repository is the main location for your custom configuration because based on that, XDG config deers used by easy build is defined. So you need to be careful about that. After exporting these two modules, these two variables, you load the easy build custom and then everything will be defined according. So for instance, if I do that, export easy build prefix I go then to that. I export my install path in a local folder of mine and I export EB custom repository. Then I show you the configuration. So you have to reload of course the easy build custom in order to take this into account and then things get changed with the now exported variables. For instance here, the robot paths will take also my local path and the prefix path has changed. It's no more my home as well as the install path that is not the same as the prefix. So anyway, that goes to my sandbox. And in this way, for instance, if you want to build anything, you can rely on what easy build finds in the prefix which points to the generic prefix for users with all the software provided by CSCS stuff. But then you can write your own install path that is now in my sandbox. So you can find that convenient in some cases. So I show you here which are the variables that are depending on this EB custom repository variable. So I mentioned already XDG config here. So it is one of the main variables used by easy build. There is also the easy build robot paths include easy block, model name scheme and tool chains. And under the XDG config there is the easy build dot D folder which is used by easy build in order to define the standard configuration file that I will show you later on. So I have a question about the XDG variable there. Is that really easy build specific? I think this is a variable which is used by other programs as well. So wouldn't it breaks things if you would change this variable when you load the easy build module or is it not the problem in practice? Because I think there might be other programs that would like to store their config files and they will read this XDG config directory and they will want to look for the configuration files there or? Yeah, I mean, practically for the moment I never had such a problem in the sense maybe I never built something using this XDG config gears variable. That might be a possibility also because in this folder, let me show you the content. So there is nothing else just a single CFG file read by easy build. So I think it's, did I export it correctly? Export. There's actually another way to achieve the same thing. You can set an easy build underscore config file environment variable to point to an easy build configuration file. That works even though easy build and has to do with two step setup of its configuration. It reads that file first and then read does the configuration. So that works as well. You can do a more direct approach if that's a problem with the XDG one. Yeah, for the moment it was never a problem, but yeah, I mean, your point is correct. If there are programs using the same folder you might have some conflicts but still this is what you find in my custom repository for instance, so that is then defined to be the XDG config years for easy build. Yeah, I think there's something to keep in mind because I'm not sure how common it is, but I mean, this is something which is created by, I don't know, is it a PAM module or something when you log in, I mean this directory. So I would just feel uneasy just overwriting that path because you never know what will happen, not okay. Actually, yeah, you can check it on the system. Before loading, I don't think you have it. In the sense there is some XDG variables, yeah, you have it but yeah, it's ETC, XDG, something that you could never write anyway. So yeah, until now, I honestly don't know even what's there but since I, as I said, it was never a problem until now but yeah, it might happen. So it's a good observation. Can you actually add a warning here in the tutorial after like in this section, just a little warning to, so that we remind it. Yeah, yeah, yeah, yeah, please. Yeah, let me write it down because- Thanks a lot. Otherwise I'll forget it. Yeah, I mean, I will probably push some more changes to the tutorial just to make it less CSC as dependent and less, let's say custom, less customized for this specific day. So this is also a good input, thanks. Okay, other than that, in this folder, you will find also the, so for instance, in this local copy of my, in this local clone of the CSS repository, you have the easy config files. So easy configs as usually in alphabetical orders and then with all this, for instance, the CPE, custom tool change recipes. I think we still use also for 2105, the old format. So you'll find everything you need, typically to install things at CSS and you can put more things on your own if you define your own ones. Now, the last bit to it for the easy build configuration file, for the custom implementation of easy build at CSS is the easy build configuration file. So as I mentioned, most of the things that we define in our easy build custom module are through environment variables, except few things that are defined in this site-wide configuration file available in the easy build.d folder. So it's here, easy build, once you have your local clone, you will find it. It's easy build.d, it's here. What it defines are just some list of hidden dependencies that you don't want to show when you load a certain module with the, you don't want to show all the dependencies loaded and the list of hidden tool chains because on the Cray access system, the tool chains modules were hidden. Now, since we used the tool chains are metamodules from the EX system, we cannot hide them. So they are not hidden here. So, and then the list of allowed loaded modules. So the current content of the file is this. You can find it also clicking on this link online and our repo. And well, we have a long list of high depths. We have a short list of high tool chains and you don't find here the custom tool chains because we want them to be available, to be visible. But the classical tool chains, the classical Cray tool chains are hidden in general. So we don't need them. And then the list of allowed loaded modules like DDT, easy build custom, and XOLT. So well, not much stuff, but of course, if you prefer, you can have instead of defining a lot of things through environment variables, you can have them defined here, although maybe it's more convenient for users to define environment variables. They can of course override the definitions here because for instance here I define module syntax TCL, but this is overridden by the implementation that we have in the easy build custom module. Last, but not least, just a quick overview of what we have on Pitsteint. I mentioned before on Pitsteint, we have two different, it's an heterogeneous system. So we have two different architectures to be targeted. And then you can target them by loading a meta module that we have created. One is Dain GPU, one is Dain MC, GPU for, of course GPU, MC stands for multi-core. And then after loading that, you load the easy build custom module. So in this way, you can actually have your own, your settings consistently defined for the architecture that you are selecting. And other than that, the prefix, if you don't specify that on Pitsteint, you get not only the system name in the prefix folder, but also the architecture that you are targeting. That's something that you might also consider in case you want to implement that in your local site, having for users a specific, a different location for the architecture module files. Then building your programming, you know that. So, you know you have to do EBE dash R to recursively resolve dependencies. And I guess that's more or less it. In order to search in your path, you know that you have to use EBE dash dash search or dash S. So these are things that I've put from the main user portal of CSS, just for the generic users, but you have been already through that in the tutorial for this EBE, so you know that. I guess that is more or less the end, which leaves us some time for discussing the questions that you have posed already in your conference page, if they have not been addressed already, and also to do some stuff on the spot in case you wish to do something specific. Otherwise, I will show the questions. If you don't have specific questions now on this, I will just show again the slides with the questions, with some selected questions that you put on the conference page of your last conference. I selected here in the Q&A page, I think seven main questions, but they also have sub-questions related to you. They're extracted from your conference page. I think most of them have been answered, but I guess it's good also to go through them in case you want to raise more additional questions. For instance, the first one, where are the CPE tool chains defined in this EBE framework? So as I've shown, the CPE tool chains, the custom ones, are defined inside the tool chain folders that you find also under the EasyBuilders project, and you can also customize that under your own EasyBuild XDG config gears, and then you can include that as you include your custom EasyBlocks. So that allows you to override any definition that is already available in the main EasyBuild, in the standard definitions. So for instance, you can also, if you wish to override the classic CRE tool chains, you can also do that. You can put a custom tool chain definition in your own XDG config gears, and then include that with a command line or with the environment variable. You can find our own implementation in the EasyConfig in the CSS production. There is here something in the chat. Okay, it's just for last. I don't know if you have additional remarks about that, how to find CPE tool chains. Oh, that question came from me and it was well answered in the slides. Okay. There were a couple of files that I overlooked that I had to look in a different repository. Okay. Then how does dash dash opt-touch work with the CRE CPE tool chain? So, as I said, the opt-touch flag for our EasyBuild custom file uses an environment variable that is set when you have the corresponding architecture module file loaded in your environment. So in this case, it was called CRE-PE x86 Rome that defines the CRE target architecture AMD Rome. And that's important to avoid unloading the module or unsetting the variable if you are not really sure of what you're doing because you might end up building something that is not optimized. Can you add multiple values there because I mean, CPU target and an accelerator target because I guess that will be needed when compiling for GPU. Well, in that case, I think we have, we load a different module file of CRE and that is conflicting. I mean, for instance, on pit-dient where we have two different architectures, we cannot load at the same time like the Broadwell and the Hartzwell file but the GPU architecture is actually, this is targeting the CPU architecture. In order to target GPU architecture, when you use the CRE compiler wrappers, there are other configuration used. For instance, if you use NVIDIA, the NVIDIA compiler has a set of optimization flags for the specific GPU architecture. And when you use the CRE compiler wrapper on the CRE system, these configuration options are automatically used by the wrapper. So you don't really have to worry about that. If you have to change that, this goes rather than to the configuration of the custom tool chain, mainly to the configuration. Well, you can put it maybe if you want a specific, a custom tool chain targeting a GPU and you want to have specific options, you can put it there like until now it was not used because we use the CRE compiler wrappers. And then there is one for you and Kenneth together, probably, but I think in plain easy build from what I remembered, you can give through OPTAC options for multiple tool chains simultaneously. So I wonder if that is still possible if you're using the CRE tool chain. There was a syntax name of the compiler, I think the column and the options comma or something like that, I don't know by heart. Yeah, and then you can do GCC colon flags dot comma, I guess Intel colon flags, but for CRE it's a single value, at least currently. So there's, I don't think you can specify multiple things right now, but that could be implemented. And that's, I think this is purely done in the CRE tool chain module. So that will read the opt arch value and just try to map that to the corresponding module. So you could make it a bit more intelligent by giving it a list and somehow taking the context into account. If we want to be finding that's where to look because I think for the system compiler, you can also give default options. Yes, yeah. So if you would also use the system compiler, then it would just be fun to have everything in a single opt arch that I can nicely hide in my easy build modules to put the optimal opt arch for the kind of system for which I am compiling. So with Lumi, I also have those modules like I like, let you have a module for the CPU section and one for the GPU section on PISDaint. I have like four modules like that for Lumi for the four different partitions for which we are gonna use easy build. So that for each module I can set the optimal options. And then if you load my equivalent of easy build custom, it also takes that into account to set opt arch and so on. So I mean, the setup is very much inspired by what I saw in the CSS repository, but I have a different custom file, of course, because we have different ways of working with our systems at some points. Yeah, sure. Okay. Then another question was, how to arrange searching for custom easy blocks for you there? So I've seen in your Confluence page that there have been some additional remarks about the fact that sometimes it is annoying if the target folder is empty because easy builds returning an error. Then of course you can specify multiple easy blocks folder. So as I mentioned, easy build comes with this include easy block option or also the environment variables where you can put multiple folders. So for each folder, comma separated folder, you can put start.py. And then easy build will read all this. Yeah, but it's with the start.py that I get a problem. If there is no file in there, but it may be because I'm using a hierarchy start slash start.py. Okay. That may be the reason, maybe I should try without, but if I do that, I get an error message. If there are no files in there yet, I get an error message that block doesn't match anything or so. And that's maybe that's too restrictive right now, but it's easy build protecting you against yourself. So making silly typos and then figuring, spending an hour trying to figure out what's wrong. But maybe we can make that not an error, but a warning or make it somehow, you can make it somehow clear that it's okay if nothing is found. So that's a general easy build issue. So for that, I suggest you open an issue in the framework repository. And I think you can easily change that or make it more flexible if it's too restricted. I think this will enable you anyway, providing multiple easy blocks, folders to give users. Because that's exactly what I wanna do, but that's where it started to go wrong. And of course I didn't notice when I was doing it just for the easy blocks for Lumi because there are already had some ready, but then for a user, I set an additional folder that the user can then use in his or her setup, building on top of the custom easy blocks that we already have on Lumi. So, and there I noticed that error because of course initially user starts with nothing. Yeah, yeah. I could in my module file, I could of course, test if there is something in that module and not add it, but then if a user after loading the module starts adding stuff, it is not picked up. So I wanted to avoid that also. Okay. Yeah, I think it's pretty convenient. But in general, let's say you, apart from the error that you got, you were asking also custom easy blocks for users. In our experience, users are not so advanced to provide their own easy blocks. If they do that, they already know how to actually customize the path. And we are, let's say already happy when users provide their own easy config files because most of the times we provide the easy config files for the users. So I think it's pretty advanced. But anyway, if you have, let's say the need to customize easy blocks in some way for the different lost members, maybe you can also use this kind of approach having multiple easy blocks folders. So if you provide an easy config to user that requires a custom easy block that's different from the one in the easy build repositories, you store it under your production repository even though you don't put easy config center. Okay, yeah, it has never happened until now. In case we need to provide something that is really different with respect to what we have, like also a conflict with an easy config, not only an easy block, we usually provide that from a branch to the user. And then we ask the user to keep that in its own, in its own copy of the, in its own clone of the repository. This is sometimes a problem when we have updates and the user actually does not immediately update the easy config files, so it might happen because if we have it in a branch and not in our master repository, in the master branch of our repo, of course we don't maintain it. So yeah, it's anyway a corner case because I would say until now it was not really necessary for users to have their custom easy blocks in our experience. But of course, yeah, it happened with custom easy configs. Then I think how to use external modules and their configuration file. I think we have seen that. Yeah. And how do module parts affect the way easy build works with HMNS, so hierarchical module naming scheme. Now here, maybe I can say something, maybe also Kenneth can correct me because he's way more experienced than myself. Now in my short experience with the hierarchical module naming scheme with Elmod, I noticed just advantages until now. In the sense from both the maintainers point of view and the user point of view, I can smoothly use the different modules, compile with different set of tools and compilers without mixing them. With updates, you have immediately, so suppose you want to maintain rolling updates of the CP versions, you can put on top of the current installation the new modules without interfering with the existing ones because they go to completely separate locations. So I find it pretty convenient both as a maintainer as a user. The only drawback that I could see is that you need to play a bit with the easy build prefix and easy build install path if you want to take advantage of the modules that are already present on the system in a path where you don't have right access. So suppose you want to build something on top of some modules that we already provided as CSE stuff. Then of course you need to set the prefix in the same way we do that. Otherwise in a hierarchical module naming scheme, easy build won't find it because of course it's hidden in some different module path. And if you for instance want to build something based on a module that is not immediately accessible to your module path, easy build won't find it and it will try to rebuild it if you are building in a local install path. So the prefix should point to the general prefix and the install path to your sandbox or your place where you want to build. In this case, easy build will see what it has been already installed and it will install on top of the existing modules. In a flat module naming scheme, this is easy. You don't have to worry. You can have prefix and install path set as you wish because anyway, your module path will be global. You will see all the modules anyway. Easy build will see everything. That's the only difference that I can see in terms of practical working with that. Yeah, so in my experience, it's a bit... It depends a lot on how proficient your users are and what kind of environment they are used to. If they're very used to a flat naming scheme, it's a bit of a jump to move to a module hierarchy. But in the end, I agree. Look, it's better because it's at least less easy to load things that are not compatible with each other. It complicates two things. In my view, like Luca mentioned, the stacking of software, so building stuff on top of a central set of modules is a bit more difficult, perhaps especially if you're also building your own compilers and basically have your own hierarchy on top of another hierarchy, then it can get a bit complicated. We're actually planning some improvements in Easy Build there to mitigate the problem a bit where Easy Build cannot see stuff that is installed in a central hierarchy because it uses a particular way of finding those modules. We have some things in mind that we can improve there. And another aspect is if people are very used to looking for stuff that modules avail, you really have to teach them about module spider and that there's another way of searching for modules that are not currently available for loading. The module spider is something that I pushed upon very hard yesterday when I gave it tutorial in Antwerp because we do often get questions to install software that is already on the system. So that's a bit re-educating the users, but other than that, and maybe, I guess for Lumi, that's a lot less relevant for more novice users. So I always use bioinformaticians as an example. If you have to first explain them what the compiler and MPI library is before they get to their software, they may be annoyed. But I guess for Lumi, that's less relevant. There is a question by Enric in the chat about building singularity containers in EasyBuild. Yeah, I don't know if Luca has experience with this. I guess it will be sourced rather than singularity. Yeah, so we have also singularity. So on PITS 9, we have provided singularity to allow users to run the Docker images. And we also have implemented an in-house software which is called SARTOS, that also it is a runtime container environment. Now, at the moment on the e-existent, on the e-existent, we don't have it yet. It's of course in plan. I don't know exactly the ETA for that. So when it will be available, maybe already next month, but I cannot really promise that. About the differences, building singularity or EasyBuilding, create environment. Well, I guess there it depends whether you want to integrate with what Grape provides or not. If you don't, then I guess there's not a lot of difference whether you're on the Grape system or not. But as soon as you're running MPI stuff, then it's very different. So that was mostly my question. If there are any differences when running an MPI environment and doing EasyBuild and constructing containers with EasyBuilders from? Well, so once, well, let's say you can build of course your image on your laptop with whatever tool. In general, we suggest Docker because it's compatible with the SARTOS implementation that we have on Pitsteint. You can also use it in Singularity. Both systems like SARTOS and Singularity, both modules, both let's say tools, they provide hooks to make sure that the MPI that you have used, that you have linked in your own image is mapped to the MPI library that you have runtime on the HPC system with Grape. So that is managed with the MPI, ABI compatibility. So there are some rules to be respected like, there are some MPI versions that you need to use in your own container in order to make sure that the hook is working correctly. But apart from that, it's not really, there are not so many differences. Okay, great. Those are all of you, thank you. No problem. Anyway, for that, we have also on the user portal, CSS user portal, some documentation, you might search for containers. There is a basic page that describes you how to do that. And also it points to more in-depth documentation for if you want to see more about the hooks for MPI and also the hook for NVIDIA stuff if you want to run on GPUs, of course. Then pinning CP modules. So I wanted to put this question here because it was also mentioned in your conference page. So pinning CP modules versions in tool chains or not. So this is a recurring question that we have started since the beginning because initially we did not pin version because Cray engineers were not suggesting to do that because the program meta modules of Cray, they load defaults. And then it is a concern for reproducibility, of course, if you load a program file, a program module file that is loaded at a time when the defaults have changed. Now, Cray provided also a solution to that with the CP module file, which is called CDT on the XC system. So this file ensures that if you load it, the defaults are loaded consistently. So having both the load in your easy config file for the tool chain, loading the program module plus the CP module, we make sure that the defaults are loaded consistently with respect to the version that you are selecting, making sure that also your build will be reproducible. However, you can, as I mentioned, override that in the sense you can decide that you want really to define all the dependencies, list all of them in your easy config for the custom tool chain with also the versions. In that case, you need to maintain your own, let's say configuration, maybe a YAML file with the versions for each P release, maybe extracting that from the module RC file itself provided by Cray. So parsing, some additional parsing would be needed. It's not dramatic. As I mentioned, we just found convenient to use what was already provided. Of course, this is sometimes error prone because as I mentioned, there was a problem with the CP module in the 2104, but of course the problem got fixed. So maybe you have to wait the next P release, 2105, and then you go on. In the meanwhile, you can provide a custom version of the CP module file to fix the possible error. In my opinion, it's better, it's not better, but it's easier. It's simpler to rely on the modules already provided. As I mentioned, the program meta modules and the CP modules, because it makes easier to automate things. But of course, for everything that I've mentioned, you can implement your own custom solution with your preferred way of doing that. I did find an issue actually with the Pro-GNV modules. They read, I mean, even though they have a particular version, they read a file which is on the system which may change over time and may cause that when you do a rebuild that there is a different set of modules loaded. The versions will not change, but you may add or remove modules from that configuration file which is somewhere in ETC. I don't know if it was somewhere in your slides even where that configuration file is. Yeah, yeah, it was in the beginning, I think, yeah. Yeah, exactly, because these are the configuration files that you can set for the initial login. You can change that. Yeah, okay, this is, yeah, it might be a problem. But this might also happen if you pin the module version, in the sense that you can also, you have to load anyway external modules. So the ones that you find at the login in your environment, generally are the ones that you want to load in a tool chain. So supposing that there is a problem also, also there are sometimes you have bug errors, so you might need some fixes and you might need to provide some custom thing as well. I mentioned the DSMM module recently, but it might happen with other modules as well. So yeah, of course, if you pin the versions, it's less error prone and more reproducible. That's for sure. More easily reproducible, sorry. And that's the last question, in fact, how to keep the easy build software stack reproducible. So has to do with it, of course. Yeah, as I mentioned in the slides, we have this regression project. So the Jenkins project to test the regressions in the builds. So it starts from scratch in a separate environment in order to make sure that all the recipes that are referring to a specific programming environment release are still working. That's also a way to make sure that once the system, the underlying system changes, no changes are affecting the build of the recipes because as you pointed out, it might happen that some module is actually looking at a configuration file that changes in the meanwhile. Yeah, so that's something. Then it depends also on the level of reproducibility that you're looking for, because our regression project, for instance, looks for sanity in the sense that the build can proceed and succeed as it did before. But then for the moment, we don't test if the results of running out the build software are the same. So this is something that we address separately with the reframe regression framework. And I guess one of the key messages here is if you wanna reproduce an installation exactly as it was done before, it's important to use the same easy build version as we were using before as well, because we do make changes in easy build versions, which are 99.9% backwards compatible. So we try very hard not to break things. And I think that, and Luca can correct me if this is not true, but I think that's also been the experience at CSCS that they barely ever have to deal with any breakage that occurs from jumping to a new easy build version. It's usually outside influence. Yeah, absolutely. I mean, we currently, we still update easy build installations very often. And we never had problems with reproducibility due to a new easy build version. The problems were usually due to changing underlying software system configuration. So yeah. But yeah, if it really as close as possible to reproduce a build, then you should use the same easy build version as well, because we do install additional things or two reconfiguration options because a problem popped up. It's mostly bug fixes that affect differences between installations between different easy build versions. But it's possible you may not get the exact same installation. Usually the installation you get with a newer version is better in some sense, but it may not be exactly the same. So that's definitely important to keep into account. The way I'm organizing Lumi at the moment I was thinking of fixing an easy build version for each version of the Cray PE. Yeah. There's many people that do this as well. Cray PE because then you have to test anything anyway. Yeah. There's people on, let's say, non-cray systems who do this as well, who somehow pin the easy build version to the installations they are doing. Like in Ulyk, they have this built stages, which is sort of, I guess, like Cray PEs. It's very close, at least the central idea is very close. And they only ever update easy build when they make a new stage. So they stay with the same version within a stage. So that's also something that you could do. Okay, are there more questions? Because otherwise I would wanna say a very big thank you to you, Kenneth, and to you, Luka, for all the work you've put into this. And especially to Luka, most of the contents here was done by Luka. Yeah, okay. You have put a lot of effort in the other three sessions. Yeah, it's most coming also from easy build documentation and also our documentation. So something is maybe redundant, but at least I hope it gave you an idea of how we manage things. And also in view of your, let's say, need to maintain maybe different software stacks for different architectures, maybe this experience can be useful. Sure. And maybe from my side to wrap up, if there's any questions related to easy build, the best place to ask is in the easy build Slack. And so there's lots of people there who can probably help you.