 Okay. So I think we can maybe walk through the example together. That's maybe still helpful. Who managed to finish the example? So installing EB tutorial 101, having the full easy config file. One, two, three, four. Yeah. Okay. Is it useful to still walk through it and point out some things? Okay. Let's do that. So let me make sure my environment is set up correctly. Will parter slash temp prefix is set to home easy builds, trace enabled, only easy but loaded. Yeah. That looks okay. So I'll start with the minimal stuff like this. And if we try installing that, easy build will not get very far. So this is, I think, a pretty meaningful error. There's no software specific easy block for EB tutorial. You can tell it's looking for a particular, in this case, it's a Python class. So an easy block provides a Python class, and this is the name of the class it's expecting. If there's funny characters in the software name like a dash, it will translate this to underscore minus underscore, because there cannot be a dash in a Python class name. There are examples of software that have plus signs or pipes or what in software name. So there's an encoding mechanism for all of this. So that's our first problem. There's no easy block specifically for this software. That's not unexpected. So we'll need to figure out how to pick us a generic easy block that could work. So the starting point for that is to go and look at the sources of the software. So there's a link here in, at the top of the example, this tells you where to download the source star ball and it's sitting in a GitHub repository. So we can just check that and see what's in there. It has a make file that's good news. It's telling us to do pseudo make install. That's not so good news. We won't be doing that. But this does tell us it's a CMake based installation. There's a CMakeList.dxt, which is pretty standard for software installing with CMake. So that looks like a good hint. Okay. And like I mentioned, in the easy block part here in this section, there's a link into the Easy Build documentation. So this is a list of the generic easy blocks available in the latest version of Easy Build. If you don't like the light team, you can go for the dark team as well. I like this one better. And so this is a long list of course. Now, I guess the clue here is to look for stuff that seems to be using CMake. There's two, three, four options here. The CMake make one looks like it's the correct one. So CMake, make, make install is also make. So this looks like it could be a good match. The other CMake ones, so CMake make CP. So this says CMake for configure, make for build and then just copying files because there's no make install target. That does happen for stuff using CMake. CMake Ninja, so Ninja is an entirely different build tool, something else than make. And there's even some Python packages that you CMake for configuring, which is a bit strange, but it does happen. So the CMake make easy block is the one that seems like the best fit. This has a whole bunch of specific easy config parameters that only really make sense when using the CMake make easy block. There's a long list here. We won't actually need to use any of these. And there is something. Yes, the recording is running. Yeah, no problem, thanks for checking. There is actually a hint here in the readme file that says MKD build, CD build, and then CMake, dot, dot, what the hell is this about? Well, when you're doing a CMake based installation, CMake very much wants you to do that in a clean build directory. So that's the hint here. Now, easy build is smart enough to already do that by default. So you can check the custom easy config parameters and here you'll see separate build there. So perform the build in a separate directory. This is set to true. So easy build does this by default. The CMake make easy block does that by default. So you won't actually need to do anything special here to cover this part of the installation procedure. The CMake make easy block will also do make. It will actually do make minus J number of cores. That's a bit better than the regular make, even though here there's only one C++ file. And then it will do a make install into the correct installation directory because when it runs CMake, it also tells it then to configure, to be installed in a particular installation directory. All right, so let's take a look what happens when we edit the easy config file and add the easy block. Just out of habit, I'll be adding these things in the order that we usually use for easy config parameters, but the order doesn't really matter. If you put this line at the very end, it will still work in exactly the same way. So this is more out of habit than anything else. And to make it follow the same structure that's usual. I'll go ahead and add the module class as well. This is actually an optional one, but it's usually there. And if it's there, it's at the bottom of the easy config file. All right, so let's try this again and see how it screams at us now. So it looks like it's getting somewhere. It looks like it's fetching the files, the source star ball. At least that's what it's suggesting here. It's preparing the build environment by loading the tool chain module. It's trying to do the configuration and then it blows up because CMake is not available. So it's a CMake based installation. The CMake make is your block will try to run CMake and it's expecting that to work. So that's not there. Why not? If we check in the OS, there's no CMake here. So how do we fix this? We add CMake as a build dependency. So we need CMake to build the software. So we can tell easy build, build dependencies equals. So this is a double with the name of the build dependency and the version goes here, like this. Now, which version do we use here? Okay, well, let's check. Do we have any modules for CMake? We do, we have three. Which one can we use? Well, the one that matches the tool chain that we're using here, GCC 11.2. This looks like it matches. It is GCC core, not GCC. So that's a bit difficult to explain, but GCC core is what we call the base tool chain of all the base compiler only tool chain of all our tool chains. GCC is very similar to that with one small difference. You can do a module show on GCC and you will see this loads GCC core. It also loads binutils. So binutils are extra tools to create binaries, libraries and all that stuff. And these are not loaded when loading GCC core. The reason it's structured like this is because we want to be able to put GCC right next to Intel compilers or C-Lang or AMD compilers, but then on the same level essentially. And this will come back when we talk about hierarchical model naming schemes, how this is structured. You will see this coming up there as well. So the version has to match here. So that means we have two choices and let's just try the most recent one and assume that's gonna be better than the older version which it usually is. So that's the version we put in there. All right, so now we can check with dash, dash missing or just dash M. It should be happy because the module is there and it is, it's only missing EB tutorial itself. You can see there's already 24 modules required here. So that's for CMake, that's for GCC Toolchain and all of its dependency. So there's a big chunk of things already pre-installed. If we try to run this again and this may have been a difficult one to figure out for some people. It's loading CMake as a build dependency. So now the CMake step should definitely work, right? It didn't. So what did it do? It tried to run CMake like this. It adds a couple of options to CMake like where we want the installation to actually end up. It does a build type release. There's also a build type debug possible but release is a decent default. It's asking CMake to generate a verbose make file. That means when make is being run that it will show you the compiler commands being run. So by default, they hide that. I don't know why, but I'm annoyed by this. And I forgot what this part is about, but it's important. Otherwise it wouldn't be there. So this failed and this failed pretty rapidly. And here we see part of the error message but not the full one. So we'll have to dive into the log file and scroll through to the end. And then here start scrolling up again to see the full error here. So this says CMake was inspecting the source directory and there's no CMake list.txt, which is weird because this one says there should be one. Okay, this is our build directory that EasyBuild has set up. So why is it not there? Let's inspect it. There's this EasyBuild object directory. That's the build, the empty build directory that EasyBuild creates to do the actual building. But also there, this is empty and this only has that sub directory. So what's going on? What did we overlook? Let me show you the easy config file again. Yeah, there's no sources, right? So sources is, it's a bit weird, but sources is optional because you can have easy config files that just glue things together and then there's no sources. Okay, so we need to tell EasyBuild, let me go back here so I can copy some stuff and not make silly typos. This is the source star ball that is needed. So we'll do sources equals a list of things. So square brackets, the name of the source star ball, I will have to put quotes, single or double around them to make it a string, a Python string. So the easy config file is Python syntax. That's good, so that looks correct. Now there's one thing here I don't like which is we have a version here and we have the same version here. Remember that EB try software version option? When you use this EasyBuild will only change this line. So if you have it hard coded here it will not change this part. So we need to try and get rid of this. There's a trick for this. EasyBuild has what's called template values. So this whole thing has a template value. It basically says just plug in the version as soon as you know what the version is and use that to complete the file. That's good. There's actually, this is a standard pattern, software name, dash version, that dot, dot, gz. This happens all the time. So we even have a constant for this. So this is exactly the same as this one. It's actually this, exactly the same as this. So a template for the name, a template for the version dot, dot, gz. And this looks a bit easier on the eyes and less room for mistakes. Like if you make a typo here, EasyBuild is gonna scream at you. This constant doesn't exist. So less room for mistakes. That's one part. The sources have to be there. EasyBuild doesn't know where to download the sources from. We saw that in the troubleshooting as well. We better tell it. Source URLs is how we tell it to download stuff. In the exercise, this gives you the full path to the tar ball. So we can put that in here in a string, but the file name itself should not be part of the URL. EasyBuild just glues this after this when it does the download. Okay. So let's try again. Now EasyBuild will first download that small source tar ball unpack it and then configure, then CMake is still not happy. So what the hell is going on? How do people even use CMake if it never works? No, in this case, it's actually looking into the unpacked sources. It's picking up the CMake list of TXT. And this says, there's an EB tutorial option that must be set. So this is hard required for this software. Okay, fine. So we'll make it happy. We'll set it. How do we set it? To set a CMake option, we need to do it minus D. This equals a string. And that should be added to the CMake command. So to the configure command being run by the CMake, make easy block. We can do that through the config parameter. So we start with minus D because it's CMake, the name of the option to set. And then here we add a random string to make it happy. So the EB tutorial tool just requires us to give me a string that I use in the binary to say hello. All right, this looks good. Now let me show you a funny or maybe useful thing. If you make typos in here, easy build will of course refuse to continue because it doesn't know what this is, but it's actually giving you the suggestion as well. So it has typo detection. As long as it's close enough, it will tell you why you probably meant this. And it's not gonna auto-correct it for you, but it is gonna make a helpful suggestion. So fix the typo, try again. And now at least CMake is happy and we're almost done. Again, we're failing in the sanity check. So what's going on here? Here it says I found a non-empty bin directory, that's okay. And I did not find a non-empty lib or lib64 directory that failed. So this is the default sanity check that easy book performs. If it's not told anything else, it expects to find a non-empty bin directory and a lib or lib64 directory that's not empty. In this case, when we check our installation directory, so we get the pod here to the location where stuff is being installed, we will only find a bin in there. It has an EB tutorial binary. So we'll need to customize the sanity check. We do sanity check parts. So that's the files and directories, but these we'll look for. Of course, we'll look for EB tutorial as a binary and there's nothing else in there. So in theirs, which has to be there, I can actually show this as well. So let's say we only want to specify a file and there's no directories in there, other than bin, which has a file. So there's no point in checking for bin. Easy book won't be happy. It says, ah, the mandatory DIRS key is not there. So, okay, fine. If you're being picky, we'll add DIRS there and we could add bin, but there's no point. This has a non-empty bin directory and this has a specific binary in that directory. So which is way more strict. So there's no point in doing both checks. We can just make this an empty list. And on top of that, what's good practice is to not only check for the existence of files, but also try to run a command that makes sense. And in this case, there's no help or version. And this is a tool that just says hello. So that's quick enough to run it as a sanity check. So that specifies the sanity check. And again, here to complete the installation, we can either redo the whole thing from scratch or because CMake takes forever, we can do a module only to tell easy build. The installation is already there. I just give you a better sanity check and now you should be happy and it is. It completes the whole installation. Any questions on this? Jörg, yes. Yeah, we should check that Mike again, and we'll have to unmute. Oh, no. Are you still in that Zoom meeting? Yes, Jörg. The bit ice target is where to get that tarpaul from. Because I've noticed Git has a commit and I try to do down it very complicated with the specific commit which was on the webpage, if you see what I mean. But that's me and my constants struggle with GitHub, if you see what I mean. Yeah. So this is a bit weird and some silly scientists wrote this, but they planted the tarpauls straight in the repository itself, which is strange, but that's where the tarpaul is. It was actually, it was given away in the exercise as well. So it's a bit, that's very atypical. Yes, you typically have like a tagged version and then download that as a, that's true. Maybe we should change that, but yeah. And sometimes, yeah, the software makes you go through a maze, right? Just to find the sources already, it does happen. Okay. There's a second, you have a question? Yes. Yeah, I just want to ask about licensed software. If you have any license, you know, software behind the file with behind the... License server or... Use the name of the password that you need. Ah, for the download, do you mean? Yeah, if you need to, yeah, for download or sometimes software, there's some even more complex ways to access the sources or... Yeah. So what you can always do, what you can always do is download it manually. So if you check how EasyBuild is configured here, this is the source path. If EasyBuild can, it will download the sources too there if they're missing. So this should be a directory with write permissions. If it's too difficult to let EasyBuild download it, you can download it manually and just move it in here. So the sources, well, they'll have a good example, but maybe tutorial. You could just download the tarball manually, move it in here, and if EasyBuild can find it, it organizes source tarballs by software package, where you can even drop them directly in here. It will still be happy. You can also copy it next to the EasyConfig file. That's another location it considers. So it tries to locate some, yeah, it considers some locations and then it will be happy. So then it will not try to download. The download thing is like a best practice, oh, not best practice, best effort. If it can't find it, it will try to download it. If not, you can still see it in and that's okay. I mean, we need to do this for some license software. Okay, is there anyone who got started on the exercises as well? Yeah, some people did, not everyone. The first one is a bit sneaky. So this tells you change your EasyConfig or EB tutorial and customize the message that is printed to include the username of the account that was used to install the software. So you can use dollar user for this, but you'll have to be a bit careful. And let me show you why that's the case. So if we change this to say add dollar user in here, this will not work. It will just have dollar user hard-coded in the message because this is in single quotes, right? So this will, let me edit it a bit here. So this will essentially run a command like this and then this will never be resolved because it's in single quotes. So what you actually want is that this is done in double quotes. Because then in the shell, dollar user will be replaced. So that means we need to use double quotes here and then we need to use outer single quotes. So for two easy build itself, it doesn't matter single or double, but it does matter to how the command will be run and how the shell will resolve environment variables that are in there. So that's the tricky part single double. You have to be sometimes careful. And sometimes you have situations where you're including options that both need single and double and then you don't have anything for the outer quotes, then you can use triple quotes as outer quotes, which is also valid and white. So that's exercise one, essentially. Exercise two is then, you know what? There's a newer version of EB tutorial that you can install. Here are the sources for that. Let's get an easy config for this. And as a strong hint here, you should not make any manual changes. So how would we do this? Try software version is one. Yeah, why we can do try version, see what happens. No, it's try software. So here it's, it usually gives you a hint as well, but it's not close enough, right? There's too much of a gap. But with try software version, easy build will start with generating an easy config file based on the one that we have for 101. Give it version 110. The location of the sources agrees enough that it will use the correct download URL and it will actually complete the whole thing without ever copying or editing the easy config file. So that's useful, if you're lucky. And then the bigger exercise here, I think that's the last one in this part, yeah. This is creating an easy config file for a PI EB tutorial, another software package essentially with this Python bindings on top of EB tutorial. So that corresponds to this. So in here we have a setup.py, which usually is a good hint. And this has a package, which has a singular init.py and something very small, which does an import numpy. Oh, where is this gonna come from? And this does a sub-process.call to a binary. So it looks like this has a bunch of dependencies. I won't do this one step by step in the interest of time. But there is a generic package for Python package, generic easy block, sorry, for installing Python packages, which is called Python package, which is used actually quite often. It's one of the most used generic easy blocks probably, which has a bunch of options as well, quite a lot. Some stuff related to PIP. There's actually three key easy config parameters in here. One of them is you, well, I'll mention the two most important ones. One of them is usePIP. So this just tells easy build usePIP to do the installation, don't do Python setable by install because that's actually even hard to discourage currently. So if you do usePIP equals through, it's gonna do PIP install dash dash prefix and then some options after that. So that's an important one. The other one is sanityPIP check. So this tells easy build as a part of the sanity check, please run this command, PIP check essentially. And this makes sure that all the dependencies for that Python package are available, at least if they're properly declared in the setup.py. So if you go back to the example here, the setup.py probably says, numpy is a required dependency. If you don't include numpy, PIP check would fail and say, hey, numpy is needed and it's not there. So that's good practice to tell easy build to run that as well. And that's like an extra safety mechanism to make sure everything will be there. This part is a bit annoying as well. We're gonna have a cleaner way of specifying that, but for Python packages specifically, easy build always doesn't import check. So it does a Python minus C import name of software. And sometimes the name of the software, like in this case, PyEB tutorial, doesn't correspond to the name of the Python package itself. In this case, it's EB tutorial. So you should be able to customize that. To customize that, you currently have to do something like this, where example is the name that should be used in the import check or import example. And all of these things are a couple of hints to compose this PyEB tutorial. Easy config. To make it very realistic, there's zero documentation. There's no readme file. It's just go figure it out yourself. That does happen. Yeah. And the specification here is also make sure that it actually works. So that's the PyEB tutorial commands that should be there is also running correctly. The reason that's part of the exercise is even though in here it clearly specifies you need NumPy as a Python package, if you actually look into the code, it will do that import NumPy. So this will fail also if it's not there, but it also does a call to an EB tutorial binary. So if you're not adding the EB tutorial from the first exercise as a dependency, this part will fail. And you will only notice that when you try to run the PyEB tutorial command. So it's always good to add sanity check commands and at least have some confidence that everything is there and sort of working. And this is the whole story here. Text is a bit small, but it starts with Python package as an easy block. It starts with name and version. In this case, we're also adding a version suffix that says this is PyEB tutorial for Python version this. So this is another template here. It's gonna drop in that Python version in the label that is used to tag the module. Homepage and description is pretty standard. Source URL is no surprises here. We have the check, we usually add checksums as well in the easy config files to make sure that the download of the sources worked. You're not getting corrupt tarpaul or a malicious tarpaul like a man in the middle attack. And there were people here that were not using the raw download URL and then you get an HTML page from GitHub with the wrong checksum. And if you include the checksum, easily will tell you something's wrong. Dependency is here. So Python is definitely needed. We're including the scipy bundle because this one provides numpy, which is a required dependency. And the eB tutorial, which provides the eB tutorial command. Use pip is set to true. We don't even have sanity pip check here. It should be there, but it's not. We need to tweak this to tweak the import check. Otherwise, easy build will try importing this, but probably with underscores, it's smart enough to know this can never be correct. But even with underscores, it's gonna fail because it doesn't have the pie parts. So that's definitely needed. We're checking for that binary. We're checking for a non-empty site packages directory in the installation. I were running this as a sanity check command which should work. So that's a bit of a puzzle to put together. Python packages are very common nowadays and they're getting more and more common. So that's definitely a good example to work through as well. Yeah, and then it goes on and explains all the details that I just did. Okay, any questions on this before we move on to the last part? So the last bit talks about module naming schemes. Let me flip back to the slides. Because at least in here there's some stuff to point to or some condensed version. So the module naming scheme used by EasyBuild determines what the module name will look like and how the module names or the module files are organized in the module tree that EasyBuild generates. There's a couple of naming schemes included with EasyBuild. The one we've been using up until now is called EasyBuild Module Naming Scheme, the default, which closely follows the easy config files. So it does name slash version tool chain, version suffix. So it looks very similar to the name of the easy config files. So there's a clear mapping and everything is available all at once. And like Stefano mentioned, yeah, sometimes there's too many modules and gets a bit messy. You also get all modules even when they're installed with different tool chains. So you actually get things that you can load together which are not compatible with each other. You may be hitting problems when doing that. So maybe there's better ways to organize the modules. One alternative way to organize your modules is what's called the hierarchical module naming scheme where your modules are organized in a tree with a couple of levels. So you have a core level where you typically only have compilers like GCC, the one we've been using. The second level is then modules that are installed with this GCC compiler. And the third level is modules that are installed with one of these MPI libraries. And what happens is if you load this module only after loading this module, these modules will become available. And after loading the open MPI module, these orders will become available. So modular veil is a lot smaller as output initially and only when loading a compiler you will see additional stuff becoming available. And that way it lowers the chance that you're loading things together that are not compatible with each other. The modular veil output is less overwhelming. The module names are a lot shorter because you don't need to specify in here that this was installed with GCC, you already know because that's where it came from. Same here, so things are almost software slash version and that's it. So it's a lot cleaner in terms of module names. Yeah, and they have gateways, what I call gateway modules are GCC that opens up another bag of modules, an MPI that opens up another bag of modules and so on. Let's see what else we have in the slides. Yeah, so comparison between the flat module naming scheme that we've been using up until now and the hierarchical one. So with the flat one, all modules are visible. That's good because you can see them. That's bad because you can see everything, right? And it's a bit overwhelming and a bit more difficult to search. There is a one-to-one mapping with easy config file names that sort of guarantees that the module names will be unique. So you will never get an accidental clash between two easy config files that map to the same module name and the hierarchical module name. Name scheme, you need to be a bit more careful with that. The log module names are just ugly. You see this thing popping up all the time and people don't know what this means. And if you're loading two modules with the built with a different tool chain, you may be causing trouble. The hierarchical module name scheme is shorter module names, good. The tool chains are actually not really visible in there, at least not in the module names. They are hidden in the structure of how the modules are organized. The modules are not all visible at the same time. That's good to some extent. That's bad because they're a bit more difficult to find. Even though with L-MOD, you have next to module avail, you also have a module spider command, which allows you to walk the whole set of modules and, for example, if you're looking for an HDF5 with module avail, you will not see it unless you have these two levels already loaded. With module spider, you will see it and module spider will tell you, oh, if you want this one, you'll have to load this and this first. So it's actually aware of the module hierarchy. L-MOD is also smart that if you're, if you have a GCC version loaded and you load something else, it's gonna try and swap all the modules that correspond with each other. So say you have OpenMPI available 4.1.1, once with GCC 11 and once with GCC 12. You load GCC 11, you load this MPI, and then you change your mind and say, oh, wait, I should be using GCC 12. You swap the GCC. L-MOD will automatically reload the OpenMPI as well that was built with that GCC. So it is smart in that way. Yeah, so especially with L-MOD, this is a pretty powerful mechanism. The downside at least I see is that these gateway modules like this GCC and this OpenMPI, if you tell a bioinformatician, you need to load the compiler first before you see your bioinformatic software, you're gonna say, why do I need to care about the compiler? That's not what I worry about. So it creates maybe a bigger distance or a bit more, need to be a bit more aware of what's going on. Okay, so that's hierarchical. I actually go through an exercise that really shows this step by step. There is a hierarchical module naming scheme included with EasyBuild, which works quite well and which took us a while to get correct. So it's not that trivial to implement this. But if you want to have your own module naming scheme because you think you have a way better way of organizing modules or you don't like all the casing going on in the default naming scheme, like if you want everything lowercase or if you never want to mention Python in the module name or God knows what, you can. Usually you can implement your own module naming scheme, give it to EasyBuild and EasyBuild will just generate the modules or at least the names of the modules based on how you like. There's an example or an exercise in the tutorial on this. If you want to do this, you'll have to do an EasyBuild, include module naming schemes and give it a list of module naming schemes you want to make EasyBuild aware of and then you should actively change the active module naming scheme to the one you want to use as well. So this is another example of a plugin where you can say, hey, I have this module naming scheme. It's not a part of EasyBuild, but you can give it to EasyBuild and it will use it when it needs to. And yeah, this is I think part of the example to use DryRend to test your module naming scheme. If you do minus capital D, it also mentions the name of the modules. So you can actually see what things will look like without actually going through all the installations and generating all the modules for us. Yeah, and this is the hands-on example. I'll maybe do this as a demo myself because otherwise it may take a bit of time to work through it. There's a couple of important details here. We'll have to be very careful how we set up our environment. We'll actually need to actively hide all the modules that we currently are seeing because otherwise EasyBuild will get very confused. You cannot mix a flat module naming scheme with a hierarchical module naming scheme because sometimes module names do agree, like for GCC, for example, GCC 1120, we have also in the flat one, we have it in the hierarchical one, but it's not the same module file. So in the hierarchical one, it's a gateway module and the flat one, it's not. You need to be very careful which one you pick up. So it's better to hide all the ones that we currently have. What we're actually gonna do is we're gonna tell EasyBuild to generate new module files, but without redoing the installations. The installations are exactly the same and are totally separate from how the modules are organized. The modules are just a view on the software and are totally separate from the modules. So what we're doing here is we're telling EasyBuild, we have software installed in here and we don't wanna reinstall that. We want to use the hierarchical module naming scheme and we want to install the modules not next to the software, but in a totally separate directory. And when you do this, you configure EasyBuild like this, you run this command, so you say, let's install HDF5 with all dependencies and we're only gonna generate modules because the software is already there. So EasyBuild will generate the modules one by one, do the sanity check for all those modules and it should all pass because the software is there. And then you can start playing with the hierarchy with the module use on, not modules all, but on the core subdirectory of modules all, which gives you access to this first gateway module for GCC. And then you can work your way down the tree like this. So let's show this hands-on because I think that really helps in understanding what's going on. Let's see if I can get this right. Follow my own instructions, that's probably a good idea. So step one is hiding everything we have. So a module with ALS empty. Now we do have a problem, we don't have EasyBuild anymore. EasyBuild was installed as a module and it was unloaded. So Elmod says inactive module, it's no longer there. The easiest way to get around this and that's also what is instructed in this particular part of the tutorial is to just do a pip install of EasyBuild and do this. There is a way to get first EasyBuilds in place in a module hierarchy, pick that up and then install a hierarchy like this, but that's a bit more complex. It's definitely possible. So the easier work around here is just to do a local EasyBuild installation, which will take a while because this is NFS, but I'll let it work its way through that. Then we'll have to copy paste from the website here. It's gonna be better. So this is all explained step by step in here as well. Like it says, we have to, we make very sure we're not mixing modules. That's gonna cause trouble. Unload everything, do an unused. And then here do a pip install. And let's hope this doesn't take three hours. It shouldn't. Once the pip install completes, we need to make sure that EB command is found. And let's also make sure it's using Python 3. So we'll need these two changes to the environment. So now it's collecting and installing all those 16,000 easy config files on NFS shares. So that's why it takes a while. Let's hope it doesn't take too long. This actually gives us a bit more stuff. And just make sure that everything is fully correct. Yeah, this is maybe also an interesting one to mention. So whenever Easy Build does an installation, it creates a log file in the installation and the software location as well and the installation directories as well. That's to prevent that two things running at the same time are both installing the same installation and like partially overwrite each other and you end up with something that's entirely broken. And in this case, we need to change it because when Easy Build generates the modules for the hierarchy, it also wants to create those log files. And it cannot create the log files in slash Easy Build because we don't have right permissions there as a regular user. Okay, so this is progressing. It should be almost done. Let's check here whether that's actually going. So that's completed. Let's make sure we can find the CD commands. That looks good. Let's make sure it runs. That looks good. We're gonna configure Easy Build with this. And let's do a show config on that to make it clear. So we're using a hierarchical module naming scheme. We're using the existing software. So we're not gonna be installing any software at all. We're only gonna be generating modules. We'll generate those modules in the HMNS subdirectory of our home directory and the rest doesn't really matter. The log files, we need to be a bit careful as well. Good, that looks okay. Show config, this explains what's going on. And now here we can do this robot and module only for HDF5 to generate all these modules. So this is gonna take a little while because it needs to generate modules. Okay, let's ignore that OS dependency. It's not really relevant. That's an issue in the setup. So just to explain what's going on here, it Easy Build says to install UCX, we need an OS dependency. We need the libIB verbs. So let's say the Infiniband drivers or the Infiniband libraries on the OS to be able to exploit Infiniband. But it's not really relevant here. So we'll ignore that and we actually have UCX already installed. So it doesn't matter. This will start generating modules for 42 things or so. I think it's mentioned in here. Yeah. So it's gonna start from scratch with, so it's also actually gonna do the build dependencies in this case, which is a bit silly, but that doesn't matter. But you can see it's really quickly, right? It's skipping through all of the steps, essentially. It's only doing the sanity check for all the installations. And if those are okay, it's actually generating the module. This is the GCC one and then it just continues like this. So it's cycling through these one by one and making sure the installations are actually complete. So it's not only checking whether the files and directories are in place in those installations. It's also loading the module it generated in the hierarchy. And making sure that works and that it can find its dependencies and everything works out correctly. So let it zip through that. And then this already describes, so in the end we'll have an HDF5 module, but the organization of the modules is gonna look very different. So we'll end up with something like this. We'll have a core, a core, so that's the top of the hierarchy, a compiler level that's the second level and then an MPI level. So core means it's built with the system compiler. Compiler means it's built with a particular compiler that we control and MPI means it's built with an MPI that sits on top of a compiler that we control. So that's the three typical levels. You can organize the hierarchy in other levels as well. You can imagine, maybe you can add a level for the Python version or add a level for the Blast library that you're using and basically make the hierarchy deeper that definitely works. So it doesn't need to stop at three levels. So here is where I should have installed the rich thing. Maybe I can show this if I reinstall rich because then the output will be a little bit nicer. Now it's gonna not redo everything, of course. Some of these modules are already there, so it's gonna see those and just use those. But now we see, now it's actually telling us, I have 19 easy config files that I still need to generate modules for. It should be scrolling like this. You should have a single line. I'm not sure why it's doing that. Yeah, okay. That's not gonna work. So I'll just let it zip through. But at least that way you have some idea of how many things it still needs to install. So when the rich Python package is available, it's giving you some more progress information. And it will also tell you when these things are done, can I get it to behave? Yeah, so now it's gonna not do the stupid thing anymore. I think this is what you should be seeing, that it does a counter and it gives you some kind of progress bar. And it's sanity checking Perl which takes a while because it's gonna not only sanity check Perl itself, but all the Perl modules we include with Perl. So it's gonna import the load check on all of these things, which is, I don't know, a couple of hundred or something we include. It's making very sure the installation is correct. And you can already see the different structure that it's using. So it's gonna install Perl. This is the actual module name. It's gonna put that in a GCC core directory which sits in compiler, which is the second level of the hierarchy. So you can see the structure is very different in terms of modules. This is also something we can improve. It's doing this sequentially one by one, right? We have multiple cores. It's not using them at all. We have done some work on that, but it's not entirely finished. And it could show it here as well. So once it's done generating all the modules, the plan is to do a module use on the top of the hierarchy on the core directory. If you then do a module avail or get an output like this. So very short module names. And some of these are gateway modules like this one. If you do a module avail HDF5, even though you know it's installed, we'll just say, this is not there, but it gives you a hint on module spider. If you do module spider, it will say, ah, I found an HDF5-112, but you'll need to load these two first before you can actually access it. If you start doing that, you load the first gateway module, you do module avail. You'll see you have those modules that were there before. You'll have your open API module as well, which sits in here. And in this case, it was actually, the second level is split across two directories. There's a whole bunch of things here, which were built with GCC core, not with GCC. You get all of these as well. And this is our second gateway module. When we load this, we do a module avail, that's when HDF5 pops up. So you get to load HDF5 with a very short module name. And if you do module list, it all looks nice and clean. There's no long module names and it does work as expected. This will still take a while and let it go. I think it gives you a good idea of what's going on. And being, so this does tell you that when you already have a software stack and you don't like the view on that software stack, on the side, you can start playing with a different module tree and see how that's working out, without having to redo all those installations. So easily splits these things quite well. Also in playing with hidden stuff, you can do something similar to this. There's an exercise on this as well basically go through the same thing for sci-pi bundle and then see how that's structured in the module tree as well. Any questions on this or anything else that we've seen today? So this easy config file that we have to write every time we do the installation for a new software or is it? So every time Easy Build does an installation that needs to be an easy config file. Yeah, so we have to write it down again like every day by ourselves? Well, it depends. So the Easy Build comes with a big collection of easy config files, right? So that's the first step. Doing eb-dash-search to see if there's something there is the first step. The second step before you start writing your own could be to look on GitHub. And that's actually something I will briefly cover tomorrow because in here there's lots of open pull requests for software that is not merged yet. So this is here, this is one example. Like Easy Build Sec is not in the latest Easy Build release but someone has worked on an easy config file for this and it's in the process of adding it to Easy Build. So if we just in here you say Easy Build Sec or let's see, I need to install this. I don't wanna start from scratch. There's an open issue here. It's actually an open pull request. You can click this and here in files changed. You can see what this easy config file looks like. So that could be a starting point if this is what you need and it goes a little bit further. Yeah, meanwhile this is done. So this is in a pull request which has a number 17791. You can tell Easy Build, let's try and install this ourselves. I wanna pull this easy config file from the pull request and let's see what's missing. So here I'm actually not giving any easy config file names and just telling it pull everything that's included in this pull request and let's try and see what happens if I try to install this. And so here it says only Easy Sec is missing. All the dependencies that it requires you already have. So that could be a good thing to do first rather than swearing and saying, I need to install this and figure this out myself. So for this Easy Sec file that depends on a specific software or something. Yeah, so here if you wanna, you can inspect the easy config file, right? You can see what's going on here. So this doesn't have any dependencies at all. This is installed with a system tool chain because it's basically a pre-compiled binary. We just download the binary. Yeah, this is using the binary easy block which says I'm not gonna extract the sources. I'm just gonna copy the file to the installation directory and I'm done. It's maybe not a very good example. Let's see if we can find something else. That's recent. Yeah, so you can see there's 800 open pull requests. About a hundred of those are from York. So York is helping out a lot with adding new software all the time. This is another example clip. So clip looks like this is a Python package. And if we do a dash, dash missing on this, it's a very recent tool chain. So that means everything will be missing but it does give you some idea of the dependencies that are required for this. It takes a while to figure out the whole dependency tree and it's actually using the hierarchical module naming scheme as well. So because that easy but it's configured, you see things are a bit in a weird organization. But this tells you if you want to install this, I'll need to install 104 things out of 111 required because it's a newer tool chain. But you have something to start from, right? It's not finished. So because the pull request is still open, either nobody has looked at it yet or there's an issue that needs to be fixed but you have something to start from and you can pull it in and start playing with this yourself. So this needs Python, this needs Kura, this needs PyTorch, and includes two extensions. So it gives you some idea on how to install this thing. The exercises we did today, morning. So those we didn't require to write the easy config files, right? So I was just curious, like, if we are going to install some new softwares, do we need to do the things we did today afternoon or not? Yeah, yeah. So you need to tell EasyBuild, if it doesn't know yet, someone has to explain basically to it how to install it. And in the beginning, that can be a bit daunting. If it's a standard Python package, so this looks relatively standard. The only non-standard thing is it's not on PyPy so we're pulling this from GitHub and it needs another Python package that we don't have in our Python. So it's always a bit of a puzzle and it can be a bit of work, but yeah, someone has to do that work first, unfortunately. A very good place to ask for help on this is the Slack, yeah. So we have a Slack which has, it's hidden behind this guy, 735 people there. So yeah, and I think it's a very helpful place, even if you're quite new to EasyBuild, there's usually people there that can help you out. Okay, any other questions? So tomorrow, the idea is to go through some of these advanced topics. I will actually let you pick tomorrow what you want to see first. It's going to be up to you. I won't do this in order. We'll figure it out together what you want to see. If you want me to show something else, that's not in this list. I'm also happy to do that. I won't have slides on that, but I can do a hands-on demo if there's some particular aspect of EasyBuild you know about and you want to learn more about. That's definitely welcome. And then in the afternoon, we'll take a look at Easy, which is sort of the next step in working together as a community, not working together on a tool like EasyBuild, working together on the software stack itself and building it up in a way that it works on many, many systems. Okay, if there's no questions, we can wrap up here and we can continue tomorrow. Sorry? Yes, yeah. I'll stop the recording first before we talk about dinner.