 All right, so let's continue. Did people manage to run some of these commands on missing dependencies and that's all looking good? So you're getting essentially this answer, only three things missing. So if you do end up having to rejoin your SSH session or restart your session, don't forget you'll need to do some things to set things up correctly again. Like the module use command, you may have to do that again, or making sure that the easy build module is loaded, you may have to do that again. If you log out and log back in again, you have a fresh session, so take that into account. Okay. So we'll continue with the more serious stuff, which is installing software with EasyBuild. This is what it's all about really. So when installing something with EasyBuild, you use the EB command, you give it the name of an easy config file, and you hit enter and that should work. At least that will work as long as the dependencies are available. If they are not, EasyBuild will scream at you, and you'll have to use the robot option or the dash R option to enable the dependency resolution, and then it will first install the missing stuff before it continues with the installation that you actually want. As we've shown before, it will show you building, installing, so the different steps it's running through, but without any details on what's actually going on in the background, so that can be a bit annoying. Like if it's setting their building for a while, you don't know what it's doing. That doesn't really tell you much. You can improve that by enabling trace mode with dash dash trace. We may actually start enabling that by default in EasyBuild 5 because it's just more informative and it at least tells you what's going on. If the software is already there, that means if EasyBuild sees the module, regardless of where the module is located, it will just tell you it's already there, I won't do anything, and you can make it rebuild something using the dash dash rebuild option or the dash dash force option. There's a small difference between these two, but that's not important. When you do an installation with EasyBuild, it walks through a series of steps, which essentially starts with parsing the easy config files of figuring out what it has to do, which software reach version, which tool chain, all of that, it will fetch the sources, either find the sources that are already downloaded or download them if they are missing. We'll then go ahead and unpack those sources, apply any patch files if there are any prepared to build environment. That means loading the modules for the tool chain and the dependencies, setting the environment variables like Cflags and CCs, so all of these that you can inspect via the dash x option. And then it will start doing the actual installation procedure, configure, build. If there is a test defined, also a test, running test suite, for example, and then doing the actual installation. If there are any extensions specified, it will install the extensions one by one after doing the main installation. Once that is done, it will do the sanity check, which typically means looking for specific files that should be there in the installation, a binary, a library, maybe running some sanity check commands like a dash dash help of the main binary to make sure the binary actually runs and can find all necessary libraries. If that all looks good, it will start cleaning up after itself. That usually means removing the build directory to make sure it doesn't stay behind. Generating the easy build module file is the next step. Maybe we'll fiddle a bit with permissions. That depends on how you've configured easy build. You can, for example, configure easy builds to make the installation directories read-only for all users, because typically once the installation is complete, you don't want to modify it again. And that's also to prevent that one installation starts doing stuff in the installation directory of something else, which can actually happen. So if you want to make very sure that's not the case, you can configure easy build to set read-only installation directories. Optionally, it can generate packages like RPMs for the stuff you've actually built, but that's entirely opt-in and that's a separate step. And if there are any test cases defined, so that's like typically bigger tests than what's being run between build and install, it can run those as well after generating the module. So that's kind of an extra validation step. That's not used too much actually. So for every installation, basically running through this whole cycle, some of these may be basically skipping that step, depending on how the installation is defined. How that's done is specified in an easy block. And you have ways of steering these different steps via easy config parameters that you specify in the easy config file. Okay, then once you have software installed, you'll have to do a module use to make sure the modules are visible to the modules tool. And then you can check whether that software is available and load it to actually start playing with it. You can also stack software installations on top of each other and that's exactly what we will be doing. There's in slash easy build. We have pre-installed a bunch of things like GCC, like a FOS tool chain. You will be installing software in your own account on that prepared environment, basically on top of what's provided centrally. So in two separate software stacks because you will not get right access to the central one, of course. And that's very similar to what researchers could be doing in their own account on top of what's centrally available. So that's a pretty realistic setup. So that's why it's important that you need to use this module use to get the central installed modules available and that you make sure that easy build is configured to install stuff in your home slash easy build. If you've already set the prefix, the easy build prefix environment variable that already be the case and you don't need to specify it on the command line. Okay, so that's very, very short. In the tutorial website, it goes into quite a bit more detail. And let me actually work my way through this as well. By the way, to jump back to the overview page, there's a link here at the bottom. So it's not actually here in the menu yet, we'll fix that. But if you go to back to overview, we can scroll back to the agenda and jump to the right part of the tutorial. So now we're looking at installing software. And this gives a bit more information in the slides and gives me stuff I can easily copy paste. So let's do a sound tools installation and our prepared environment. Now I need to check whether everything is set up correctly. Module pop looks okay. It has the central modules available. And the ones that we will be installing in my home account are there as well. I'll also check the easy build configuration. Make sure there's nothing funny in here like the module syntax, I'll get rid of that. So we're dealing with the standard setup. It actually doesn't matter whether it's SQL or Lua or not support both, but we'll stick to the standards configuration. So make sure your prefix is set to your home directory slash easy build and that your build directory is set to a temporary directory. That's gonna be important for some of these installations. That's the case, you're good to go. And we can copy paste this to install sound tools. And you'll see this is not gonna tell us much like it did when installing easy build. It just walks us through the steps and it only says how long a step took, but without giving us much more details about what's actually going on. So it seems to be hanging in some of these steps which take a while. We would like to get some more information on that so we can enable the trace option for this which we will do in a second. So this gives you some idea of how long these things will take. The build step will take a little bit longer depending on how many cores you have and how many other people are building stuff on that same system. So here it took nine seconds, which is pretty good. So easy build will also auto detect how many cores are available on that system and automatically use all of them unless you tell it otherwise. And it's smart enough if you're starting or if you're running in a slurm job or the slurm environment where you have let's say a partial node, only 10 out of 100 cores. Easy build will notice it's only 10 cores and only use 10, but has a smart, it's checking the affinity basically of the process that it's a session that is running. All right, so samples was pretty easy. That looks like that worked. That's good. I have no idea how to install Samtools, but Easy build does. So if you wanna do BCF tools, let's then do BCF tools as well. Easy build knows how to install this. There's an easy config file for it. So this should work. And in this case, it's actually not gonna be happy. So it will start with fetching the files and unpacking and then preparing the build environment. And then it runs into trouble because it says, oh, there's dependencies I need to load. These two, I can't find modules for them, so I'm stuck. And it does give you a helpful suggestion. Maybe you wanna use dashes robot to enable the dependency resolution. So we're still not doing this by default. We'll probably keep this as well. It's sort of a mechanism to protect you from accidentally installing a hundred new things if you're using a new tool chain, even though you didn't want to or you don't have all the modules in view which may be installed somewhere else. And we wanna protect yourself from accidentally doing that. In this case, we can check, hold on, is stuff missing. That's just missing option on SM. Three out of 23. Okay, that's acceptable. So I'll just enable the robot to go ahead and do that. So this is gonna start with one of these two dependencies. In this case, htslib. It's gonna go ahead and do that. Gonna go ahead and do that and installing these things one by one. Now, I'm annoyed enough here that I'm a bit blind about not knowing what's going on. So I'll cancel this and I'll redo this with dash dash trace. So I have a bit more information and I don't wanna do dash dash trace all the time. So rather than setting it on the command line, I'll set it in the environment. Easyable trace equals one. So it's a Boolean option, just giving it a one or a true or whatever, doesn't matter. We'll enable the trace mode as well. So now at least for this session, I can forget about having to enable trace. And you can tell it's gonna restart htslib because that didn't complete. And now you can see it's giving us more information about what's actually going on, which modules it's using, which commands, which models it's loading, which commands it's running, also in which directory it's running that command and it's giving you a log file with the output of that specific command as well. If you wanna snoop on what's going on, you can, well, at least if you're quick enough. We log in here as well. In a different tab, so I can just tail. We can tail the log file. And it should be showing what is in there. Yeah, see, so this is the output that's being generated by the configure command while it is running. So in a separate session, you can actually get a lot more detail about why this is taking a while. Okay, so I'll leave that running. It's already completed one of the installations. It seems the htslib is already there. That's completed. It's now doing GSL. And after it completed GSL, it will do BCF tools itself as well. So we'll free to do that yourself as well, just to see how that works. Going back here, what else is there? Yeah, so missing modules. That means we have to enable the robot. We can enable the trace output with the environment variable that gives us a bit more information. In this case here, we actually completed BCF tools without trace and we're redoing it with trace enabled. So that's what you have to do with that just rebuild. So that gives us a bit more information and more stuff to think about while we're drinking our coffee. And eventually it's still chewing on GSL. That may take a little bit. Eventually you'll also see that it completes the BCF tools installation and the sanity check will tell you that the binary has been found in the installation. So that all looks good. So the log in node is probably sweating quite heavily right now. Yeah, a load of 43. So many people are doing that build. It's gonna take a while, but just let it do its thing. That's fine. And then like we did with our easy build installation, if we want to start using this software, we have to make sure that the location of the modules, so this directory and more specifically the dash all directory in here is in our module part. And once you do that and you do a module avail, you should see the SAM tools which we installed. You should see BCF tools and the two dependencies and the easy build you have installed as well. This BCP one is probably from an earlier quick test as well. So you may not see this unless you also installed it. So you should see that handful of modules on top of what provided centrally. This is gonna be a bigger list. That's in here. And then of course, as you know, to actually use stuff, if you wanna start using BCF tools, initially it will not be there, but if you load the module, your environment will be updated and the BCF tools will be available and working. And we're pretty sure it's gonna work because I think I still do GSL. I think as a part of the sanity check, it will actually run a command like this to make sure that the binary it produces is actually functional. Good. Then after loading modules, you always need to be a bit careful. I can show this once the installation completes. You may need to do a module purge to unload everything again if you did the test like this. If you load modules, if you have modules like this loaded, especially something that includes the tool chain and dependencies, and then you run another easy build session, easy build is gonna spit out a warning because it's not a good idea to load modules ahead of time and then run easy build because easy build is gonna see all those tools in the environment and it may affect the installation procedure as well. So I think by default, it will give you a warning. You can make that warning a hard error so you make sure it never happens and this is how I have my easy build configured. So let me see if I can show that. I'll have to do a module use in here. So now I'm running in a separate session because I don't wanna interrupt the BCF tools installation and it's perfectly fine to have multiple easy build sessions even with the same user running on the same node. They're totally isolated. They will not get in each other's way. There's nothing shared. So that's absolutely fine. What I want to show is, I think it's detect loaded modules or something. So I don't know all the configuration options by heart and this is taking a while. Yeah, so there's an option detect loaded modules. By default, it will warn you about stuff but you can set it to errors to make it a hard error. So let's actually do that. Let's say we load GCC and then we start installing something. It doesn't really matter what and I'll make it a test installation temp and to make sure it actually tries to install and make it a rebuild. So here it's gonna produce a warning that says, hey, hey, hey, you have modules loaded. That's probably not a good idea. See, even though it does the installation but it gives you a warning and you can make that a hard error like this. Now, this is a typical option where you probably don't want to set this on the command line all the time. So again, here we can do export, easy builds, detect loaded modules equals error and then we don't need to do it on the command line anymore. And now it's gonna flat out refuse to start the installation because modules are loaded. It makes one exception. So we have easy build loaded as a module and a bunch of other stuff. If I unload only the modules that are dependencies for GCC. So now I have only easy build module loaded. It does allow that because of course it knows that having easy build loaded, that's not a problem. So it makes one exception. The easy build module, it will not complain about. All right, and in the meantime, PCF tools was completed. We can check whether we have that available. The module parts will already be set up correctly, I think. Yeah, it's in here. PCF tools is ready. We can load it and we can run it and that should work just fine. Okay. And the hour stacking software. So we're building stuff on top of the central software stack in slash easy build. Good. So now there's a couple of exercises in here. Let's take 10 to 15 minutes to work through these. So I'll let you do that yourself. So they're not exactly the same as the demo. They're always a little bit different. The solution is hidden back here. If you click this, it will show you the solution. It's a very good idea to try first before peaking. If you click this, it will actually show you what the answer is. But just take a look at the exercise and try to figure out how to do what the exercise tells you. And there's, I think, only two small steps in here. So it should not take us for real. So give that a try and I'll demo that to complete the section before we move around. Any questions, of course, are also welcome. Yeah. Is that, Mike's still there, York? I need to recharge. That was the reason why I removed that. No, I'm hearing you, I think. Can you hear me? Stefan, I don't think I should. Yeah, I had one question regarding the modules, yes. So we installed some tools, we installed ECF. We see more modules in module avail. Yes. We saw 100 different modules, applications, 80,000 of them. Yes. So I'm assuming there is a way to hide the modules that we probably go through that in now. Yeah, well, it's not really covered in the tutorial, but I can explain that, I'm happy to explain. So there's two ways, essentially, to make sure that modules are not visible to, let's say, end users. One way is to let me find something else to install. To install, I think there's an older piece of, there's a hide option in EasyBuild as well, where you can say, install this, but make it a hidden module. And let me show you what that does. It's not hide. I think it's hidden, like this. So install this, but hidden. This is actually a fun mistake, because you can see EasyBuild gives you suggestions here about what you may be doing. So I did this installation, but if I do module avail Bzip, let me make sure I'll do it here, because here my environment is set up correctly. To do module avail Bzip, I don't see it, even though it's installed. Right, that's because if I look in here, and actually didn't have it configured correctly, that's the result of working in different shell sessions. It's actually in here, but fine. For the demo, it's, no, where did it, ah, in slash temp, of course, yes. I was being careful, like this. So here you see, I installed 108 and 106. 106 is not here, it is actually here, but as a hidden file, which also means in module avail, you will not see it. You can load it, but this is a bit annoying because you have to load it with a dot as well. Because the dot is really part of the module name. So that's not really nice, right? So you can install stuff explicitly with dash, dash hidden. It also has hide depth. So if you wanna say, I never wanna show Zlib, if you need to install that, install it as a hidden dependency, which means a hidden module. So it has options like this as well. So EasyBuild has some configuration options for this. The better option is actually in Lmod itself. So you can also tell Lmod to hide specific modules. And it does it in a way, it just makes them invisible, but it doesn't change the module load command that you have to do at all. So there's no dot involved to make it hidden. And for this, you can check the Lmod documentation. It's always a bit of a challenge to search in here, but yeah, so it has a concept of hidden modules. And it does that by, here in your site package. So this is like a configuration file for Lmods. In here, it has a way of hiding stuff. And I forgot how it's exactly done, but fine. There is a way, you can do it with a hook, I think, as well. So Lmod also has hooks where it says it's visible. And so you can, in some Lua code, you can control which modules are visible or not. So also on the Lmod side, regardless of how visible this configured, you can control it. Now, sometimes you may be tempted to start hiding stuff to not have too many modules, but there are cases where modules are both relevant for end users and also as dependency for other stuff. Right, Python, for example, maybe people want to load a specific version of Python themselves. Python is very often dependency of something. So should you hide that or not? In the case of Python, I think it's pretty clear that probably no, but there's other stuff where it's less clear, right? Even, let's say, HDF5 as a library. Some people maybe want to talk to HDF5 directly. If they don't see the module, they get confused. So yeah, it's a bit of a balance between having, let's say, a shorter list of available modules versus keeping everything clear. Now, even if you hide modules with the Lmod approach, you can still do module avail show hidden. You just have to explicitly opt into showing hidden modules. So there's a couple of options there to, okay. Yeah, so take a look at that first exercise. See if you can get that to work. If you figured it out, use the green sticker. So this tells me a bit on how you're doing with that. And then I'll just take a look at our agenda to make sure we're not running too far behind, but okay. Oh, and the timing here is actually a bit off. It's actually one to two, and then two to three, of course, for writing easy conflicts. So also the troubleshooting part we can actually do in about half an hour. So we're not doing too bad on time. Good, if there's any questions during exercises, let me know. That's intentional, yeah. Yeah, as a part of the exercise, yes. Make sure you know how to do that. I'll give you a blue sticker as well. Make it green if you do. The first one, make it blue if you're finished all too. So that's, then I know we can actually move ahead. And I do want to spend some time on troubleshooting part. I think that's the best part of the whole tutorial. So you need stickers your core. So the stickers are for letting me know how you're doing on the exercises. So make it green if you finish the first one, blue if you're finished all too. And red if you need helpers. Tweety floors. Yeah, yeah, it's going okay. You can give us some time. Don't need to rush. It's good to have some hands-on stuff. And just demos is not enough, right? You need to make mistakes and learn. And something isn't clear. That's a good way to ask questions. What are trigger questions? No, not yet. That would be a very good feature to have. I think it's something we may even need before we declare it stable. Yeah. Interesting stack that I want to start. Yeah. Yeah, or just having a starting point for an easy stack file. Now, the thing is there, what do you do? Because generating an easy stack file is not that difficult. But it will have everything, right? And what you probably want to do generally in an easy stack file is include the applications you care about and just let easy builds figure out the dependencies. And how that will be done is not entirely clear. How do you know whether something was installed directly or as a dependency? Easy build doesn't keep track of that in the installed models. So if you have PyTorch, PyTorch will probably be there in the list. But PyTorch is also a dependency for something else. So it's like, but maybe you used to a flat dump of everything and let you filter it down. That could work, yeah? So that's, yeah, that makes me wonder a bit how we're actually going to do that. But as a starting point, just a flat everything list alphabetically or model veil is that, right? Yeah, that's basically what easy build would be doing. This is my software stuff, okay? I'm gonna redo it. Yeah. All it is? Yeah. Take it off. No, just buy a cluster and install it that way. We're gonna have to do it now. We've got a new cluster. If you're doing this well, you shouldn't be running into many missing dependencies, I think one, two, maybe three, forgot to take a peek at the solution, but most of the stuff that's needed for the installation should already be there in the central software stack. So how many people have the installation running currently for the first part? One, two, three, four. Some people are done. Also running in the back. Yeah, okay. So then maybe while you wait for that to finish, maybe someone can walk me through how that's done. I prefer that to doing the demo myself. Now make sure I'm doing that in a clean environment, only easy build. All right, so the exercises we want to install H5 Pi, specific version of that using the FOS 21B tool chain. So what's step one? How do we get started? You'll have to tell me what to type because now I forgot everything, like this. Yeah, that will give us all the H5 Pi, easy configs, then you can start scrolling and picking the right one. You can filter it down a bit as well because the version is usually directly after that. So you sort of know that that, and that's already way better example. And then we get two options, FOS on Intel and the FOS one is the one we want. Okay, what's the next step? Check for missing dependencies. How? Yeah, this whole thing, yeah. For some reason it's reading this index. It's something we need to look at. Jasper, I keep forgetting this happens, but yeah. So it looks like it's pretty happy. It's only missing one thing. So we did a good job in preparing this. And then, so we found what we need. It looks okay in terms of missing things. And now it says installing slash temp slash user. So how do we do that? While leveraging software available from here. So how do we do both of these things at the same time? Yeah, that should already be the case because it found stuff, right? Well, there's actually an important detail here. Let me show you this. So in this case, the module path has two entries. The central software stack and our own stuff in slash home. That is not entirely the same as doing this. That's having only one. And that matters when, so here we'll cheat a bit. We want to install in slash temp slash user. The location where modules go in here, easy build will, while it's running, automatically add that to module path. So it sees the things, it's installing itself. With this, we'll pick up the modules from here. We won't pick up the modules from here anymore. So as soon as you start doing things in three locations, central in our home directory and a temporary path just to play around in, then we have to be a bit careful because now easy build will not see the modules in here. It will only see the modules in here and the ones itself installs in here. So those will always be visible, even if they're not in module path. But these will no longer be there. They have to be a bit careful if you start playing with multiple locations, what is visible. So everything that's in module path, easy build will see. And in addition, the stuff it installs itself in its own installation part in that session, it will also see but nothing else. So when you do the missing, because we're told to install in dash dash prefix, we should actually be doing the missing like this. Because then we know it's only seeing stuff in here. And it will not blatantly lie to us when it tells you what is missing. It's probably gonna be the same answer in this case. Because none of the stuff we've installed already as a dependency for H5py. And then you let that run, this probably takes you a little while. So we'll let that go. And this is basically what's explained here as well. It looks like before, yeah, so depending on whether the package config is installed or not, there may be an additional thing missing, but it looks like we have that installed and it prepared the entire mentor already. Oh yeah, and another part was enable trace outputs, like I've showed, you can configure easy build to do that. So you know a bit more about what's being done. Yeah, and then the second part, so when an installation completes, try using that software. So this is a small, quite small part. So this is a small Python script or a small piece of Python code to actually use H5py. So make sure that it's actually working as expected. Just creating an empty HDI file. So I'm still having trouble with the first part or waiting for it to finish. Yeah, but now it's gonna be installing lots of things, right? Yeah, so maybe you were a little bit late. Yeah, so we have some stuff installed in here. Central software stack and have to make sure that's visible. So if you do a module avail GCC, for example, it should be seeing a GCC module and it's not. That's why it's giving you this long list. So in the slides and also in here, it tells you in the configuration part, that's not in here, it's in the slides also. Did you find the slide? Yeah, it's here. Yeah. Yeah, okay, so this part is basically important. Yeah, I think when I first logged in, I used this. Yeah, but maybe you started a new session after the break or something. Yeah, so that it has lost that, so you should do that again. So this is your stuff. Yeah, what we want is the central modules. And we do that exactly to avoid that you have to install everything. This would take hours. So if you now do GCC, it's there. You now check for like this, what's missing. You should only see one thing missing. That's a lot better. Yeah, and also here with the robot, the robot is an option, so it needs a double dash. Yeah, okay, and you can add the robot, it doesn't hurt, but since only one thing is needed, it doesn't make a difference in this case, but it's okay. Any problems here with the first part? Still sweating? Yeah, are you working on the login mode or just for SSH? Yeah, because mine is already done and I'm also in the login mode. Yeah, login, yeah. It's okay, you're on the login mode there. Maybe just finished. Ah, yeah, okay. Just not showing you full output. Good to move on? Yeah, so that last part is just using the software we have installed. Ah, so let's throw this in a small Python file. Let's call it h5py demo.py. And now the job is to try running this. Now, if you just do it naively, even with Python 3, that's not gonna work, because the module is not there. So we have to figure out how to load the module. If you do module avail h5py, we don't see it. Why not? Because I just cleaned up that module part, we only have the central modules and we are installing stuff in Home EasyBuild and in modules all in Home EasyBuild. So that's our prefix. Well, that's not correct. It was installed in slash temp, right? Yeah, so even I can make mistakes, it does happen on this all. Yeah, so we need to add this location, of course, to the module part, then it's happy. Then we can load it. And if we then try to run, this will work. And now we can actually use the Python command directly because as a part of the dependencies, we'll have a Python loaded and this provides a Python command without the three or specific version after it. So this will now work. So which Python, that looks Python we have. And this has created a file called empty.hdf5. Good. And then to make sure that's actually a valid hdf5 file, we can run h5stat as command on that, which is also provided h5stat, which is provided by the hdf5 dependency that was automatically loaded for h5py as well. Okay, any questions on this? Before we continue. No questions for now, okay. So now things are gonna get fun. With troubleshooting, that will still, that's actually an important skill when using Easy Build. Things will still go wrong, right? And you'll have to be able to figure out what has gone wrong and then why it has gone wrong. So that's very useful. There's lots of problems that pop up, especially when starting to play with different versions of things that are not supported in Easy Build yet. When you're trying to work out a new easy config file for something that's not supported in Easy Build yet. So there's, it will rarely go smoothly. Let me put it that way, even for an experienced Easy Build user. Missing dependencies, maybe source files are missing, you're running, or Easy Build is running commands, not in the correct subdirectory of something. Commands fail. So they have a non-zero access status or how do you get the output of that? Maybe you're running out of memory or out of disk space or you're somehow triggering a compiler bug. So there's a new thing, a new problem every day. Now, what is very helpful, Easy Build keeps a very thorough log of what was going on during the installation and often it gives you at least a clue to what could be going wrong. So one thing we're currently not very happy about in Easy Ourselves is the error messages it produces. They're a bit of a puzzle always. You can probably do a way better job. And that's something we're gonna work on in Easy Build 5. So we're gonna come up with a way to produce more meaningful error messages and already give you some handles to start figuring out the problem. An easy link to the log file, a clear error message on which command was being run. In this case, it's pretty clear but sometimes the command is like three pages long. So we'll have a better way of reporting on that. Here, it's telling us we're a bit lucky. It's a pretty clear error. So we were running the main command and that produced this as output as a first line and the second line is the actual error. So in this case, we're a bit lucky. The error itself is visible and the output produced and the error message produced by Easy Build. That's not always gonna be the case. So you get like the start of the output of the command but then the error will actually be way lower and this only prints the first 300 characters or so of the output. So if the error is not on top, then you won't see it. Okay. Now there is always gonna be a log file. Every time you start an EV session, maybe you've already noticed this line on top, a temporary log file in case of, I guess it should say error or failing installation or crash. So this will tell you what was going on at the time the installation failed. Now this is gonna be very verbose. It's gonna have everything that Easy Build did, all the commands, all the modules that were loaded, the environment changes, some progress information on the easy block, so where stuff was being run. It gives you everything that's good but it gives you everything and that's bad. So it's a bit daunting at first. By default, it will not have debug mode enabled. So sometimes in easy blocks or within framework, we only lock stuff when debug mode is enabled either because that helps us to debug when Easy Build does something we don't expect or to give you even more verbose outputs than is usual. So sometimes it can help or you can get more information by also enabling debug mode. EB dash dash debug will do that. You can configure Easy Build to always have debug mode enabled if you don't mind having large log files. That's what I do personally but it's a bit of a personal preference. There's actually two log files that are created for every Easy Build session. There's one log file for the whole session which will have everything that happens in that session and there's a separate log file per installation that's being started. So those are kept separate. The reason for that is every time Easy Build completes an installation, it will copy the log file into the installation directory as well. So you can see this for the installations we have done. Easy Build software, let's say BCF tools and the version of that. Next to the bin directory and the lib share directory, there's an Easy Build directory as well which has the log file of the whole installation. The easy config file that was used, it copies that in there as well. It has what we call a test report. Let me show you what's in there. So this has a bit of information of when was this done, which installations were done during this session, which version was used, how was Easy Build configured. So the full configuration is in here. So you can later tell what happens. It even tells you the system information. So what was the host name, what OS version, how many cores, what was available that could help. Maybe you did this six months ago and you didn't know how the system was configured, what the environment variables were and that session that could be relevant as well. So this is before Easy Build starts setting up the build environment, what was in there. I saw, for example, CMake prefix part and that's not a bit of a weird one, but let's say the locale stuff, maybe this breaks something, God knows. So all that stuff is in there and you can go back to that and figure it out. And of course the log file itself is in here as well. So this usually starts with saying which version of Easy Build, which Easy Block is being used and it goes from there with all the details. There's probably a make command in here somewhere like this and this was the output of the make command. So all the nasty details are there. Even for a complete installation later, that may be interesting to get back to and see if you can find a clue there why something isn't behaving the way you expect. If the installation fails, this log file will stay there. So you can go ahead and inspect it there and see what went wrong. So it will always be there, the location at the start of the session, it's a randomly generated part. So you get a new log file every time. And also when an error happens somewhere and the error message will, the part of the log file will be there as well. So yeah, navigating the log file is a bit, there's a couple of tricks for this. The log messages that Easy Build produces will always start with a double equals and a space followed by a date stamp and a timestamp, the location where the log message was emitted from. So this is a Python module inside of Easy Build framework. In this case, the module involved with running shell commands and which line of that code it was. And then some useful information in this case on the info level, not on the debug level. Every time Easy Build starts a step and the installation procedure will also emit a message. So if you wanna say, I wanna zoom into the build step, you can look for a message like this that says starting build step and you can start scrolling from there. When a build fails typically, you could get a variety of error messages. It could say error spelled in different ways with the colon, with the space. So depending on which builders are being used, the error messages are gonna be different. Sometimes it doesn't say error but failure, right? Or command not found or no such failure directly. So there's typical patterns that you may have to look for. And this is something we can also exploit a bit better when Easy Build produces errors because these patterns actually come back all the time. And if Easy Build is made aware of those, we can do a lot better job in reporting errors. That's something we plan to work on. For now you'll have to do the searching yourself. Next to the log file is also the build directory. So the stuff that's typically in slash temp somewhere, the build directory will also stay there if the installation failed. So you can dive in and see if there's any helpful additional log files in there or binaries that worked, the source code that was used for the build and so on. When it's an auto tool based installation procedure, there may be a config.log file that has more information. If it's CMake based, you have to do a little bit more searching because it's actually in the sub directory often. And this will have many error messages that's actually not relevant to the problem at all. So I'm not a big fan of CMake, there's reasons for that. So when it goes wrong, it's a bit of a hunt to figure out what's going wrong. But yeah, all the necessary information should be there somewhere. So that's when you start playing detective and figure out what's going wrong. So now there's also an exercise in this part in the troubleshooting part. And it will be good if we can work our way through this one. So let me show it to you on the website as well. So there's a troubleshooting section. And in that section, I'm basically explaining the same things, error messages, log files, build directory, but just with a bit more text around it. The idea is now is that you copy this easy config file. So you copy paste this text, put it in a file that's called subread.eb or test.eb, the name of the file doesn't matter. You do eb and then the name of that file, you see what happens and you try to fix the problems that you see. So there should be four or five different issues that you're hitting. Before you do that, make sure your environment is set up correctly. So the central software stack is available. So module use on slash easy build modules all. Make sure you have the easy build module loaded, of course. Make sure that easy build is configured to install stuff in your home directory and uses slash temp as a build part. And from there, you're ready to start the exercise with the example easy config file, which in this case will be broken. So the installation will not work and your job is to try and fix that. So also here use the stickers. If you're done with part one, put the green sticker on your laptop. Let's say if you're done with part two as well, use the blue sticker. So that lets me know how far ahead you are. And you'll need to solve them in order in the, as they are in the exercises. So to make sure it's clear how to get started, let me copy the easy config file contents. Module purge here first. Only have easy build loaded. And then let's call it test.eb. You put that in here and we do a, you just give the easy easy, the easy config file to easy build. Sorry, and you can see it's going horribly wrong. And now it's up to you to try and fix that. But this exercise deliberately hit some stuff that we didn't cover in detail. So you'll have to guess around a bit, but hopefully you'll manage to work around some of these issues. And there's different tactics that you can, or different things you can do to fix the problem. Some are partially manual. Some you can actually convince easy build to do the right thing. And I can walk through those. So what's the problem we're hitting here? Is that clear already? Is this a compiler issue or the first part? It's a bit hidden in a wall of text, of course. So we can clearly do better at reporting. You're getting a different error? Ah, yeah. Okay, but that's what I did here is actually try the installation. What you're trying to do is looking for missing dependencies. Yeah, it was just for... You're, yeah, you're checking. Yeah, that works. You're hitting part two of the exercises. You can fix that one first as well, and then you'll hit this one. That's also fine. You can fix the other one first. They can't find the source file. Yeah. They can't find the source. So it's having trouble finding the sources. You can eat what the link is. The link is incorrect. What do you mean with the link is incorrect? The link where? Indeed. So if we check the easy config file, it says, these are my sources. This is the checksum. There's a helpful comment here that says you can download this from there, but EasyBuild ignores the comment. This is not interpreted by EasyBuild at all. So EasyBuild cannot download the sources automatically for you. So how would you fix this? Yeah, so either, yeah, you tell EasyBuild correctly how to download it, or I guess you could download it manually, right? But that's, yeah, that's a manual workaround I wouldn't recommend doing. So now we're actually running ahead a bit. We're gonna cover this in more detail in the writing easy config files part that's coming up. But what I would do in this case is tell EasyBuild how to download this. That means taking this URL without the file name itself. So only the parts before the file name I'll somehow give this to EasyBuild. So I need to set some easy config parameter for this. Easy config parameter for specifying locations where sources can be downloaded is source URLs with an underscore, so that I need to set. And source URLs is a list of string values. So in Python syntax, that's square brackets and either single or double quotes. So if I give that line, if I add that line here, if I add that line into the easy config file, EasyBuild can, if needed, so if the source file is not found, automatically download it and then start going ahead with the installation. So if I save this and just retry, I shouldn't be seeing the same problem anymore. I may be seeing another problem as a follow on. So now it's actually, it says fetching file. So it seems like it's doing a download. I apparently don't have trace enabled or it would give me some more information. Yeah, I do have trace enabled. So download succeeded. At least this part it should be happy with. Now it's giving a different issue. There's actually some other thing I wanna show you here. So let me remove the downloaded sources to make it force, force it to do it again. There's a nice optional feature in EasyBuilds. Which is automatically used if you have the rich Python package enabled. So let me just install it in my home directory with pip. So pip install does as usual rich, should install this thing and all the dependencies that it needs in my home directory. And hopefully that won't take too long, there it goes. So I just added rich as a Python package to my Python installation essentially. And now EasyBuild will auto detect this and give me some more, some nicer outputs. At least it should. Yeah, so now it's giving me colors. That's already fun. And when downloading something, it does get a bit confused sometimes about the terminal width. So let me show it again. When downloading something, it will show you a progress bar when it's actually downloading as well. If it's fetching the files once it has figured out where, showing you a progress bar on the download as well. So it's giving you some more visible, colorful information. That's only thanks to the rich Python package. So if EasyBuild sees this, it will try and use this and provide you some nicer output. And especially when installing multiple things at a time, EasyBuild will show you a counter on how it's doing. Looks a bit messy here because of maybe something wrong with the terminal or the terminal width here, but it should give you a better output. So now we're hitting the second problem in the exercise. No longer with the sources themselves, but some other issue. Yes, question. Yeah, yes. So the naive way to try and fix this issue is adding dash dash robots. So it installs the tool chain, but now EasyBuild is telling you, I don't know how to install gcc8.5. I don't know what that is. That means it doesn't have an easy config file for this at all. So the recommended way to fix this issue is actually something else. We're not really interested in gcc8. That's way too old. That's a tip. Yeah, so we don't want to use gcc8.5 at all. We want to use the gcc that we have available because that's what you told it in, well, that's what's in the easy config file. If you check the easy config file, you'll see tool chain equals gcc8.5. Yeah. So you can check with other gcc versions you have available and try to use those instead. That's working? Yeah. Yeah, that's intentional. Yes. It's, I liked putting these exercises together. That was fun. Causing trouble rather than fixing it for once. That was fun, yes, for once. And we're actually triggering problems that you will often run into when installing stuff yourself. So I really like this exercise. It's a very good one. If you're introducing new colleagues to Easy Build, let them work through this. If they can manage this, that's a very good starting point. Yeah. Yeah. Invalid syntax. Okay, so that means you edited the easy config file and something's wrong. Let's take a look. Line 12. Yeah. So. That's telling the file. Yes, yes. So one thing you should remember when you're editing easy config files, you're dealing with Python code, essentially. Yeah. So as you can see on the sources line, if you wanna define an easy config parameter as a list of string values, you should use square brackets because it's a list. And that's okay. And use quotes to make it the string value. Yeah. So what Easy Build is telling you, I cannot parse this easy config file because the source URLs line is, yeah, it's not correct syntax. Yeah. That's one way of fixing it. I would do something else. Okay. No, but it's okay. That's one option. There's a better option. I'll explain. Okay. So it looks like it's going at quite good for you. Almost there. It's not done yet. No, it's not gonna be done either. There's another issue. Yeah. So it's loading modules now to try and run the test command and the test command will fail. And there's a good reason I'm showing it like this as well. Okay. So the issue that's still on the screen is the problem with the tool chain. So here, there's multiple options here to fix this. Either we try and install this tool chain, but can use that to do the installation. But that means installing a different version of GCC, right? Which will take hours. It's also a very old version of GCC. So rather than doing that because in this case, we don't really care too much which GCC version is used. We just need something that works. If we check which modules we have available, we have a GCC 11.2 and GCC 12.2 available. Now the intention was here to make this the latest GCC that's available. Two days ago, they released 13.1. So that's already, yeah, we need to look into that. So what I will do here is I'll take the safer approach and actually go with 11.2 because I know that's the one we tested things with. You don't wanna risk running into trouble with GCC 12. So we changed the version here in the tool chain line to 11.2 because that's the GCC version we have available. And then we retry the build. It will be happy with the tool chain, but we're hitting another problem. Let me actually kill this colorful output because it's getting in the way a bit. And then we shot myself in the foot quite good, right? Yeah, so now it's happy with the tool chain, but when it tries to do the compilation, it fails pretty miserably. And this is, let's say a more realistic build area you're running into because what's actually wrong here, actual problem that we're hitting is not shown on the screen at all. So this is the error message that these people are giving us and it says the first three on characters of the output of the main command and the actual build error or compiler error is not shown at all here. So that forces you to dive into the log file and try and find the problem. If people have figured this out, please use the green sticker, build step. If the build step works for you, if you somehow fix that, use the green sticker and I'll need a volunteer to tell me what's wrong. I have no idea. I don't know your, oh, okay. Can you open the easy config file again? Yeah, so this is an example of, you're running EBS, yeah. This is an example of an error message that we need to improve, but yeah. So the problem here is what you did is you give the source URL, including the source file name. You should only give the URL. Oh, and you're actually giving, you downloaded it manually or this is a part in your home directory or. Yeah, yeah, no. So the intention is to let easy build download the source file for you. So if you tell easy build, this is the location where this can be downloaded. It will download it and put it somewhere in the source file. So you should actually use this as a value. Yeah, so the errors are a bit confusing. So what easy build tells you is, I'm expecting to find the file there and it's not there. And now I'm stuck, right? It doesn't know what to do. Yeah, and if you give it this, and the URL without the file names with those things together itself. Yeah. No problem, no problem. This is why we do the exercises, right? Yeah, careful with the space that may constable. And also remove the file name at the end. Otherwise, easy build is always gonna add the file into the end, you'll have it twice and that won't work. Yeah, like this, yeah, not the whole thing. Until, yeah, that looks good. So now easy build should be able to download itself. Yeah, and then we'll find the next problem and I'll help you with that. Yeah, yeah, I'll help you with the tool chain part as well. I know the tool chain part you already fixed. You probably did the dash-missing first. Yeah, no, that's good. Yeah, yeah, and now you're seeing the same issue, right? Okay, I would fix it. Yeah, yeah, okay, yeah. How's it going with part three, fixing the actual compiler error working? Or at least it's clear what the problem is. Yeah, okay. So once you understand what the problem is, there's actually two ways of fixing this. And quick way out, let's say, and a good way out. And the good way out is not obvious. I'll give you that, but. So if you actually find the compiler error, it gives you a hint on how you could fix it. Well, it does work. Yes, yeah, but that's a totally different problem. So what the compiler is telling you is good, but there's a better option. So you shouldn't always listen to what the compiler tells you. So first of all, how do we figure out what the actual problem is? Because it's not on the screen. Have a look in the log file, yeah. So the path of the log file is mentioned here. So this is the installation-specific log file. The other log file that's mentioned on top is the log file for the whole session. It's also gonna be in there, but that will have even more stuff, right? So that's gonna be a bit more confusing. So let's take a look at this one. So we open the log file. Usually your editor starts on the top and now wall of text, right? How do you start looking into this? Usually what you do is you jump to the end of the log file. So with Vim, you do dollar go. And here, this is already quite clear. Let me remove the highlighting. This is the actual compiler. So it says, you gave me dash fast as an option. I don't know what that is. Did you mean dash O fast? That looks very helpful, right? So if we look where the dash fast comes from, we open our easy config file. We look for fast and in here, indeed in the build and the options for the build commands, we tell it to use fast dash fast in C flex. So the obvious way out is to change this to dash O fast and retry number good. It's actually a better option. So what the easy config file is doing here is hard specifying the compiler options that should be used and totally ignoring what easy build prepares in the build and file. So if we go and check the output of dash X, and I'll pipe this through less, because again, this could be all of text. So dash X is the extended dry run that shows you among other things, how the build environment will be set up. We scroll through to the build environment. So modules, modules. Here it says defining build environment. And here it sets up a C flags for us, which is controlled by easy build, which has a minus O two, which has a vectorized flag. And this is an important one. This is the one that tells the compiler, generate code for the CPU of the build host. So this is what does vector instructions like AVX2 and all that stuff, the things that give you good performance. Easy build does that by default, and we should try and make sure that's actually used for the compilation as well. All right, so let's say C flags is in the environment and we should use it if we can. What we're doing here is we're just totally ignoring that and we're just giving it O fast F common and use this. So the better option to fix this, and let me put the lines underneath each other, is to rather than use dash fast or dash O fast, tell it to use the existing dollar C flags and add the minus F common to that instead. So the minus F common, it has something to do with changes that have been made in GCC 10. This you really need or you'll keep getting compiler errors. And GCC, easy build doesn't know about this one for this particular software. It's very specific to the software, but the other things prepared in the environment you should try and use because that's what easy build controls. So this is the non-obvious good way out. The compiler will not tell you this of course, if it doesn't care about what's in the build environment. So this is actually what I would recommend using this as a way out. So rather than having dash fast in the build ops, having dollar C flags in the build ops that will make easy build pass the build environment that's prepared by easy build to the make file and then that's used by the compiler. And if you use this, you'll see the same that you saw with the dash O fast. So we'll go through the build and then fail and the sanity check. Was that clear? This is really, it's also in the tutorial website. So if you click the green box, it will also tell you to do that and explain the difference with using dash O fast. And that's something we generally have to take out and take, keep an eye on as easy build maintainers as well. Whenever we can, we should try and make sure that the installation is done with the compiler flags that easy build provides. And it's not always easy to force that, but we do try to keep a, keep our eyes open for that. Okay. So when we do this as a rebuild, it should work going through the install step. You can see it's passing that dollar flags to the make command here at the end. So that looks good. This should take hopefully not too long, right? And then we're done. Almost. So now it fails in the sanity check. So the easy config file specifies, you should run this command with dash dash version and that should work. If it doesn't, something's very wrong, like I expect this to work. And here we're a bit lucky because the output already tells us unrecognized option dash dash version, right? Okay. What the hell is going on? Again, if we open the log file and we scroll to the end, we actually get the full output of trying to run that dash dash version command. And this tells us there's a dash V option which outputs the version of the program. So there's no dash dash version. There's only a dash V. Okay. Yeah. That's an easy fix, right? It's a bit silly of them to not have a dash dash version but we'll survive. That's specified in here, sanity check commands. So there is a feature counts binary. We know that that part of the check past and we just pass it dash dash V option. So we fix that. We save the easy config file. Now we have two options. Either we redo the build and in this case, it's not too bad because the compilation took seconds. But if you hit the failing sanity check after waiting for hours for the build to complete, you don't wanna do this again, right? That's very annoying because the installation is actually already there. The binaries are there, everything is done. It was just easy build that had a wrong check and do I really have to redo the whole thing? No, you don't. There's an option that says module only and this tells easy build only generate the module file. The installation is already there and easy build will only generate the module file after redoing the sanity check. So it will first check, is everything actually there? Are those commands running? And if so, it will generate the module file and this will be a lot quicker and you'll see it's actually gonna skip a lot of the steps in the installation procedure. So you can see building, I was told module only so I'll skip that. Installing, I'll skip that and it just goes head to the sanity check, checks again for the binaries, maybe other things, it loads the temporary module so it can run the command that passed and then it's happy, it creates the module file and completes the installation. And this is a lot quicker, of course. This took three seconds. So even if the build would have taken hours, this would still take three seconds to just do the check and generate the module file. This relies a bit on having a good sanity check, of course. Sorry, Wilde. If you're not doing the module only, it will start again from scratch. It will unpack the sources, redo the build, clean up the install directory, redo the install and then go like this. Because what sometimes also happens is that the installation fails in the middle of the installation step. So some stuff is there, but not everything is there. And if that's the case, you really wanna redo the installation. And redoing the installation may also mean redoing the build. So resuming in the middle of an installation is very tricky. If you wanna make sure you have a fully correct installation. So that's a bit of a necessary evil to start from scratch again. That's why it's very important to know about module only. If it only fails in the sanity check, just do a module only and then you'll have a complete installation. Because that often, we didn't have this in the early days of EasyBuild. There was no module only. And during one of the, this is a funny story, in one of the first hackathons, people were, we also didn't learn to build GCC up front and give people prepared environments so people were actually building GCC during the first tutorial, waiting for a while. And someone had an open SUSE laptop and it was, the sanity check was complaining because there was no lib directory, only lib64. So the sanity check failed and we're like, ah, I have to do this whole thing again. So sometimes you'll learn by doing workshops or tutorials like this that we need new features. Yeah, so that's beating up quite a bit, of course. All right, and now installation completed, we fixed all the problems. If we want to, we can go ahead and make sure that worked. So we can load our sub-breed module here manually and we can run the feature counts dash v ourselves. Make sure that's coming feature counts. That's coming from there, that all looks good. So we fixed all the problems that you were hitting. Any questions on that? No? I hope this was a useful exercise, I think it is. So we shouldn't only show you stuff that works but also stuff that goes wrong. I think that's very useful. Okay, yeah, sorry? Yeah, maybe that's gonna be good for the recording. Yeah, try. When you showed us how to change the easy config file, the compiler flag, how do I know that the minus O fast is not what I want to use? Unless I've got an in-depth knowledge of compilers because I naively assumed, okay, whoever generated that easy config file put in these flags for a particular reason. Yeah, so yeah, usually the easy config files that at least that we have in the central easy config repository are curated and reviewed. And you wouldn't see something like a hard-coded dash O fast there, at least not easily, the reviewer may have overlooked it. This was actually a bad example of an easy config file. So it was intentional, we put in something hard-coded there. And that's also why I mentioned this and refreshed that easy build sets up a build environment and you should try and use that because there's things in there, sometimes depending on the compiler version will add an extra flags to fix problems that the compiler always runs into if you're not using that flag. So yeah, it's good to use the stuff that's in there. That's not that trivial to realize that I do know that. So that's why it's also covered in the hands-on. Good, thanks. Okay, we're now about half an hour behind schedule, but there is a bit of buffer in the program. So it looks like we're doing quite okay on time. I'll continue with the writing easy configs section. So that's really starting from scratch. We want to install something new. So how do we tell easy build to do that? What kind of information do we need to feed in and how? Okay, so we're looking into adding support for additional software, something that easy build does not know about yet at all, wherever that software may come from. For every installation that easy build performs, there is a matching easy config file. Either you're creating it yourself and you're feeding it in or it's already in easy build itself or it's somehow generated by easy build. That's also possible. Most easy configs we have, I showed you the numbers before, actually use a generic easy block. We have 40-ish generic easy blocks. We have, what was the number, 250-ish software-specific easy blocks, but we install 3,000, we support 3,000 different software packages. So a big part of what we support is through generic easy blocks that are reused all the time. Sometimes you do need a software-specific easy block when things get really complex or really tricky and you need to have lots of logic to do things based on the compiler or the libraries or pass the right configuration option. Sometimes you need to go into an easy block. We won't be writing our own easy blocks. That's a bit too far for the tutorial. I'll quickly cover that tomorrow. We won't have exercises on that, but I'll give you the high-level approach to doing. So we're gonna be using a generic easy block, actually two different ones for the exercise. Now, when do you need an easy block? So this is a bit of a thin line. When can you get away with using a generic easy block? When do you really need to go software-specific and do some Python coding? That's a bit subjective, let's say. I always say easy blocks are implement once and forget. So if there's some tricks you need to do, depending on which compiler version or compiler you're using, GCC or Intel, it's good to add some logic for that in an easy block. And an easy build will always check that logic and do the right thing. And you won't have to keep changing or easy config files if you're switching to a different tool chain. Sometimes you can get away with a lot in an easy config file that's using a generic easy block. There's lots of flags. You can have build-opts, install-opts. You can even get very creative with commenting out some parts that easy build does. And sometimes you're just taking a step too far. If you're writing your own easy config files and you're happy with doing that, that's fine. But that's something we're not gonna accept in the central repository there. We will try and help you out to figure out a better option. So there's critical values of something. So some settings have to be correct for the installation to succeed. If that's the case, maybe that should be done in custom easy block. Any logic that's required based on the compiler tool chain or the libraries or whether or not specific dependencies are included. Very often when you're including dependencies, you have to tell it in the configure command as well where those are installed. If that's necessary, probably good to do that in an easy block and not keep doing that in an easy config file. Some installations involve interactive commands. So running a script that asks you questions that you have to answer. Easy build has support for that, but you cannot really specify that at least not currently in an easy config file. So then you need to run one of the functions provided by framework and feed in a Python dictionary that maps questions to answers. That can be done from an easy block. Sometimes you need to generate your own configuration files rather than just running shell commands. And that's a bit difficult to do from an easy config file. If you're hacking around and basically disabling stuff that a generic easy block usually does by using hash tags essentially to comment out parts. Yeah, that's not really nice. So you shouldn't do that. Or if things are very complex, TensorFlow, OpenFoam, PyTorch, I think AlphaFold maybe as well. So we have custom easy box for good reasons. Whether you like it or not, sometimes it's just a necessary evil. So let's assume we can get away with just writing easy config files for now. So easy config files are Python syntax, the troubleshooting exercise showed that and collectively they tell easy build what to install. The order in which the easy config parameters are in the easy config file doesn't really matter. Whether it's name version or version name or even tool chain above the others, that doesn't really matter. It's all key value. But we try to keep the same structure in easy config files all the time. That's helpful for humans. If you have an easy config file that's very close to what you need, you can copy that and just start doing search and replace essentially. And also when comparing easy config files across different versions of the software, we can easily tell what has changed if things are in the right order all the time. There are a couple of easy config parameters that are mandatory. It's actually a very small set. You always need to specify the name and the version of the software. We force you to give a homepage and a description for the software that goes into the module file that generated and you always have to tell which tool chain to use. That's it. That looks a bit weird because why are sources not required? Don't you always need sources? Well, there are, we have easy config files that basically glue other things together. Like a tool chain like FOSS, there's nothing installed in FOSS itself. It's just a GCC and an MPI and a BLAST library combined together. We generate a module for that. And that's it. There's nothing really installed. So forces are usually there, but there's no hard requirement for them. Also the easy block to use does not necessarily need to be in the easy config file. If it is specified, it's usually a generic easy block. If there's no easy block, easy will use the software name and try and find an easy block that matches the software name. So for OpenFoam, for example, it knows OpenFoam, I'm not told which easy block. I'll go and look for an easy block for OpenFoam. It will find it and just use that instead. And that's just to avoid repetition in the easy config file. Whatever can be derived, it will derive automatically. A version suffix is often used to tag easy config files if things are configured a bit differently or if you're using or whatever reason you're sticking to Python 2 as a dependency, that's like an exception. And you can add some kind of label to that in the version suffix. Sources where to download sources, often patch files and check sums for both sources and patch files are usually there in the easy config file. Very often you need dependencies or build dependencies or both. The config opts, pre-config opts and then the build and install variants of that are very often used as well. And also sanity check parts and sanity check commands is very important because that's how easy builds can check whether things are looking okay at least. So those are the most common ones. If you know what these do, you can usually get around quite well and the others are mostly exceptions or used in very specific cases. Then manually writing easy config files or changing existing ones is one option but easy builds can also generate easy config files for you. So using an existing easy config file, here example 1.0, we can tell easy build, you know what, take this one but just change the version to 1.1 and let's see what happens. So it's called try software version because there's no guarantees that this will work but easy build can try this. So what it will do is it will open this easy config file, it will replace the version parameter with 1.1 and then go ahead and try the installation. If it works, you'll have a working installation, you'll have a copy of the easy config file in the installation directory with 1.1 and you have everything you need. So you don't need to manually create the easy config file at all. That works for software version, that also works for the tool chain. So for example, with the subpread example, we had GCC 8.5, we edited the easy config file to use it 11.2. We could have done a try tool chain or even try tool chain version 11.2 and just let it do this rather than editing it manually. So it's just simple search and replace basically. There's other try options as well. They don't always work very well. It depends a bit on how the easy config file was written as well if it's a well structured easy config file, it often works but yeah, sometimes it's still necessary to copy the easy config file or try and modify it. And it's called try for a reason. There's no guarantees this will work. Maybe you're trying something that doesn't work. Like for GCC, it's probably not gonna be as simple as try software version 13.1, will probably need to do a bit more stuff there. Yeah, and the generated easy config files will be saved if the installation works and also if it doesn't work, you can find them in the temporary directory somewhere. Easy build will give you the part of the generated easy config file as well. There's also a copy EC function where we already used show EC to just show an easy config file. There's a copy EC function as well. Why? Because if you wanna manually copy something with CP, you need the full part to the easy config file. If you just give it the name, easy build will locate the easy config file. If you just copy EC, you can copy it without giving the full part. So you don't need to do a search first and then a manual copy. Easy build knows how to do that. And you can combine copy EC with try software version or try tool chain. So if you wanna say, I have an existing easy config file, I know I'm gonna have to fix the tool chain, do a try tool chain, but also copy the easy config file, don't start the installation, give it to me and I'll need to do more stuff. So you can combine these things and it's smart enough to not actually start the installation. Okay, there's a good exercise on this as well. So let's try to start that. There's a coffee break starting at three. So that's about 20 minutes away. We can get you started with the exercise. And I think you can in about 15 minutes, you can walk out to have a coffee. Coffee break takes about half an hour. It gives us a bit more extra time to play with this and to let you try and come up with a fully working easy config file. This is the starting point of an easy config file for the first part of the exercise. So name, version, homepage and description you get for free. The rest you have to figure out yourself. So you can start with copy pasting this into a text file, giving it to Easy Build and just see what happens. Stuff will be missing and it's gonna guide you through what's missing and what you should be adding, at least a little bit initial. There's two exercises. There's an EB tutorial. That's a small C program, I think, which compiles very quickly. So it's not about waiting for the compiler to finish. And then there's another one that does Python bindings on top of that. So two separate easy config files you can puzzle together if you get that far. So that's, I think, a good exercise as well. Yeah, and we can spend, let's say, until the end of the coffee break. So about 45 minutes on this and then we're doing quite good on time. Any questions on this? How am I doing in terms of pace, by the way? Is it too quick, too slow? Okay, yeah. In the back as well, not too fast or too slow? Yeah, if you need help, just wave, we'll help you too. Yeah, okay, so this page has all the text, of course, but in the example, I make sure you keep an eye on the preparation step as well. So this just reminds you, make sure that you have your environment properly set up. Again, the same thing, Easy Build configured in the same way and make sure you have those centrally installed modules available. So you're not, please don't spend time installing Python or something that's also gonna take a while. It's already there, you can use it instead. The slide, there's a link on the landing page. The tutorial landing, yeah, it's not very, very visible, it has like a here link. No, no, no, so you need to do the link to the, yes, I just provided it, yeah, cool. Yeah, if you click overview here, so that's the link is there, yeah. And it's only the slides for today, the ones for tomorrow are, let's say we're still revising them. Yeah, that's a good starting point. That's the first problem you should see, yeah. So to make sure you read the exercise as well, there's some hints in there to help you get started. But it's supposed to be a relatively realistic software package, so something that you may hit in the wild that maybe a scientist came up with. It's not too nasty, but meanwhile, Jasper added the workshop for today to the menu here on the left as well. So if you're having trouble finding a page, you can collapse this out and see it a bit easier to find. Maybe I'll give a little hint. There's the first issue you'll hit when you use this as a starting point. So this very minimal, easy config that only has the mandatory stuff is EasyBuild will scream at you like I don't have an easy block for EB tutorial. You have to tell me what to use. And like I said, let's stick to generic easy blocks for now. Let's not write a bunch of Python codes for this because EasyBuild has stuff to help you. So there should be in the easy block part here, there's a section on generic easy blocks. And this has a link to the EasyBuild documentation that has a list of all the generic easy blocks that EasyBuild has. And based on the name and what you find in the sources of the EB tutorial software, you can probably figure out how to get started. By the way, while you're going through the exercises, if you have any suggestions or ideas on improving EasyBuild, just general stuff, they're always welcome. And so experience from people new to the tool are very valuable. Because we just get blind for some stuff. We're so used to it. That could be something that's a bit of a nasty error. Can you show the easy config file? Yeah, okay. So what's actually going on here is the easy block you're using assumes there's a source file. It's trying to use the first entry in the list of source files and that's an empty list by default. The error message could be a lot better there, clearly. Maybe if you check the, well, there's already stuff missing. I can tell you that. If you check the sources, check what's in the sources. How do you pick an easy config, an easy block to use? You pick the random one or that's not the one I would pick. If you look at the sources of the EB tutorial software, yeah, that's my first step. Whenever I have to install something, I'll just take two minutes and see what's there. To have some kind of hint, like is this Python, is this C++, is it? No, no, no, there's an, if in the example, there's a link to the impact sources. Yeah, absolutely. Yeah, but that's maybe good to show. So I know when you give exercises, nobody listens to what you say, a sense. So when you're working on the example, it's very helpful to follow this link here that shows you the contents of the tar ball. So this is like what you would be hitting if someone asks you or if you find yourself having to implement this, implement an easy config file for this, the first thing you typically do is you look at the sources, like what's there. Are there any hints in here? Maybe if you're lucky, there's even a readme file that tells you something. So that's definitely a good hint to get started. It doesn't look very complex, that's already helpful. Well, there's some stuff in here you can take a closer look. Sorry? Yeah, it looks like it's helpful, right? It's giving you, it is, there's some nasty details in here, but it's very realistic. I'm not sure who this Google person is that added this, but this guy is probably works for him. Yeah, the order doesn't really matter. Yeah. It's more or less in the order that the parameters are needed to perform the installation. Okay. That's roughly like it usually starts name version, homepage description, tool chain because you need the tool chain to install stuff, then sources, then dependencies. Module class usually goes to the very end because that's only needed when generating the module. Sanity check goes right before that because it does a standard check first and then all the stuff in between. Yeah. And usually if you take a look at the existing easy config file, you'll always see the same structure because we try to enforce that and we're quite good at making sure the order is the same. It's just very helpful for easy build. It doesn't matter, but for humans, it's very, very helpful. We are getting something to work as the first step. So like you're doing is fine. But if you would send us a pull request for this, we'll say, oh, thank you, but can you really order things a bit? Yeah. But that's a good approach. Just make it work first. That's the first time. That looks wrong. Yeah, so that's not adorable. Can you show me the easy config file? Yeah, okay. So when downloading something from GitHub, you need to be a bit careful. If you go to the exercise, it actually gives you the, so a bit higher, you're both preparation. This is the URL you should be using. This is the raw URL. So what you're actually getting is the GitHub page that shows you that file or that directory. And that's not adorable. It's a good exercise, but because now you get to figure out how to get rid of the terrible that's there. Yeah. So if you just try it again, you would see the same problem, of course, because the file is already there. But yeah, you can remove it manually or easy build has a forced download options. But that's, yeah, it doesn't matter. It's the same result. That looks better. It got, yeah, it's unpacking the sources. So, yeah. And maybe enable the trace option is going to help you a bit. You should really make that a default easy build file. Yeah, yeah. Yeah, okay. So in the exercise on top, here it says the sources can be downloaded here. Yeah. Yeah, yeah. So yeah, usually you find it on the website. Yeah, so if you read the error, the error says, oh, sorry, I need to find this source star ball. And the paths I considered trying to find this is this location and this location and this location. And it even, it doesn't give you all the options. So it's that easy build is just telling you which parts it considers to find them. Yeah. Yeah, so this is one example of an error that could be a lot better and it should be better. It should tell you, I tried to download it from this location that didn't work and either you fix the source URL or you copy the file to one of these look something like this. So it should be a bit more verbose and explain better what's going on. And so, like for other, any, this is the software where we're writing the easy content files, so. Yeah. Okay. Yeah. It's a lot of information at once, but yeah, it's a bit structured to force you to learn. It's good that you're hitting problems. Because the next, the second time or the third time you're hitting the same problem and say, oh wait, I've seen this before. Has anyone worked through the guided example? Got a fully working installation of the EB tutorial part. Not yet. That's fine. We have time. I think coffee has served York. Yeah. So people who want to grab a coffee, you can keep working on it during the coffee break as well. We can actually go a bit beyond the coffee break because for that last part, we don't need the full hour at all. And there's even in the agenda, I think we saved up, we saved up half an hour for Q&A at the end as well. So that's an extra buffer. Well, I'll go and grab a coffee myself. I'll kill the recording for a while.