 Okay. So welcome to day two of the workshop. Today, we're going to look into a couple more advanced topics of EasyBuild. I'll mostly be doing demos. There are some hands-on exercises in these sections as well. I'm not sure if we should take the time to go through them. I would rather do the demos and then maybe show other stuff that's not in this list. So that's maybe interesting. Maybe Jasper can already think of some stuff that I can show. So I'm happy to improvise and show you stuff that's interesting. I'll start working my way through the slides. I don't have extensive slides on all of these things, but the tutorial website has some more extensive text on that and some exercises, I think on each of these. Okay. So the first thing is contributing to EasyBuild and EasyBuild community. So over time, a community has grown around EasyBuild and that was really never our explicit intention to sort of happen. And it keeps snowballing into something bigger. Today, it's really all over the world. So the map you can see here is the visits to the EasyBuild documentation. That's the best way that we can tell where EasyBuild is being used because it's open source. People can just install it and use it and nobody has to tell us or ask us. So you can see it's all over. It's Australia, US, South America, even a little bit in Africa, in Asia, and of course, a lot of usage in Europe. As far as I know, it's hundreds of HPC sites. I have no better way of quantifying. Also, companies are using it. Some of the bigger sites, the logos are shown here. So Ulig was an early adopter in Germany. So was CSCS. Is CSCS still on there? Maybe not. Lumi now. So EasyBuild is being used on Lumi as a way of not only managing their central software stack for their users, but they also have special documentation for their users to self-manage their own software stack. On top of that, if that's what they want to do. The Fred Hutch Cancer Research Center in Seattle is using it. Lots of universities across Europe, even Microsoft is playing with this in their cloud environment and so on. And so is the funnel, by the way, in Asia. We have a very active Slack channel. Over 700 people there. If you look at the weekly active members, there's about 100 people active. So talking or answering questions, asking questions. And over the years, we have over 300,000 messages that have flown past there. So this has really become the most active place. The mailing list is gradually going down in activity and Slack is taking over. And we also do regular conf calls. So every two weeks, we have an EasyBuild conf call, a one hour conf call, where we give updates on recent development. So what are we working on or have been working on the last two weeks? And what will we be working on in the coming weeks? How closely to a next release? What's going to change? What's going to improve? Which bugs are we fixing? And so on. And of course, every now and then we try to have a physical meeting as well. We had one this week in the room next door here, Monday through Wednesday, our annual EasyBuild user meeting. It was the first one we had physical again after the whole COVID pandemic. So a big part of the community contributes back to EasyBuild. Why would you want to do that? If you create pull requests or PRs, that means you're getting feedback on what you've done. Maybe there's improvements you can make. Maybe there's things you can do better or different that improve the installation. So you're getting help from people that really have a lot of experience in this. EasyBuild maintainers, but also frequent users like York is a very frequent contributor, for example, and being connected to the EasyBuild community is very helpful, not only for EasyBuild, but people ask questions about other stuff related to HPC and the EasyBuild Slack as well. You have some sub channels left and right, and all of that is very welcome. So it's a way to connect with the HPC community as well. If you get your work into the central EasyBuild repositories, that also means you will benefit from the work we do after that has happened. So if we notice a problem with that installation, you will fix that bug and also push it back to the central repository. And then if you keep an eye on the release notes, you will notice that somebody has improved what you have done. And you can pick up on that. Also enhancements like adding additional extensions, improving installation in one way or another, maybe writing a better sanity check. So all that stuff can be done after you have made your contribution. And sometimes we figure out performance improvements as well, where by configuring the software differently or by using a different compiler option, you can make the installation perform a lot better. So all of that can happen once the contribution has been done centrally or you're enabling that. Some of the links are wrong here in the slides. Maybe Jasper can clean those up while we go on. It still has the old ISC links, at least some slides too. So there's different ways to contribute to EasyBuild. Just giving feedback like you've been doing a little bit already. Yesterday and this morning, whether that's positive or negative, it's still feedback, so we cannot improve stuff if we don't know things are difficult or things are wrong or things are confusing. So if you have any feedback, that's always welcome. You can open issues in the GitHub repositories for that. If you're not sure where to do that, just aim for the main EasyBuild one. And if we need to, we will relocate your issue to another repository so you can move things around. You can give feedback in Slack as well, so just drop a message in there or ask a question and that will tell us something. The mailing list also still works and the comfort calls are open really to anyone interested in EasyBuild to just listen in or to ask questions and join the discussions. Next to reporting bugs, you can also share suggestions and ideas for things that could be improved or additional features we could implement. We had a lot of discussions like this at the EasyBuild user meeting early this week. So there's some ideas there that we'll definitely start looking into in the coming months and weeks. And of course, you can contribute back your EasyConfig even if you think it was a trivial update, just a version bump or just to a newer tool chain that could still be useful because it tells people that, oh, this worked, so if I'll just try this, it should be okay. And maybe next to a version bump, you need to make a small change and extra dependency or an extra configure option that is suddenly required or important. If you figure that out, you're saving other people time and that's how we all collaborate. For the more complex installations where easy blocks are involved, if you're making any changes there, improvements there, updates for new software versions, all of that is also welcome. Of course, new support for new software is very welcome as well. We now have about 3,000 different software packages supported, excluding versions and tool chains. And we have new ones coming in almost daily. So for every EasyBuild release, we add in the order of 50 to 100 new software that's supported to EasyBuild and that happens about every eight weeks. So that cadence is not really slowing down. You even see it speeding up, more software coming into EasyBuilds, probably because there's more research software out there than before. And if you're up for it, you could dive into framework and help with implementing new features. So if you're a bit familiar with Python coding, you could help with that as well. Another thing and this is something we worked on quite a bit recently is the EasyBuild documentation. So it is pretty good. It has a lot of stuff out there in there already, but it definitely has gaps. So if you notice any gaps, either let us know or if you're up for helping out with the documentation and you have a bit of experience with writing technical documentation, that's contributions there are very welcome as well. What we have done recently is basically rework the whole EasyBuild documentation to port it to Markdown and MKDocs, which makes it a lot easier to contribute to before it was restructured text and Sphinx, which is not that easy to work with. So that has improved a lot. One thing I'm particularly maybe proud of even is the integration we have with GitHub. So back in 2016, I think York, that's when you first started contributing around 18, okay. So well, back in 2016, we started getting questions from people that wanted to contribute back that had, oh, I have this easy config file and how do I get this into EasyBuild? Our answer was you'll have to learn Git and create a Git branch and push that to GitHub and make a pull request. And that was a bit daunting for many people. So they were not familiar with Git yet or even with GitHub. And at some point we said, okay, this needs to change. We don't want people to email us easy config files and then having to push that into the repository ourselves. So we implemented some GitHub integration. Long story short, this basically means that you can make pull requests for easy config files or even for easy box straight from the EB command line. So you can tell EasyBuild to open a pull request for you. And also the other way around, like I think I already showed yesterday, you can pull stuff in from a pull request as well and do that installation. That has been very helpful. It has helped a lot of people to make contributions. There's a flip side to that as well. Thanks to that, we're getting way more contributions than before. So we're now up to about two and a half thousand easy config pull requests per year. So that's excluding easy box and framework. Everything combined, you have about 3,000 pull requests per year. So if you think about per day, you're on the order of 10 pull requests per day that we need to process, that's quite a bit. These features are very well-documented because that's very important to the community. What this really does is if you're opening a pull request to EasyBuild before you had to do all these manual steps, if you have an easy config file, you should rename it to the correct name with the tool chain and the version and all of that in the right place. Also move it in the right location of the easy configs repository. So in a sub directory with a letter and the name of the software, then you should make sure you're using the latest version of our develop branch. So you check out that develop branch and you pull down the latest version of that. You create a new branch that has a sensible name. You stage the easy config file or committing that. You make a commit with a sensible commit message. You push that to GitHub and then you have to go to GitHub and click around there to open a pull request and this takes way too long, right? If you have to do this thousands of times per year, it's a bit daunting, it takes a lot of time and it's very easy to make mistakes as well, which then again, the maintainers have to help contributors with to fix those mistakes. So that was a bit daunting. So all of this is now automated. You can just do eb new PR, give it the easy config file and easy build does all this for you and also does all the interaction with GitHub for you. So in the end, you'll just have a pull request. So it's a single command. You don't need to know any Git. You don't even need to interact with GitHub yourself. It renames the easy config file, puts it in the right place automatically. It will even generate a decent branch name and come up with a decent commit message. If you don't give it something specific, it'll do that all of that for you. Okay, so maybe before I go into those stats, let's do a quick demo of that. So let's see. I go into my own copy of easy configs. I probably have some stuff in here that I don't have a pull request for yet. These things are not finished, but I'll open a pull request anyway, cause why not. So with new PR, make it a bit bigger, with new PR, you can just give it an easy config file and it will open the pull request. Now this does require a little bit of configuration first. Let me show you what exactly. So you need to tell easy build what your GitHub login name is. You need to give it a GitHub token so that allows it to interact with GitHub on your behalf. Yeah, and that token needs certain permission so it should be able to create gist in your GitHub account and so on. And you also need to make sure that there is an SSH key in place so stuff can be pushed to your fork of the easy configs repository. To check whether all of that is working, you can run check GitHub, easy build, we'll check all of this and as long as you get an okay here, it will say all the checks past, which means we can use new PR, we can pull stuff from a pull request. We can also, this is more for maintenance, review pull requests. You can update an existing PR and you can upload a test report and what that means I will show. Okay, so let's open the pull request for this easy config. I actually know, I'm not sure what state this is in so we're about to figure it out. You can see it's going through all these steps. It's fetching the develop branch. It's copying files to a temporary directory. So it's basically creating a temporary copy of the repository to set up that pull request. It generated a branch name that includes the software name and a date stamp. So it's going to be unique. It's pushing that to my fork of the easy configs repository and then it's opening the pull request. It derived a commit message and PR title from the easy config file automatically that mentions the tool chain, mentions the software name, the version and if there's a Python dependency, even the Python version, then it shows you the overview of changes and it gives you a link to the pull request. And if we click that, we'll find a pull request that corresponds to the easy config file we've been using. And this has been put in the right place as well. As you can see here, this is located in the proper directory. So this is step one. And you notice no git commands, no clicking on GitHub, easy build with everything for us. The next step here is then we have a pull request and now I need to think where I'm going to do this. Let's see, because to reach our clusters, I need to do this firewall dance first. So what we can do now is not only open the pull request and ask that this is added into easy build, we can also show it works for us. So we can do a test installation and tell it with easy build and tell easy build to also push back a test report into the pull request. So let's see if we can get that to work. I will log in here to our clusters. That's working as it is. We jump to a screen session, which is gone, did they reboot the login nodes? No, excellent. Let's take this guy and this is not working. So all of our clusters are specific names. Does anyone recognize the names? Slacking, don't find, no, there's a roll Pokemon. We just like having fun with our cluster names. We'll start a quick job here. So what we're gonna do is take this pull request number and easy build should be pre-configured here. I hope that's working. You can really see that. So I'll just use from PR and I'll check what's missing to complete this installation. Hopefully not too much. I'm hoping that tool chain is already there in this environment. Let's figure it out. So you can see it's pulling down that easy config file from the pull request. This is very noisy because I have a hook, a parse hook installed. So every time an easy config file is being parsed, it hits that hook and it's telling you that's happening and it looks like all the dependencies are in place. I can actually do a test installation of that. So let's do this. From PR we'll do the test and I will tell it to also upload a test reports. That's just an extra option. So it will now go ahead and try an installation and whether it worked or not, it will push back a test report into the pull request to show us whether that worked or not. And along with that pull request, along with that test reports, there will be some information on how easy build was configured, what type of system this was tested on, operating system CPUs, it will have all that information in there. Hopefully this will not take too long. It's a Python package. So it's probably not very time consuming. Looks like I did a really good job because this is working, I think. Check, sanity check looks okay. All the checks look good. It's completing the installation, generating the module file. And once it has done all of that, it will tell us I've added a test report to the pull request. If you go back to the pull request, we'll see there's a test report here that says, gives us some information, what type of node, operating system, CPU, there was even a small GPU available here. If you click the gist, that comes along with it, you can get even some more information. What was built? So maybe some extra dependencies needed to be installed as well. How long did it take? What was the full easy build configuration? What was the system information? So how many cores, what type of CPU? What environment settings were there? So all of that details you can get. And this tells the maintainers and other contributors that this installation worked in this type of environment. And you'll see test reports like this in all pretty much all the merged pull requests. So I'm pretty sure if I check on this one, there will be first of a failed test report. So this tells us how something was wrong here. So this looks like it was an issue with the checksum. So that was fixed. Then the test was tried again. And that looks like a lot better. We have bots as well where we can ask. So please test on this test cluster. And then the bot will go ahead and do that for us. And eventually come back with a test report as well. So there's lots of testing of these contributions that happen. And you can do that yourself as well. Even for your own pull requests or for pull requests of others. So opening pull requests and lots of automation around that to make contribution a lot easier. The result of that is that, so this was basically implemented, I think end of, I wanna say end of 2016, but this guy suggested may have been end of 2015. So you can see a big jump here in contributions. This is number of easy conflict pull requests per year. So in 2016, we had around 1500. And at least in 21, we were very close to hitting two and a half at 2500. Easy conflict pull requests per year. And the automation is definitely a big part of that. The other two graphs show you how many unique contributors we have. So the green line here is easy conflicts over, so almost 400 different people have opened the pull request for an easy conflict file. For framework and easy blocks, that's a bit more complex, but also there we have well over a hundred people who have contributed over the years. And this shows how many unique contributors we have per year. So last year, we had over 120 different people opening an easy conflict pull request. So you can see there's lots of contributions, lots of people working together to improve easy build pull request by pull request. Okay, any questions on this before I move on to one of the other topics? So it's very difficult to make it easier to let you contribute. There's not much more I think we can do to lower that threshold as much as possible. Okay, so this is one I really wanted to show. Right now we can either just move on to the hooks or you can pick your own adventure and I can start with one of the other ones first. Does anyone have any preference on hooks or implementing easy blocks or submitting installations as jobs or using easy build as a library? Easy blocks, any other quotes or we'll probably have time to work through all of them. Slur and Python as well. So I'll keep the hooks for last or yeah. Okay, so yeah, since we have interest in all of them let's just do them as the order that they're here. So hooks is a pretty recent feature. Two or three years ago we implemented this. So what we often found that people were doing is copying easy blocks and changing them and then not contributing back because they were doing site-specific stuff in there and then slowly diverging from what easy build was doing centrally. So they found it difficult to maintain that. Some people are even doing that with easy build framework. So copying easy build framework starting fiddling around in there and then they got themselves stuck in a corner and couldn't really easily go back to new easy build versions. So what we did is we added hooks throughout easy build frameworks. So hooks are basically there's a check whether some Python function is defined and if it's defined it will run that Python function and you get to implement that Python function yourself. So there's like I showed there is a parse hook. That's a small function that is triggered every time an easy config file is being parsed and it's up to you what happened in there. If you don't give it a parse hook it will just say skip nothing to do. If there is a parse hook it will run through your code and you can do extra stuff. Maybe change the easy config file on the fly before easy build is actually using it. So there's lots of tricks you can do. So this is Python code that you implement yourself. There's hooks throughout the whole installation procedure every time a step starts a hook is triggered every time a step completes a hook is triggered. There's a hook when generating module files there's a hook when easy build starts. So the very first thing that easy build does after loading its configuration is trigger the start hook. Also at the very end there's an end hook. So hooks all over the place. Yeah, lots of examples for this. This is really well documented. A very good example of using hooks. It was a talk at the easy build user meeting last year by our friends in Brussels who have a very hybrid cluster. So they have all generations of hardware new generations of hardware even lots of different interconnects and fabrics. So they need to be very careful how they install OpenMPI for example. And rather than tweaking their OpenMPI easy configs they've implemented hooks that change the configuration on the fly depending on where OpenMPI is being installed. And that was what the talk was about. So this is really enabled by hooks. There are some example hooks as well included with framework in the contrib directory. I think this is mostly our friends in Sweden who have added this. So they were very proud of their hooks and they wanted to show it off to the community and they added it in there. So they come along with easy build basically and you can use them as examples. This is a very short example one that we actually use in Ghent. So this is an implementation of the parse hook. So a hook implementation is just a small Python function that you define that takes some arguments. And this self here is an easy block instance. So this is an instance of the easy block class and easy build framework that corresponds to whatever software is being installed. So one thing you can do is you can check what the name of that software is. In this case, if it's example that will mean this part of the code will run. And here, I think this was taken from Vasp, I think from our Vasp installations, we protect them with a particular user group and that user group is very specific to our site of course. So it happens here if we're installing this software package and we're on this particular cluster, we need to make sure we're using the correct group to protect an installation. So we're basically injecting this value into the group easy config parameter before easy build starts really doing the installation and getting ahead of that. So we can do this in the parse hook which is triggered right after an easy config file is being parsed. This is one particular example. So you do need to know a bit of how the insights of easy build work to get this to work. How do you query the name and how do you change easy config parameters on the fly? So that's something you need to know and that's a bit discussed in the documentation but we keep this very stable. So we try to even the internal API, we try to keep it very stable because we're very aware that people may have Python code that hooks into framework and assumes that everything is working as it is before. So if we make changes, we try to do that in a way that it won't break existing code. There's an example exercise of this in the tutorial website as well. So let's take a look at that. So this explains it at length of course, it talks about the available hooks. So you can query easy builds, which hooks are available. This will give us a list. So it starts and it gives us the hooks in the order of execution. So at the very start, you have a start hook when easy build starts for every easy config file. There's a parse hook being triggered and then there's a pre and post hook for every step of the installation. When we're doing extensions, there's a pre extensions hook. So before it starts any extensions and also for every extension, there's another hook that it does. So this is like a loop inside. And then here all the steps and at the end, there's an end hook right before easy builds finishes off. There's another hook that's triggered. Okay, so that's the available hooks. This is explained here in more detail. There are some cave, there are some caveats here as well. If you're implementing a hook with buggy Python code, you're gonna break easy build. So easy build will run that code. And if you cause a Python crash in there, then game over. So easy build doesn't protect itself against buggy hooks essentially. Maybe it could, but yeah, probably better to let it crash and make sure nothing unexpected happens. So you do need to be a bit careful of course. We sometimes get bug reports where we say, this looks very weird. Like what's going on here? This is really unexpected. Then the first thing we typically do is, can you please give us the output of show config, DB show config, this will tell us if any hooks are being used. And if so, then we ask, okay, can you please show us what you're doing in your hooks because that may be where the problem is. In some cases you also need to be a little bit careful. There's a templating mechanism like these percent name, percent version, things you can use in easy config files. If you're fiddling with easy config parameters, you sometimes need to be a bit careful. Sometimes you need to disable that templating mechanism temporarily. Or in this case, for example, you're querying the value of the sources, easy config parameter, which may have template values. If you're not disabling templating, you will get the values with the version actually in place hard coded. And you may not want to do that. So depending on what you're doing here. And if you're manipulating easy config parameters, you need to be a bit careful as well. If you're updating list value, so basically multiple values, you have to be a bit careful that the updates will actually apply and not silently be ignored. So this is wrong. This will not do what you think it does. This is taking the list, you're actually getting a copy of the value, not the actual values. If you append to that, you're appending to the copy and then the copy is removed and the append disappears. If you do a dot update, you will get what you expect. Okay, there's no exercise in here. I thought there was, maybe we removed that for some reason. But yeah, if you want to give this a try, you could do something like this, a parse hook, where you just print something, for example, if the parse hook is triggered. To see if that is happening or you could print particular values of easy config parameters or check that some stuff is correctly set. So hooks are a very flexible mechanism. You can do a lot of stuff if you are not afraid to write some Python code. Okay, any questions on hooks? Clear? Yeah, so this takes a bit of exercise, a bit of experience, of course, to get to use them, but once you do, this is a very good way of making sure you can make some site-specific customizations to EasyBuild while still sticking to the standard EasyBuild. So it's very easy to keep updating EasyBuild and keeping your hooks alive, a lot easier than copying framework and making changes to there directly. Then implementing EasyBlocks. I only have a single slide on that because this is well covered in the tutorial website. So whenever things get a bit more complex, you may have to either implement your own EasyBlock for a complex software installation or modify or update an existing one. So if there's a new version of OpenFoam, for example, and they've changed their mind again on how the configuration options are named or whatever happens, we often have to update an existing EasyBlock. Or if you're very unlucky and a researcher found something that's very difficult to install, you may have to implement one from scratch. This is also Python code. It's essentially a Python script. You will be talking to EasyBuild framework and defining how the configuration steps should be done, how the build steps should be done, how the install steps should be done. That's the typical tree you need to play with. But also maybe the way that EasyBuild prepares the build environment, maybe you need to do something extra in there, set additional environment variables or take some other stuff away. Maybe you want to do something special in the module file that's being generated. So that can also be done from an EasyBlock. So also here, a bit similar to hooks, you get a lot of flexibility but it does require a bit of experience. For this, let's take a closer look at the documentation or the tutorial here because this does have exercises. We can actually work our way through these if we want to. We're doing quite good on time. So maybe that's what we're doing to give you some experience with this. So the basis of this is like I already explained, there's generic and software specific EasyBlocks. I hope that distinction is clear. Generic is like Python package or CMakeMake. Software specific is TensorFlow or Openform. There's an important difference in the naming. So whenever the naming of the Python class that's defined in an EasyBlock, let me just show you an example. It's gonna be easiest. So if we look into the repository and look into the EasyBlocks directory, you'll find again this letter organization and you'll find a separate generic sub directory. That's where all the generic EasyBlocks are located. And let's take CMakeMake as an example. This has some licensing stuff on top. Some utility functions that can be useful. So here, this is a way to determine which CMake version is being used to set up some CMake environment variables. And this is the actual EasyBlock, which is a Python class. You see the name here is CMakeMake. And just by the name, you can already tell this is a generic EasyBlock because if there's a software specific EasyBlock, it will always start with EV underscore. If it doesn't, it's a generic EasyBlock. The reason we have to do this is because if it's software specific, we're gonna use the software name in the name of the class and there's funky stuff that starts with a number, which is illegal in Python as a Python class name, which has dashes. So we have to do this encoding mechanism that is explained here. So for example, for net CDF Fortran, this would be the class name that you use as an EasyBlock. Prefixed with EV underscore and the dash replaced with underscore minus underscores. You need to do this encoding mechanism to make sure it works for any possible software name. This is important because EasyBlock derives the EasyBlock from the software name. So that's why we have this encoding mechanism. Generic EasyBlocks, there's no need to because we picked this ourselves. So we can make sure it's a valid class name. In this case, you can see that the CMakeMake EasyBlock actually derives from the ConfigureMake EasyBlock. So it takes the ConfigureMake EasyBlock and only modifies things on top of that. And you can probably guess what the biggest modification here is. The difference is between running Configure and running CMake. So that means it's probably mostly the Configure step here, which is being changed. And if you scroll down, I don't think you'll see changes to the build step or the install step at all. There's something small in the test step, just setting an extra environment variable, but build and install are exactly the same between ConfigureMake and CMakeMake. So there's this hierarchy of EasyBlocks. You can also see this on the command line. If you do list EasyBlocks, EasyBlock will tell you which EasyBlocks it knows about. And here we can see that hierarchy as well. Let's go and find CMakeMake. We'll be higher up on the, here. So here you see the hierarchy where CMakeMake derives from ConfigureMake. So why do we have list EasyBlocks? Because you can inject your own EasyBlocks into EasyBuild. There's an include EasyBlocks option as well, where you can give it your own Python files and then it will be integrated into EasyBuild, just like the EasyBlocks.com with EasyBuild itself. If you have EasyBuild configured to include extra EasyBlocks, they will also pop up here and you will see where they fit in the class hierarchy. Let's take a quick look at the software-specific EasyBlock. Let's take OpenFoam because we've mentioned that one a couple of times already. This is a pretty big one, a pretty complex one. But the first thing you'll see here is the class name as promised starts with EB underscore and then the name of the software. And this is actually how you can tell it's a very specific one, it derives from the generic EasyBlock class. So this is by no means a standard installation procedure. It doesn't derive from ConfigureMake or CMake or Python package. It really starts from zero and that means you will have to definitely implement the Configure, the Build and the Install step to get a full complete installation procedure that EasyBuild can perform. So this is over 500 lines of Python. It is a lot because it supports, I think, three variants of OpenFoam. So it does lots of checking of the version and the structure of the source codes to know what is going on. And this is actively maintained by the community as well, of course, so that's the good news. That's the naming. Also the file name has to be what EasyBuild expects to be able to find stuff and you can ask what that should be. So there's functionality in EasyBuild that you can make sure you get that correctly. The structure of an EasyBlock always looks the same a bit. So at the top you have the imports, the functionality we import from framework. Then you have the class definition and you start customizing some steps. Maybe you have functions in between here that utility functions that only make sense for this EasyBlock. It's really up to you how you structure this, but in general, that's how it looks like. Like I showed, you can derive from existing EasyBlocks. So in this case, we only want to customize the Configure step, kind of similar to what we do with CMakeMake and everything else will be done by the ConfigureMake EasyBlock. So that saves us lots of copy pasting of code. In an EasyBlock, you can query the EasyConfig parameters as well. So let's say we have this in our EasyConfig file. We can query the name and the version. There's a shorthand for that self.name, self.version which corresponds to self.cfg, so short for config and then the name of the EasyConfig parameter you want to check. These are used all the time, so we have shorthands for these, but anything else like sources, like config ops, you can query like this. If you also need to make changes to EasyConfig parameters, so on top of what's specified in the EasyConfig file, you can do this in an EasyBlock as well. So you can do a hard reset of an EasyConfig parameter like this, so just assigning it a new value. That's not always recommended because that means you'll blatantly ignore what's in the EasyConfig file, which is maybe not a good idea or maybe unexpected for the end user. So often you'll find yourself updating an existing EasyConfig file, so the EasyConfig parameter. The dot update actually adds example to this list rather than just overwriting it. So it's like dot append, but in a safe way. Like I mentioned before, with hooks, you have to be a bit careful if you're updating EasyConfig, yeah. So this does not work as you may expect. You'll have to use the dot update or the change will just not apply. There's a whole bunch of standards, EasyConfig parameters like sources, like config ops, that can always be used in any EasyBlock, but an EasyBlock can also define its own custom EasyConfig parameters. And you will definitely see that with open foam. Well, okay, not with open foam, but let's go with, let's go with Worf, let me move this guy away a bit. So the Worf EasyBlock, the weather modeling software, I know this has a couple of custom EasyConfig parameters. So there's a build type that must be set, it's mandatory, and you need to tell EasyBuild if this is an open MP build and a shared memory parallel, distributed memory parallel, so that's MPI or a hybrid build. So every EasyConfig file should have build type equals to one of these three sets. And it's very specific to Worf, it only makes sense for Worf. And you will see this being queried lower down in the EasyBlock. So here we're checking what type of build type is being used. If it's not one of these, then EasyBuild will complain and say, hey, you gave it the wrong value, this doesn't make sense to me. So it's the EasyBlock itself checking that, whether that's set correctly. And also here it's actually used to do the installation. Yeah, so an EasyBlock can define its own additional EasyConfig parameters. And here we have full freedom. One thing to be a little bit careful with, you need to check whether there's an existing common EasyConfig parameter as well. So maybe you wanna make sure it's not gonna accidentally override one of those. In the constructor of the Python class, so the init method, you can sometimes already define some class variables that you then later will use or update in particular steps of the EasyBlock. So that's also a common pattern that you will see. For file operations, you can leverage the functions from the file tools module. I think this links to the API. Let's see if the links link is working. Yes, so this is the API part in our documentation, which shows you all the functions you can use from the file tools module. One of them is read, let me do it like this. Read file, come on here. So read file is a function to surprise read a text file. So you can just import this function from the file tools module and use this in the EasyBlock. There's a write file as well. There's a function to patch existing files. There's functions to move files or compute checksums or apply patches. Apply patch for example. So for applying patch files to something. So there's lots of functionality here that you can use and that you don't need to implement yourself in Python. You can just leverage that straight in EasyBlock. Also for running shell commands, there's functionality in the tools.run module. There's two main functions there, a run command function to run regular shell commands like make or cmake or got knows what else. And there's a run command QA. This is for interactive scripts where you need to answer questions. So you can give EasyBuild a dictionary that maps the questions to answers that should be given to the questions and you can generate that dict based on a whole bunch of logic and the operating system or the tool chain being used. And that's how you can populate that dict before you start running the interactive shell command. This is actually something we're gonna re-implement quite heavily in EasyBuild 5 because we're not happy with the current implementation of this. It dates back to the original version of EasyBuild almost. It was written originally about 15 years ago. That's quite old back when we still supported Python 2.4. So you can imagine, there's a lot of things we can improve here and redo. And this is really at the heart of EasyBuild so it's important we get this right. We wanna be careful with this. The API of this is currently utterly confusing. So there's weird options in here that don't make a lot of sense. That's another reason why we want to re-implement this. So the best way to use this currently is to check what existing EasyBlocks do and copy that. So you'll find this in here. This is in the Worf EasyBlock. You'll find a call to run command QA which gives it a command which is defined here. So it's a configure script which some options include before and after that. It will take these dictionaries that tell EasyBuild how this question here compiled for nesting. And EasyBuild just says, okay, I'll answer one. So I'll always compile for nesting. And then here this is probably for another version of Worf where they slightly changed that question. You can imagine how this goes. There's also a no QA here. This is when the output of the command gets stuck for a while and then EasyBlock sometimes thinks, oh, this is a question and I don't know what the answer is, so I'll give up. And rather than letting it do that, we're telling it here, just be a bit more patient. That's just an end of line output and it's not really a question. There's stuff coming after that. So there's lots of logic here being applied to run this. I really wish this was not an interactive script. It looks like it's a standard configure script. It's really not. It's very nasty to it Worf, but EasyBuild can handle it quite well. Another common thing in EasyBlocks is manipulating environment variables. That's pretty basic even in Python or West.entry on or West.setEnd can be used and getEnds can be used to query environment variables. We have dedicated functions for this as well because then EasyBuild is aware of what is being changed. It will lock the changes. Whenever you do a setFar, it will not only set that environment variable, but also make a lock message what the original value was. So you can notice later in the lock that something changed that may be important later for debugging. Whenever you use setFar, it will also pop up in the output of extended dry run, for example. So if you do an EB-X to get a quick report on what the EasyBlock will do. If you use setFar rather than the standard Python functionality for setting environment variables, you will get a proper message in this output already. Also, so that's important. Logging as well. So you can add your own custom log messages in EasyBlocks, which sometimes makes sense, especially for the more complex ones. You will no doubt see this here as well, where for example, we're saying, we're rewriting the optimization options in this particular file, and this will be part of the EasyBlock. So you know where we are and whether this option was enabled or not, you can tell that later. Also for reporting errors, and this may be another thing that we improve on quite a bit in EasyBuild 5. There's an EasyBuild error class that you can import and then raise whenever something goes wrong that you cannot recover from. You can do custom error reports here, and that's probably something you can improve on quite a bit as well. So raising an EasyBuild error basically abort the installation and give you that error message at the end. Then especially for software specific EasyBlocks, it's highly recommended to implement a custom sanity check in the EasyBlock itself, because then the easy config file doesn't need to worry about that. Checking for specific binaries, libraries, like it's going on here, or running a command at the end of the installation to make sure things work. This can be specified in the EasyBlock directly as well. And as you can tell, we're just specifying a dictionary here and a list of commands. And then we're passing that down to the parent of this EasyBlock to actually go ahead and do that. So we don't need to do file checks or running commands ourselves. We just pass it up and the standard EasyBlock will take care. The standard EasyBlock class will take care of that. You also get software specific parts in EasyBlock. So there's one EasyBlock for OpenFo, what we support a decade of versions, essentially. And sometimes you need to do things only for certain versions. So you can include version checks. And this is still wrong. They should be importing from somewhere else, but you can do version checks in here and say only this example binary is there for old versions, for version 1.0 and newer, it was renamed to lowercase. So this is a very simple example, but you get the idea. And over time, of course, this makes the EasyBlock bigger and bigger. And at some point, and again, we'll do some of that, hopefully in EasyBlock 5, we'll clean up some of this old junk for software versions that are no longer relevant. Or at least if it's very clear that they're no longer relevant, we can clean out that code a bit. One attention point when implementing EasyBlocks is to make sure your EasyBlock is compatible with extended dry run and module only. This is a bit tricky sometimes because in both of these cases, well, for extended dry run, it's even weirder because here, you're going through the whole EasyBlock and EasyBlock just keeps all the real steps. So if it hits a run command, it will actually make that just a print essentially. So it's not actually running that command at all. And you always need to keep this in the back of your head when you're implementing an EasyBlock because if you assume that command was really run and produced some output and you're checking on that output, then the EasyBlock will fail because there's no output actually being generated during a dry run. Also when doing module only, lots of steps are skipped, configure, build, test, install, these are, that code is all not executed. And if you're somehow relying in the step where the module is being generated on things that were done in the configure or the install step, that all just utterly break. So you need to make sure you take into account that those steps may be skipped and maybe redo some stuff. So there's tricks here. So we're running a command and we're expecting to get an output and an exit code. Here we say, we expect to see success in the output. And if not, let's first check if we're in dry run mode. And if so, we'll just say, okay, I'll ignore this one, okay. I'll assume this went okay because there's no actual output being generated. And only when we're not in dry run, then we raise an error and say something went very wrong here because we didn't see success at all. This is for module only where we need to be a bit careful. We have a self.command class variable here which is being used in the sanity check to pass down a sanity check command. But here we're careful, we make sure that this is not set to none. If it is, we'll recall this set command function. And what we're doing here is basically not assuming that configure step already did this for us. So if configure step is skipped, then we'll still have the none here and we'll just make sure we'll protect against that in case only the sanity check is being run and configure step has not been run. So writing a good easy block also takes some practice and experience and some caveats left and right. You can, of course, just ignore this and your easy buck will still work to do the installation. But then when you try module only, you'll run into a hard crash and that's not so nice. But there's an exercise in here. I'm not sure how far we should take. How many people here are familiar with Python? I think some Python code. Yeah, some are. If you are, you could go through this. If Python is entirely new for you, this will be a challenge. But it does give you the answer in the end. You can take a peek maybe if this makes sense or try to have some kind of mental image of what they should look like. We have some other stuff to cover still. So maybe we shouldn't go through the exercise right now. But if you're interested in implementing easy blocks yourself, it's a very good one. So it gives you a very focused exercise. Again, for EB tutorial, so the one we were yesterday using to write an easy config file, the assignment here is, okay, let's not win an easy config file and rely on CMakeMake. Let's write an easy block for this and make everything happen in the easy block as much as makes sense, which will basically condense the easy config file to the bare minimum. So that's a good example. And the follow-up one is then, hold on, the first one is for, oh yeah, implementing an easy block from scratch. So not using CMakeMake at all but doing everything yourself, which is good for an exercise, but in general doesn't make sense. And then the second exercise is starting from the CMakeMake easy block, deriving from that and only doing the stuff that's really custom to EB tutorial and using as much as the functionality from CMakeMake as you can. So these are two very good exercises if you're interested in figuring out how easy blocks work and understanding them a bit better. Okay, any questions on this? Yeah, we should use that mic that's gonna help for the recording. I'm just going to ask, do you ever get software vendors and application developers contributing easy blocks and are they sort of recognizing that easy build has been used? Not to that extent, currently. We have had pull requests from, who was it? I think the Gromax developers who said, they like when somebody builds Gromax from source that they add a tag, which tool was used to build Gromax. And at some point we got a pull request, let's see if I can find it. We got a pull request from one of the Gromax developers. There's lots of Gromax changes. Maybe it was only to an easy config file, not to the easy block, but of course I won't find it now. Yeah, anyway, sometimes we do get, maybe not, well, I don't think we've seen easy blocks really, but we do get issues or small pull requests that say, this part you're not doing correctly, so I suggest you change that. And then maybe we just take their idea and do it in a different way, rather than doing it in the easy config file, we move it to an easy block or something like this. But we do get smaller contributions from this. The hope is actually, and I'll get to this in the afternoon, that thanks to the easy project, we'll see more of this. Because right now the value for software developers themselves to change something ineasible is relatively low, because only when people are using Easyable to install their software, they're making a difference. With Easy, we hope that changes because we're gonna make it so easy to give people access to their software that we start getting help from developers that say, okay, you're basically relieving ourselves from the burden of having to answer all those installation questions. So we'll help you with making that central installation really good. So we see that as a way stepping towards the developers themselves as well. Yes. Now that is breaking news, which even you don't know, but as everybody knows in here, probably it's always very tricky to install, they are actually interested. I just got the email. Okay, yeah, that's good to know. Yeah, yeah, Stefan. Yeah, so I think you're doing a lot of passing of the output of CLI commands, I think, in this kind of easy blocks. Well, it depends. So in Worf here, I'm not sure we're actually, we're not capturing the output at all here. So we're not checking the output. Easy will collect the output, throws it in the log file. But what Run command does by default is checking the exit code. So if the exit code is zero, it says, this looks okay to me. So I'll just continue and move ahead. Not all commands have a proper exit code. So it may make assumptions that are not correct. If that's the case, if we know that it's a command that you cannot trust the exit code of, then we'll start looking for the success patterns, for example, in the output. And this is actually one of the reasons we have insanity check. So originally, I think the trigger originally was MATLAB, where the MATLAB installation script will just always do exit zero. Everything is fine. And often you end up with a totally empty installation. So there's nothing there, but the script said exit zero. That's when we started doing the sanity check and say, okay, we won't rely on exit codes alone. We'll also do some active checking ourselves. Yeah, my question was because my experience, when you pass in the output of a CLI command from one version to the next, even very small versions. Of changes, yeah. It really just breaks, it just changes. So it's nothing that you're not. Yeah, we're doing that as only when we really have to. So as long as we can rely on the exit code, we'll do exactly that because that's way more reliable. Any other questions on easy blocks? So this of course is, there's a lot going on here. The documentation is quite good, I think. It really goes step by step. It breaks things down. It highlights all the stuff you should be taking into account. The good news also is we have 200 examples of software specific easy blocks, 40 generic easy blocks. So there's lots of stuff you can look for in the code. If you're wondering how a particular function from file tools or whatever can be used, you can check the existing easy blocks as well. Maybe one more thing I'll quickly show is the use of include easy blocks. So let's see how I show this. Let's take Worf as an example. What I'll do here, I'll download, not like this. Parallel link. So I'll download the existing Worf easy block. I'll make a small change to it. So to make it really silly, let's make it report an error as the first thing it does. So raise easy build error like this. And now we wanna make easy build use our custom easy block to do a Worf installation. So let's find a Worf easy config file we can use. Which one we use doesn't really matter because we won't get very far. So let's grab this one. And now we can do an include easy blocks. We just give it the path to our custom easy block and that will make easy build use our easy block rather than the one it has itself. And this works for both an easy block that's included with easy build already or a totally new one that it has no idea about before. And you can see we're getting our own custom error. So it's running our code. That's a very nice way of also for developing for quickly testing stuff. If you're working on your own easy block, you can just do include easy blocks and keep testing it until you're happy with it and then make a pull request or not. You can keep it in your own repository and some sites are doing that as well. That all works fine. Next topic. Yeah. This is also a very powerful one. I only have a single slide on it, but I have a very good example that you can also try and prepare the environment yourself. So with easy build, we're often installing not one thing at a time, but a whole bunch of things at a time. If we're installing something and it has three missing dependencies, maybe those dependencies also have dependencies and you quickly ramp up to 10, 20 installations that need to happen to get a single thing in place. For this, it would be nice if you're sitting on top of a big HPC cluster to actually use all those nodes that you have available anyway. So what easy build can do is it can submit the installations as jobs to the system. So you're basically distributing the installation as long as dependencies can be installed independently of each other. It can do multiple installations at the same time. The way to get easy build to do this is you have to configure it a little bit, but very minimal. This is actually enough. You just tell it to use Slurm as a job backend. The default backend is different and we're gonna change it in easy build five, but Slurm is definitely the most commonly used one. And when you do an installation, you basically just use the dash, dash job option to make it submit those installations as jobs rather than performing them on the go. You can combine this with dash, dash robot and easy build will look for all missing dependencies and submit each of those installations as a separate job. And it will not only do that, it will also set dependencies between these jobs to force them to run in the correct order. Of course, the dependencies have to be there first before they can do the final installation. So there's an example of this in the tutorial websites, which we can try on the backend. I was testing it yesterday and it's a very good example because it will work partially and I'll show you what that means. So there's a bit of configuration. I won't go through all the details here. This is actually enough. Configuration can be a bit more fine grained because you can tell easy build, how many cores should be used per job, how much time should be requested per job. So you can do all of that if you want to. If you want to have more specific options, slurm options that you want to control, you can set the S batch environment variables for that because all easy build does is really generate a small job script and then just call S batch on that job script to submit it. So as long as these variables are set, they will be picked up by the S batch command automatically. You can combine job with robots to get all the dependencies one by one. And there are some caveats here. You need to be a little bit careful how easy build is configured and where it's configured because whenever a job starts, you need to make sure that easy build will be correctly configured in that job. So there's some details here. So when you start using this, definitely be a bit careful or first do it in a test installation prefix rather than doing in our production system. Yeah, if things go wrong, the log file is usually on the local disk. So if you're submitting that as a job, that means the log file will go in the local disk of the worker node, which you'll probably lose as soon as the job finishes. So there's auto cleanup on most clusters. So maybe you'll need to reconfigure easy build to use to put stuff in a different location like in your home directory, for example. So you won't lose those log files, those temporary log files, if something goes wrong. If the installation completes these, the temporary log files are cleaned up automatically. So there will be no junk left behind there, at least for working installations. And with log files, the log files that easy build creates when starting an installation, if the job gets terminated, for example, if you asked for one hour for a job and you're building open foam on four cores, there's just no way that's gonna be done in one hour, the job will be terminated. The log file will stay in place. So easy build will not be able to clean up that log file. So if you resubmit that installation, you'll hit that log file and easy build will just say, there's a log file, I'm not doing anything. So you may need to clean up the log first or when you resubmit it, you can tell easy build, I know there's a log file, just ignore it and go ahead and do your stuff anyway, because I know there's no installation running anymore. Can clean it up manually or you can tell easy build to ignore the log. So, and you can follow along with me if you want to in the prepared environment, as long as you have your environment set up correctly. So pretty much the same as yesterday. When you log into the prepared environment, make sure you have the easy build module loaded. So you mean to do a module use slash easy build modules all and then load easy build. Make sure easy build is configured to install stuff in your home directory slash easy build and that you're using a custom build path. It says here, Devsh mem for shared memory but slash stem slash user just works fine as well as it did yesterday. And the only extra thing is telling easy build use slurm as a job backup. That's really the only thing that changes. So let's do all of that in here. Well, in my case, I already have most of it, just so it's clear, let's do all of these. I'll change this guy to slash temp because I'm not sure I won't run into problems otherwise and I'll turn to use slurm. If you do a show config, you should see all of that popping up as expected. Backend slurm, temp as build path and prefix our home directory slash easy build. Once you've done that, we can check what we're gonna install. So the project here is not the SBATCH account, the slurm account is not necessary in the prepared environment. Everyone can just go ahead and submit jobs. What we'll do is try to install this Augustus easy config and let's first check if we are indeed missing enough things to make it sensible to submit as jobs. Yeah, so we have 11 things missing. That's quite a bit. So if easy build has to install this sequentially, it may take a while. This is where we can leverage the power of our slurm cluster and to get it installed, we can give it the same command. We add dash dash job to it. We'll also tell it to use four cores per job and five hours per job, which should be plenty. We'll enable the robot and to get some sensible output, we'll enable trace as well. So when you run this command, easy build will generate job scripts, submit them with the right number of cores and wall time and all the other configuration will be passed into the job script. So as long as you have easy build configured to install it in your home directory, that's also what the job will do. So let's run this and see what happens and everyone can try this. There's only four worker nodes in that cluster. So the jobs will not be proceeding very quickly but it should work just fine. So easy build will resolve the dependencies. It will also download all the missing sources. So in my case, that was already done. Everything is cached. In your case, you will see it pulling in the sources for everything first before it submits the installations. And then it will submit a bunch of jobs. Now with SQ, we can check that those are indeed there. And you can see here, the four nodes, I was a bit lucky, the four nodes were available. So I stole all four, my jobs are running there. And for the others, some are waiting. Priority, basically waiting until a node becomes available. Others are waiting for a dependency to be completed first. So easy build sets dependencies between those. And of course, the Augustus job can only start when everything else is done. And you will see this progressing. This will take a couple of minutes. I tried it last night. It wasn't actually fully working because the boost one, and I think the sweet sparse one will fail because there's not enough memory on these worker nodes. They're too tiny, essentially. At least too tiny to install them with four cores. I didn't try this last night, but one trick you could do is change the command a little bit and tell easy build to only use one core. So that's parallel one, but still request four cores. You'll basically, you'll only use one core, but you'll have more memory available for that one core. And that may make the installation work. It will just go a bit slower. This is how we install all our software in Gantt. We make no exceptions. We submit everything as jobs to our clusters. We have seven, eight different clusters. So we basically submit the same installation seven, eight times to each of the different clusters, different CPU types, different generations of Intel and AMD. And it just all works pretty fluently. So that's a very powerful mechanism and it's quite easy to use. Did I cover everything? Yeah, and in the end, you should see those installations pop up and you could just start using them when everything is complete. I'm pretty sure unless you use the dash parallel one, it will not fully work, but some of these installations will definitely finish. So it has done some of them already. And if we check the modules, let's say the last 10, yeah, we should see that these have been installed. The time is one hour off to where we are, but all of these were installed in the jobs. So that seems to be working quite well. Any questions on this? Whenever we do an easy build regression test, this is exactly what we do. We just give it, let's say three, four generations of easy config files, all what's included in easy builds, which is usually a couple of thousand installations. We submit them all as jobs, we go off to weekend and on Monday morning, we can tell how easy, how well easy build was doing, how stable it is and how, to what extent it's ready for release. There's always surprises there. Something's failed for silly reasons. Gremlins biting on the cables or there's always something that goes wrong, but it is a very easy way to get a quick care, a quick view. And we can usually harvest some empty worker notes or kick off some researchers to speed that up. Yeah. We tend to have some in the cluster just for certain tasks. Yeah, you can do one of these export here. So like you can control the account or the partition or reservation. You can just set one of these environment variables and SBatch will pick up on them and submit the jobs in the reservation. Yeah, that works fine. So all easy build really does is generate the job script and run SBatch on it. So anything you can do with SBatch and control in the environment, you can do here as well. Yes, Jörg. So basically, if you've got dependencies, say 10 dependencies, you're kicking off 10 individual jobs, but you make sure that the dependent one is not running before the other one. It just sets dependencies between the jobs that's a standard Slurm feature. Yeah. So you're telling Slurm this job can only start running when this one is completed. Yeah. Are there any advantages doing it this way instead of just submitting Augustus and using 12 cores or whatever? Depends how many installations are pending, right? So here we're, even if your nodes have 100 cores and you say, let's submit jobs with five cores, it's easy to find gaps in the system. And you can tell it was doing four installations at once. On a single node, you can only do one installation one by one. And sometimes with 12 cores or 100 cores, you're not really getting any speed up anymore. But sometimes it makes more sense to do four builds at once than one built with 16 cores. And really, if you don't really care too much, you just submit them. And as soon as the job is done, the node is free again and other stuff can go. So there's a bit less babysitting. The reason I'm asking is A, we are using PBS Pro, so I can't use that. Yeah, not well. So Slurm is one of the backends. I think we still have Torq support, which is also using Qsup. So you may be able to get away with telling easy build, use Torq or use PBS. I forgot what the setting is. And it may work with PBS Pro. We don't do really anything special. We just Qsup and generate a slightly different job script with HPBS tags and that's really all we do. So we don't really rely on anything special in Torq or PBS Pro. So it may work with PBS Pro already. Yeah, but what I found quite interesting is you basically pulling it apart in which jobs you can submit immediately in which one are the dependencies. That would be quite an improvement. So, I mean, you know what we are talking about tomorrow. I don't understand that. What I'm doing is, Zoe, I'm just submitting one big job. And then it's going through the whole list of dependencies. Yeah, okay. You want to be able to do that as well. I want to do it that way. Yeah, we don't have that option currently. So when you do it as a job, combined with robots, it will do, so it doesn't pass the robot into the jobs. It does the resolution first, submit them all separately. You could have an option that says, I want to do the robot in the job, right? So just submit only one job or yeah, some more fine drink control. Yeah, it is the resolution which I found quite interesting. Yeah, thanks. So I think by now, yeah, other people are trying this as well. That's very good. I think these two are going to fail because there's not enough memory on these nodes. At least that's what I was seeing last night. Okay, so that's the submitting jobs part. There's one more advanced topic. We're doing quite well on time. You can also use EasyBuild as a Python library. So not only from the EB command line, but using it straight in Python code. So you can import stuff from EasyBuild. All the functionality that is there in framework, you can leverage if you want to. There's one small caveat, an important one. One thing you always have to do in a Python script, if you want to use EasyBuild as a Python library, is set up the EasyBuild configuration first because there's many functions in EasyBuild that assume EasyBuild is properly configured, that it knows whether debug mode should be enabled, that it knows whether tracing should be enabled. So all of these things are checked in the code. And if the setup configuration was not run, you'll get strange errors, EasyBuild running into unexpected situations or even hard crashes, because it assumes the configuration is there. So always do this, and then you can really import any functionality from EasyBuild and start using it. There's, I think, a good example and maybe also an exercise on this in the tutorial. So I really condensed the information on a single slide, but that's what this first section is about. So it really does this. It's a bit more involved here because this setup configuration call generates some output as well. And here we're telling it, yeah, please be silent. We don't want to have any output when setting up the configuration. And you can actually in the Python script also pass arguments like you would be doing to the EB command. So you can, if you want to enable debug, selectively in the Python script, you can add dash dash debug in this list as an argument. The temporary directory that EasyBuild creates will not be cleaned up automatically. You'll have to do this manually. So in the options, so these are the parsed configuration settings for EasyBuild, you can ask which temporary directory was used and you can clean that up. So that's the only minor thing to be aware of. So why would you use this? Well, you could leverage the file tools module to read files, write files, copy files, apply patch files, compute checksums, download files, all of this functionality you can use. So regardless of what you're doing, all that functionality is there. You can import it and use it in a Python script. So that works fine. So here we're setting up our EasyBuild configuration. And the assumption is where this script takes a download URL like this, which downloads a file. And what we want to do is have a script that computes the SHA256 checksum and just prints it after downloading the file. So this is what this small Python script does using functions that are provided by EasyBuild. It's a small tiny example. You can give that a try yourself in the minimal, in the prepared environment that should work fine. Similarly, you can use the run functions for running shell commands. Here it's quite simple. So just passing a run command, capturing the output and the exit code and seeing whether that worked or not. And of course you can use this as a base for doing more complex stuff. So that I think is pretty clear. It first checks whether the commands that it wants to run is available. And if it is, it tries to run it along with the arguments being passed and it reports on the output and the exit codes that happened from that command. So that's maybe, let's take this as an example and try this run demo. Let's see if this works. Python run demo. So we give it a command to run. Let's call it Python again. And we want to run dash V on that. And easy, that script will first, why is this getting confused? Ah, it has make hard coded. Ah, okay. Let's change it. I want to pass the command to run to the script itself. So this is the command that we wanna run. And these are the options for the commands. That should be better, I think. Python not found, that makes sense. If we do Python three, it should work. No, it's still running, make, why? Ah, here, okay. So now it does work. So you can use EasyBull as a Python library to check whether commands are there to run those commands and then get the output and the exit codes from that. And again, the important part is here, you always need to do the setup configuration. Let me show you what happens if you don't. We'll just disable this and this. The imports will still work. But it's very likely you'll hit the crash because here it says undefined build option. So I want to check whether trace output is enabled and I can't because EasyBull is not configured and then you run into a horrible crash. That's why you need to do the setup configuration first. This library capability was added relatively late in EasyBull. So we have lots of places in the code where it never took that into account and that's why that setup configuration step was really needed. There's another example here interacting with the modules tool. So EasyBull talks to Lmod, usually Lmod, we're using Python functions as well. So there's a modules tool function to set that up. And here, for example, what are we doing here? We're creating the modules tool instance. We're checking which modules are available. So that's a full list of modules. We're then checking if there's an EasyBull module available, we're checking which modules are loaded. So that's a module list. And then here we're checking for a Bzip module. We're essentially doing a module show on that, I think, to see what that's doing. So you can run this as well. That should actually work fine. It's called this Lmod demo. And again, this is just a Python script which we'll talk to Lmod, figure out how many modules are available, figure out all the EasyBulls there. And it's loading a Bzip module and then checking what that looks like. So that's working fine as well. And there's lots of functionality, of course, you can use in framework, which I'm not showing here. This is also an interesting example where if you want to have a Python script that does something with easy config files, you can use the functionality from framework that parses the easy config file and then query that information as well. So if you need to do, this could be useful, for example, when we change stuff in EasyBuild 5, when we enable use pip by default, we can write the script that modifies all those easy config files rather than getting creative with Zed or something else. Any questions on this one? Or any other more advanced topics you would like me to cover? Yeah, Stefano has the mic in place. Yeah, I was wondering if you could talk about the integration between EasyBuild and containers, if it's possible to use EasyBuild containers and how that works and processes. Thank you. I don't have slides on this, but I can show what is there. So there is indeed some form of integration between EasyBuild and both Docker and Singularity Apptainer. There is a big difference between the supports for Docker and Apptainer. So the, well, I should say Singularity in the current release. So the Docker support is very, very basic and not really well tested or developed. The Singularity support is a little bit better. And in the next version, in the next EasyBuild version, we'll also have proper support for Apptainer. So right now it must be Singularity. So I'm not sure how much I can demo because I think we only have, there's a Singularity command in here, but I think it's actually Apptainer. So I think EasyBuild will trip over that unless I demo with the developed version. I can actually do that. So we go back to yesterday. There's a section here to the development setup. So let's do that very quickly. Let me unload my EasyBuild module, no EB command anymore. It is still there. It had a local, remove this guy, move lib, move this. So now there's definitely no EB command anymore. I'll do that in here. So I'll clone all the three main EasyBuild repositories, framework, easy blocks, easy configs. Hopefully the easy configs one will not take a huge amount of time. Lots of small files in here. Lots, this is over a decade of history being pulled in, but this will take a little while. But then if I just update my environment, I can play along with the EasyBuild developed version. While it's installing that, the support we have in EasyBuild for building container images and generating container recipes, it's an experimental feature. So it actually has been experimental for quite a while. At some point that looked like a very interesting option, but we never really saw a huge interest in it and development has sort of stopped, but we are actively testing it. So we know it still works, what is there still works. What you need is Docker or Singularity and a very old version will work fine. And if you want to let EasyBuild actually build the container image itself, it needs sudo because it does a sudo Singularity build, essentially. That's now hard coded. That's something we could change if you're do, depending on how Singularity is installed and especially with later versions, you actually have support to build rootless building. EasyBuild is not aware of that yet, but that's something we could fix. So it basically can do, it does two things. It will generate a container recipe. And then if it's told to also build the image, it will just call out to Singularity build using that container recipe. So EasyBuild doesn't generate the container image itself, it just calls out to Singularity or Docker to do that. There's a whole lot of configuration you can do here. I think we have an example and this is relatively old, but we can probably try and see what happens if we do this, but make sure this makes sense. Oh, this is quite old, okay. Let me take what we use for testing because I know that that bit will at least work. So I'm not showing you stuff that's horribly broken. Container tests, so somewhere in here, we're doing this, yeah, so that installation worked. Let me finalize that environment first. So we have our EasyBuild developed version. Yeah, this one will be happy with AppTainer, where it should be. What I'll do here is I'll configure EasyBuild to use a particular setup for the containers. That's the container config option. I will tell it to use Docker as a bootstrap agent and I'll use this container as a starting point. So that's a prepared container that we have in the GitHub Container Registry. That's our base container to use. I think I'll also need to tell it to use AppTainer, but let's see what happens if I don't. I will do this and I will tell it to only generate the recipe. I think that's what it will do by default. So let's see if we do this. May not be happy, but can please show you what it does. Okay, so what it does by default, it will only generate the recipe. If we give it the container build option, it will also try to do the build. We don't have to do here, so that will work. But I can show you the recipe it generates. So here we said we want to install this easy config file in this container. And if you look at the recipe, you'll see, okay, it does a from on that container. The first thing it will do is install EasyBuild. So we're using also EasyBuild in the generated recipe. Then it will go ahead and make sure our location where we were want to install the software in the container image is prepared. It does some lmob magic. It sets up an EasyBuild user in the container image. It configures EasyBuild by setting environment variables and trying to make sure that it all works. And then it will do an EV-robot on that easy config file in the container recipe. Once that is complete, it will update the lmob cache. So that's going to be quick when loading the modules. And in the environment section of the container recipe and make sure that things are set up correctly. So the module command is available straight away after launching your container. So if you're shelling into that container, this BZIP module will already be loaded, ready to go. So it basically does all the hard work for you of using EasyBuild in a container recipe to install software. So that is quite interesting. I don't think I can try, obtain a build and see what this will do. How does obtain a build work again? It's definition file. And then SIF path like this. And then this probably does rootless building, I hope. So we're just passing the container recipe that EasyBuild generated. And now I have no idea how far it will get, but because it's experimental, but it should work. It should give us a container that has this BZPZ config installed and we can see if that's actually working. So like I said, this is not really actively developed currently, but we were running it in our CI. So every time something changes in EasyBuild, we run this example essentially, so it should work. So it's pulling down the latest EasyBuild version in the container to install that BZP. So that seems to be going okay. And there's definitely things that can improve here. So maybe you can start from a container that already has EasyBuild or something like this. So it's making very minimal assumptions on what's in the container image. It tries to take care of everything itself. This may take a while, we'll see where it goes. Any questions on that? Yeah, okay. Any questions on other EasyBuild features? Can I do X, can I do Y? Wild ideas for new stuff it should be able to do? Ah, okay, so now it is not very happy. So it looks like EasyBuild thinks that we're running EasyBuild with Root in the container. And yeah, it doesn't like you when it does that. So by default, EasyBuild will detect you're running EasyBuild as Root and say no, no, no, you should not install the make file of a scientist with Root, that's a bad idea. So it may be how AppTainer is configured here, I'm not sure that it gets confused. But you see it got to the point where it starts to install that EasyConfig in the container image. That's the thought. So it's really the same thing for AppTainer and Singularity. Ah yeah, the question was, is there some documentation on how to configure for AppTainer? It's really the same thing as for Singularity. There is in here, probably lower case. Yeah, so the container type, the default one is Singularity, we should flip that to AppTainer, but it will only support AppTainer and with the next EasyBuild release because AppTainer is very, very close to Singularity, but it's not exactly the same. So the only thing you really need to do is that container type to AppTainer and then do a container config like I did, which basically tells you which container image to start from and that should be it. And then figure out that root issue, I'm not sure what's going on there. So that may be a bug we need to fix, but other than that, it's exactly the same as with Singularity. Okay, that brings us more or less to lunch, unless there's something else. So this afternoon, we can take a look at Easy, which is in some sense a follow up to the EasyBuild project and I can explain what's going on there. We'll also be able to do some hands-on on that. So getting access to Easy and playing with Easy, that will all be possible in the prepared environment. I'll do some demo as well on how we would do a native installation of Easy. I'll kick off an MTVM in AWS or Azure and show you how that's done if you have root permissions. So if you wanna get started and play somewhere you can, but in the prepared environment, we'll use a special container that we have to give you access to, to Easy. Let's work after lunch.