 Okay, so hello everyone, and thank you for joining me in this last talk of the EasyBuild meeting. And my name is Alex, and you might know me by the Lexming GitHub tag. And today I'm gonna talk to you about the EasyBuild framework, and we will do an overview of what is EasyBuild framework. So I work in the University of Brussels as system administrator, and I do also user support. And I'm also maintainer of EasyBuild for the past three years, I think. However, I have absolutely no idea how framework works. And actually I was the one, or one of the people requesting this talk. So I requested a talk on framework, a kind of guide on framework, because as a contributor to EasyBuild, I was finding, I was having difficulties in finding my way through the framework. So writing easy conflicts at some point is okay, writing easy blogs is also okay. But then framework, it's own beast, and it is easy to get lost in it. So I guess that I'm the origin of my own demise, I guess. But while they say that the best way of learning is to teach. So even though I have absolutely no idea how framework works, and today I know a little bit more. So that's why I'm today here giving this talk. So this is framework. And what you see here is the diagram of packages and that form or that are part of EasyBuild.framework and the EasyBuild.tools packages. So there are many elements as you can see in the diagram and there are many interactions between the elements. And this is actually what makes it difficult to navigate through framework because you might get errors somewhere, but then the traceback of the error might go as very low in the stack. And the back might be in a completely different region of the code base. And for instance, we just see here in the middle in blue is what each framework itself. And then surrounding it, there's a constellation of modules that come from EasyBuild.tools, which are basically tooling and utilities around the machinery in framework. And if you work like I do, you might find yourself in this maze at late at night when it's probably the worst time of the day to actually get into this kind of entanglement. So be careful with that. Get your coffee ready when you, if you plan to start working in framework. And let me show you a more comprehensive overview or a comprehensive guide of how framework is structured inside EasyBuild and what its role so that you can find yourself your way out of it whenever you need to. So as you know, EasyBuild is mainly composed by a trinity of three packages, which is EasyConfix, EasyBlocks and framework. So when you install EasyBuild, these are the three dependencies that will, well, the three main dependencies from EasyBuild that will be installed in your system as well. And I will assume that everybody's familiar with EasyConfix and EasyBlocks. So at this point, we'll know what those are. So I'll focus on framework. However, it's important to note that most of the people working in EasyBuild will probably come at first, not because of framework but because of the EasyConfix library. So they will learn that this is a nice software and to install complex software packages with ease. And that's what will draw them to our community. Unfortunately, our users are very good at finding all those softwares and libraries that are not yet in the upstream repository of EasyConfix. And then things become harder and they become harder, rapidly harder. So let's say that you have to install some software package that it's not yet in the upstream repo. There's no EasyConfix for it. So then what you will do is think, okay, I'll write an EasyConfix. I mean, EasyConfix are relatively easy to write. They are static files. So modifying EasyConfix, creating new ones is a very approachable task. So you will start by trying to find some other EasyConfix that is related to the software that you want to install. So for instance, maybe there's a ready EasyConfix for the software package, but for a different version or with a different tool chain. So you can easily modify that one to make your own. However, then you might find out that there's none. So at this point, usually the second recourse is to just look on the internet, just not for the EasyBuilder's repository of EasyConfix, but anywhere else in the Alliance or some other HPC cluster that publish the repositories. If that's also not successful, then the next step is to figure out how to install it yourself. So how to go from A to Z of the installation process and then translate that into a new EasyConfix that you will write yourself. And at that point, I mean, that depends a lot on what kind of software package it is, but you will see that maybe it's a software that has a CMake installation procedure or maybe it's a Python package or maybe it's something completely custom. So you might find examples in the AppStream repository that serve your purpose that you can just copy paste and transform for the software that you are targeting. But it might be also the case that your software is very custom or does not provide any standard installation procedure. And then in that case, you might be out of luck and you might have to write your own EasyBlock. Having said that, getting to the point where you have to write your own EasyBlock, it's rather uncommon. So you can do a lot of tinkering with existing generic EasyBlock. So for instance, the common copy EasyBlock or the make copy EasyBlock are very flexible. You can do a lot of dirty things with them, things that you might do and might not be easy to contribute AppStream, but that in case of need, you can use. But the best way of course, is to actually write an EasyBlock for all those installation procedures that are not already covered in EasyBlock. So in that case, you get into the phase two of the developer rabbit hole. So you already went to the phase one that is going through the EasyConfig and library and repositories. And now it's the EasyBlock library and the repository. So EasyBlock provides several EasyBlocks for different installation procedures. And if you need to add your own, you can do it starting from a vanilla EasyBlock, which will not perform any configure, build or installation steps. Or if your installation procedure is very close to an existing one, like for instance, let's say it's a CMake build step, but then the installation is something different. You could actually build your EasyBlock on top of the an existing CMake EasyBlock and then just modify the installation step, which is special in your case. And then in the EasyBlock, you will have all the tooling of frameworks to execute commands, to access the file system, check the dependencies. So all the tooling necessary to actually go step by step and fulfill all the requirements of the installation. And if that's enough, you might very well end up with an EasyBlock, an EasyBlock that works for you. And then you can contribute that to the AppStream repository. If you're unlucky, you might not have the tools necessary to do your installation procedure. And then you will have to modify framework. Or in our case, you might find a back in framework or in this tooling commands, like to access the file system, check dependencies, or you might, for some reason, you might be doing something in a new way and you might just reveal a back in framework. In that case, you will have to go deeper and jump into the, into the framework. And this is when you will find the maze of modules that I showed you before. And after this, all these phases in the right call, this might probably be at 2 a.m. At least for me, it's been multiple times. So of course, working with this complex diagram is not useful. So I'll show you now the structure of framework first. And it's main roles in the execution of Easeville and highlight the parts that you should keep in mind whenever you have to work with it. So the first thing is the workflow. So it's what happens when you execute EV to do with a simple execution command like EV install some easy config with the robot argument to find all the necessary dependencies in the repository. So in that case, you have Easeville main that will, first of all, parse all the configuration options of this session, of this Easeville session. So it will go through the command line. It will go through configuration files. It will determine all the configuration options that you've given to Easeville. Then immediately after, it will parse the easy config that has been provided in the command line. And once the easy config is parsed and validated, it will resolve the dependencies with robot, for instance. Because at that point, all the dependencies of the easy config are known. And then it will resolve the following. So it will resolve the necessary steps to fulfill this installation. How many packages do we need to install? And then it will go into the bill and install software step, which is also part of Easeville main. And then the bill and installation will recursively go through all the dependencies and the target packages. And for each one of those, it will initialize an easy block. And that easy block, the first thing it will do for each of the easy configs is to actually check in the easy config file, what easy block class is this easy config using. So as you can see, the easy config has a more lower role in determining the installation procedure and the easy blocks that will be used in the installation procedure. So the source of truth, let's say, for the installation procedure is the easy config file. And then easy block will adapt to it, initialize the respective easy block class, and then run all the installation steps defined in that easy block. And what you see here is actually all those installation steps. So this is what Easeville does from A to Z to actually get some, a single easy config installed. So what you see, we have divided this into three main sections. So that's kind of the first warm up setting up section where the steps there are basically preparatory steps. So the sources will be downloaded from the internet they will be unpack, patch, then the, and at this point, the build directory will be also created and set up. And once that's done, then it will jump to the installation and build, but to the build and installation phase. And that is the phase that it's more common to most of us which work with easy configs and easy build and easy blocks because this is the core of the installation procedure. This is where the compilation happens. This is when the installation actually happens where tests to get executed. And this is also one, the place that in both easy configs and easy blocks, we put more emphasis in configuring. So the pre-config option, the build tops, option pre-build tops, all those options that actually are used to make the installation succeed is in, take place in these three steps of the installation. Then there's extensions, which is an step that it's always executed, but only does something if there are extensions in the easy config. And finally, there's the dropout and phase where, and once the installation is done, so all the files are already in the installation directory of the software. Then is the build will just grab up and do post-installation commands, sanity checks, clean up the build directory, also generate the modules for that software installation and close the installation procedure. So what's surprising to me and probably to many of you is that actually most of this is handled by the base easy block class. So the setup and the grab up of the installation, which is most of the steps are just defined by the easy block class. And so when you want to develop your own custom and easy block, the only thing needed is to actually define a configure step, a build step and an installation step and optionally test it if you need to. But those are actually the only parts that you actually need to work on your custom easy block. Apart from that, there's a special case which is extensions, as I mentioned before. And those are handled by their own class, which is extension easy block, which is a combination of the extension and the easy block because software that uses the extension easy block class can be installed either as a standalone or as an extension. So for instance, the most common case, I guess that everybody's familiar with is Python packages, which can be installed, oops, sorry for that. I just jumped to the conclusions, oops, okay, here, sorry. So as I was saying, the extension easy block class is used for all those software packages that can be installed as either extensions or standalone. And the most common case, I guess, is the Python package and easy block. As Python packages are commonly installed in bundles of extensions or extensions of other software packages that use other easy blocks, but also on their own. So that's all you need to know about easy blocks and easy configs. And then there's two change, which is its own entity, which was also surprising to me. My understanding from a very high level perspective was that two change were basically bundles of modules. So you were putting in the first tool chain for the GCC module or OpenMPI and OpenBLAS. And that's it, so it's just a bundle of modules and that's what we call it a tool chain because it's at a lower level than the rest of the software packages. But actually it's not the case. So tool chains are their own entity and are more complex than just bundles of modules. So tool chains are defining the tool chain class, which already defines a bunch of methods to actually do the preparation of the build environment. So for instance, set environment variables and stuff like that. And then the tool chain will have components. So for instance, there will be a compiler component or the MPI component or the linear algebra component. And those components are all defined in the easy build tools tool chain package. And those are at a lower level that the actual definition of the tool chain. So these components have more logic to them. So they might, they will define some, for instance, some environment variables for in the case of compilers for that define the compiler. So for instance, the CC environment or the C flags environment variable for C compilers. They might also define some methods and to actually set those environment variables or to execute and to do certain checks on the libraries. For instance, the MPI component has a method to actually prepend certain commands with the MPI run or some other tool that is necessary to actually initialize the MPI context. And so these lower level modules with this logic define kind of this, that the type of component that we will be adding to the tool chain. And then afterwards in the easy build tool chains package, you will see, you will find the actual definitions of those components, which basically fill in certain settings that actually tell if the compiler is the Intel compiler or the GNU compiler and the same for the linear algebra libraries, open blast will define certain environment variables in a way that is different to flexing blasts and for instance, the linking flags will be also different. But for the rest, the structure of the logic is all the same. And when you have all of that, you combine that into your higher level tool chains like Gompe or FOS. And that's what ultimately defines the behavior of those tool chains. So if you need to implement and create your own tool chain, it might be very easy if the components are already there as you can very easily mix a match and different MPI implementations with different linear algebra libraries and compilers. I mean, those are just modules that you can combine. But if you want to develop your own and you will keep in mind that you will have to also think about writing down the logic of those components. And then finally, so this was for the main features in framework, I just skipped some, for instance, the easiest tags, I'm gonna skip that since it's experimental. But that's the main role of framework. So handling easy configs, easy blocks and the tool chains. And then around it, we have what it's called the easy build tools, which provides several features that are used across the code base. And you will see not only if you work in framework, but also if you work in easy blocks and custom easy blocks. So for instance, the first one is the configuration options of easy build. So how can you add a new configuration option to easy build? So configuration options are defined in the easy build.tools.options module inside an easy build options class that has the definitions of the different command line arguments and their help strings. And also in easy build, the tools that config, there's a singleton build options class that will gather all the build options for the current easy build session. So that's anything related to the compilation, for instance, or the different options of the tool channel, that stuff. And these options will be centralized in this class and are easily accessible through the build option method that will just return the respective configuration. And you will see this used across all most of the easy blocks and all the code base. So this means that if you want to add your own new option to easy build, you will have to add it in both places. So you have to add it in the options module and in the config module, so that it is accessible everywhere. Finally, additionally, we also have the different interfaces to the module tools. So in easy build, the tools that modules and there are the interfaces to the supported module tools in the system. So you will see an interface for LMOD, interface for Tical and others. This serves to abstract this interface. And this is used across the code base of easy build anytime that you need to check load and modules or load modules, this will be used. Also in easy build, the tools that module naming scheme, you will find the definitions of the different naming schemes supported by easy build. And finally, in easy build, the tools that module generator that contains the engine that it's used by easy build to write and modules. And you will find that I'm a class for Lua modules and another class for Tical modules. And that's basically used in the make module step of the installation. There are other packages in easy build tools that basically serve to interact with the host system. So for instance, if you need to retrieve information about the CPUs or in your system and CPU features, architecture, the OS, the operating system version, type of operating system, stuff like that, that's in system tools. In easy build tools environment, you will find methods to actually access the environment of the current session and also modify it. And in easy build, the tools that file tools, you will find an interface to actually handle file operations in the underlying file system. So copy files, generator of directories, that kind of stuff. There are many others. So I'll not cover everything, but probably the main ones remaining are easy build.tools.github, which provides all the codes providing the github integration. So all the methods to open PR, sync PRs, that kind of stuff. For instance, I found myself some time ago modifying easy build.tools.github because I wanted to use a different encryption method for my github token. So that's the place where you can go to actually do these kinds of changes. Another interesting one is easy build.tools.hooks, which can change the engine running with the hooks and also all the definitions of the hooks. So if you use hooks to customize your installations, you don't really need to play with easy build.tools.hooks as the hooks are already triggered in their own step, but you will need to come in here if you want to add a new hook to the system. So for instance, let's say that you want to, there's some step or I don't know, some place in the execution that where you will like to have a new hook to be run there, then this is the place to go. Otherwise, you might not need to go to check this spot. There's also easy build.tools.containers, where all the interfaces to the different container systems like obtainer, singularity are defined. And then there's also easy build.tools.job, which contains the interface to different job schedulers like slurm. And this is just for instance, because easy build can also submit jobs to the job schedulers if you want to run your builds in a job. So with all this in mind, you might be able to actually figure out how to handle either bug reports, bug fixes or development of new features. So let's say that you have found a bug, an issue, what to do there? Well, first of all, and reporting of bugs in easy build framework works in a very standard way as in any other bug tracker. So the more information that you can provide, the better, that's the main rule. So describe what you're trying to do, describe the steps that lead to that issue or bug, also provide the easy build configuration that you are using and which whole system you're running is a belong. Also, if you're using any questions. Alex, we're still seeing the tools of options slide. Is that the intention? No. So it's not changing. Now it's flipped to, yeah. So now we're at slide 21. Okay, that's the good one. Yeah. Okay, okay. So, so yeah, so this is for the reporting issues of bugs. So as I was saying, provide as many information as possible. So for instance, also if you use any custom easy blocks or custom easy config files that are not available upstream, also provide them. And also provide full error messages and trace blocks. That's always useful. However, in the case of framework, it's also the case that might not be trivial to figure out if the issue or bug and originates in framework or in easy blocks or in easy configs. As the traceback will probably go down to framework all the time, but that's something that might not be straightforward to figure out. In case of doubt, do not worry too much though. Opening issues in framework might be the best default option for issues and bugs. And if needed, maintainers can move those to another repository if they are clearly not originating framework. And finally, if you have a patch ready for framework and want to open a PR with it, keep in mind that the most important part is that all unit tests in framework must pass. So, and in the case of framework, that means that you will probably have to update the unit tests by yourself because let's say that if there's a bug in framework that you just found out, probably that means that either that error triggers on a combination of circumstances that is not currently covered by the tests. So that means that you and we will have to add tests to cover that circumstance or that the tests that are covering that circumstance are buggy actually, because that can also happen that the tests are not properly doing the assertions or not working as intended. And then in that case, the test will have to be fixed and updated. And in the case of new features, you will have to add tests to actually and check that new feature that you are adding to the easy build code base. So as I said, it's very proud that you have to write unit tests. We have in the documentation a page on unit tests. Please check it. Right now it only covers how to actually run the unit tests yourself and how to read the results of the unit tests. Writing unit test is more complicated, but that's out of the scope of today's talk. And we will at some point improve the unit test documentation to show how to actually write unit test on your own. But that's for another day. Also in the case of PR, keep in mind that the easy build framework is not integrated in the EB command and GitHub integration. So you cannot just open PRs with the EB command. And that's inherent to the complexity of framework. So the scope of framework is way larger. So it's much more difficult to define behaviors that can be executed with a simple command line. Having said that, opening PRs is not difficult. It works in the usual way. You can just fork the repository, create a new branch in your own fork, and then push your changes to that branch. And then from the same GitHub web interface, you can easily create a new pull request from your fork to the upstream easy build framework repository. So that's all I had to show for today. So to grab up, framework looks scary, but you can find your way around it. Basically, keep in mind the interplay between easy block and easy config. So easy config is the thing that defines what easy block will be used. Most of the installation steps are defined in the base easy block class. And the custom easy blocks only play usually with the configuration build and installation steps. That's also important to keep in mind, because for instance, if you're depending on what issue or features you want to develop, you might have to go to the easy build repository in case it's related to a configuration installation building an installation, or otherwise you'll have to go to framework to actually go to the base easy block class. And also keep in mind where the different features are provided by easy build the tools, because that also helps a lot. If you already know, for instance, that you have an issue with a configuration option and you know that it's in easy build the tools that conflict, regardless of where your trace back, for instance, are directing you, you can just, if you have this information, in your head, you can just directly go to the right place and that will make you gain a lot of time in troubleshooting issues. And as I said, contributing changes is a bit harder because there's no integration in the eBee command and also because you will have to take care of unit tests, but it's just a bit. It's not very bad and it's easy to learn and to get used to that. And finally, probably the most important bit of information in this talk is that all that I have shown you today here, it's already in the documentation and you can check it out in docs.israel.io slash framework overview. Finally. It will be fixed in 30 seconds, yeah. It was merged to the developer branch this morning, not to the main branch, so it's not live in the documentation. But I'm fixing that right now, it will be there. Okay, okay, okay. I was already happy, since it was merged, I was already happy with that. I didn't think that there was a deployment phase. Okay. And to your previous slide, Alex, there was actually integration for framework in the GitHub features as well. If you do eBee and UPR, you can actually do this for something like the file tools module in framework that works. Ah, that works. Ah, okay. I totally missed that, okay. Yeah. Okay, let me just finish because then there's only this and we can just go into the comments and questions. So thanks. I would like to thank my colleagues in BUVHPC, also the Israel community and the University of Brussels of CORT for hosting us and the Vlam Supercomputative Center, which is the institution that offers financial support to us. And thank you for your attention. Okay, thank you very much, Alex. To show that Alex wasn't telling blatant lies. Let's see if it's up there already because GitHub, well, refuses to push last minute stuff, right? So I'll just show the local render. Okay. I also have it there. I can share mine. Yeah, maybe you can show it. Then we don't have to switch screens. So this is it? Yeah, so this is what it looks like in the documentation. We're really finishing this up this morning. So it's basically a write up of what most, what Alex was talking about. So you will find it in the for developers contributors section and it's called framework overview. Maybe increase the font a bit on that. Oh yeah, sure. If you can. Yeah. Like that. So it's in the developers contributors section and you will find it in framework overview. And you will find a bit more that what I've shown today, but it's basically the same information structure in the same way. So the main easy build packages are in here, the workflow that I showed you. Also the stepwise installation with all the different steps, what is carried out by the easy block class, what is carried out by the custom easy blocks, also the interplay with extension easy block classes, all that you've seen. A little bit more, like for instance, the easy block, the easy stack class, but it's the same that I've shown in this presentation. I think this can be, well, it's already very good. It's very, well, let's say not very detailed but detailed enough to help you find your way around in framework. It could probably be improved, that people have ideas or stuff isn't clear that's currently there or will be there in a couple of minutes, please let us know. I think this is very good starting point to provide some more in-depth information on framework. Okay, so if anybody has any request to expand this documentation, please let us know. Also, if you find any mistakes also, that's very probable. As I said, I'm not an expert in framework, so there might be mistakes. Bart, can you come here and ask the question? I'm trying to avoid the echo, staying behind the speaker. Yeah, there's a little comment about the framework. One of my first contributions, the first one, the easy build was the PGI tool chain. And I still, when I work on the framework, you wanna get started with something. The tool chains are, I think, still the easiest bit of the framework. It's fairly straightforward part. Yeah, I'm just adding a new tool chain definition. If you wanna combine a particular compiler with an MPI and that combination is not yet supported, it's not extremely hard. You just copy an existing one, you change the copyright, you change the little components, and it's mostly a copy-paste exercise. Exactly, that actually means that there is a lot of duplication in that directory, but just compared to the rest of the framework, it's pretty easy. And the high-level stuff is easy in tool chains, the lower-level stuff, of how it's actually setting up the build environment, that's a whole different story. So that may be something we have to rewrite for, let's say, easy build six or seven at some point. Because the design is just weird on how that was done. Okay, now we have any questions, any remarks? For Alex. Anything remote, Simon? No, okay. Yeah, thanks a lot, Alex. So I'm very happy that you stepped up yourself to do a deep dive into framework after asking me for like two years or something to do it. But at least there was a collaboration, right? You were not doing this by yourself. No, no, no, no. It's been actually a very good exercise and I'm happy to have done it. Yeah, and making the extra effort to actually also cover it in the documentation is a very good move, I think. Okay, thank you very much. Thank you.