 So welcome to the first part of this easy build tutorial. This is the screen I'll mostly be sharing and looking at, but I'll flip things around a bit and make things full screen as it makes sense. I hope this is big enough fun twice if not you let me know. So on the top left, I have a couple of slides which I'll use to mostly drive the session today. But the biggest part of the contents will be on the right in the actual tutorial website. So I'll copy paste a link in the rocket chat to that as well. There are also mentioned in the slides which I've shared already in the in the chat. And I'll also have a terminal session here on on putty where I've prepared a small software stack with easy build so if you go to the project space. Of the Lumi preparation project, you'll see an easy build sub directory there and we'll be using some of those installations during the hands on but that's for later in the session. But I'll do now as I'll start with the slides and make that full screen actually like this is popping up full screen. Yes. Okay, so I figured it make it could make sense to do some short introductions first know there's a lot of people here. But I'm mainly interested in what kind of experience we have with easy build already since that will be interesting for me to know where to spend most of the time today. I'll start off myself very quickly. I'm kind of hostile work at Kent University since 2010. And easy build was thrown in my lap pretty much shortly after I started in the team it was already there so I didn't create the project. And gradually we've we've worked on it mostly for internal work and at some point we were happy enough with it to make it publicly available. And since then the community has grown around it and I've become I guess the lead developer community manager release manager, all of those things combined. Okay, I think we got everyone. Thank you very much. That gives me a good view. So most people seem fairly new and are aware of the high level concepts I think some have hands on experience some have not. Okay, that's very helpful. So for this tutorial. There's going to be four sessions. The first one is today, but the introduction and also a little bit of hands on already with installing configuring and using easy build. Then the second part is in two weeks. It says using easy build will do some of that today as well it was beyond the basics, a little bit. Rather than just using what easy build includes already how to add support for additional software yourself how to tweak existing things and so on, then more advanced topics. Think like using a custom module naming scheme hooks, submitting installations as jobs to slur. And then the last session which is not planned yet there's no fixed date yet for this is easy build on gray systems. And for this I hope to work together with the people from CSCS in Switzerland, which have a lot of experience with easy build on on gray mainly on their pit stain system with also on others. So I'm going to reach out to them to see if they have time to help out with flashing out that part of the tutorial since this is a new section. But yeah hopefully that will work out. The practical information for the tutorial as I said most of the contents is in the tutorial website, which is parked on GitHub. This is a loomy specific version. So, if you go to the tutorial, let me make this full screen. You should get this view and the part you're interested in is the last part here on the right. We did this tutorial for ISE last year as well at least that was the plan, because the tutorials were postponed. So we did the tutorial anyway in an online format outside of ISE. And this last version is a refresh of those contents so they're mostly the same certainly for today. But things have been refreshed and extended a little bit and the structure is a little bit different as well. And for the other parts, especially for the advanced topics and certainly for the gray section, there will be lots of new contents that will be added. This site is fully public so anyone else can see the contents of the tutorial. And there's an overview page here as well with just links to different sections together with the dates. So if you, I'll use it half screen here which gives you a hamburger menu here on the left. Whether you want to do half screen terminal website that probably makes sense for most of the session today. For the hands-on exercises, we will use Putty at CSC. So I think most people have an account there already. If you don't, that's not a big problem. You can certainly use your own local system as well. So there's nothing really special on Putty for this tutorial except for a small prepared software stack. So with the latest FOS 2020B tool chain in EasyBuild. So there's a couple of installations there just to avoid that you have to wait for an hour to build GCC 10 and then a couple of things on top of that. So that's already done for you on Putty. If you have something like this or exactly FOS 2020B already installed on a local system, you can use it there as well. So there's no problem. I will be driving the demos on Putty since I expect most people to be using that as well. One thing you will notice if you look into Putty is that there's a couple of modules that are preloaded and that give you a compiler and an MKL. For the most part, that's not a problem. But there are installations where that might cause trouble with the installation. So some software packages, if they detect MKL, they give that preference over everything else. And the tool chain we will be playing with has open blasts, which is like the second or the third thing some software packages look at. So I recommend you doing a module purge so the loaded modules are gone and you just have an empty environment basically. And maybe also, but that's certainly less critical, is to hide all the available modules on Putty. So we really start with a blank slate. The last part is less important. EasyBuild will probably not pick up those modules anyway. But if there's a clash in module names, something that exactly matches what EasyBuild would install, then you may get some weird behavior. So mixing modules that were installed with different tools is never a good idea. To install EasyBuild, which we will get to in a bit, I recommend using PIP3 to do the installation. So you'll be running EasyBuild also on top of Python 3. I noticed that Putty is still sent to S7, so it also has a Python 2, but since that language, I don't recommend using that anymore. EasyBuild works fine on top of Python 2 as well. You won't notice any differences, but still I recommend using Python 3. And the small software stack that's been prepared on Putty is in this directory, so there's a project space here that I understand everybody in the Lumi team at least has access to. With an EasyBuild sub directory and there you'll find modules and things like that, but I'll get back to this when we need to pick up on those installations. For today, this is the high level timeline and we're actually already a bit behind time. I'm not sure how strictly we'll stick to this time. It's flexible. If people want a break, just raise your hand or shout, and we can definitely have a short break. That's not a big issue. The idea is to spend most of the time in the basic usage part doing some hands on there. But we will also take a bit of time looking at installation and different ways of installing EasyBuild and also looking at the configuration part because that's definitely important. And where we put the coffee break or if you have two coffee breaks, that's fine. And hopefully at the end there will be plenty of time for questions, but also feel free to interrupt me if there are any questions that pop up during the tutorial. I'll try to keep an eye on raised hands, but if I don't manage to pick that up, please don't hesitate to speak up. We'll start with the introduction to EasyBuild. What is EasyBuild? This is just a list of the sections that are in the tutorial website and I'll actually use that as contents here. So I'll make this website full screen. I don't have slides for these different parts because I will just be mostly repeating what's already in the tutorial. So if you go to the menu here, Introduction to EasyBuild, this is the whole part we will cover today. And what is EasyBuild is the first section. So that's what I'll be starting with. I'll go over this fairly quickly. I don't want to lose too much time here. Very short, EasyBuild is a framework or a tool to install mostly scientific software on HPC systems. And it's a command line interface, so it's not a GUI or anything like this. From a very high level. It's a consistent way to install scientific software on HPC systems, so you're always talking to EasyBuild. And however, the software is supposed to be compiled, whether it's CMake or whether it's Bazel or some other unholy build tool. It doesn't really matter because you're always talking to EasyBuild, so you have a standard way of installing software. And as a result of that, it's also consistent. It doesn't really matter who does the installation since EasyBuild does all the work that you get consistent installations across this stack. And yet you're talking to a uniform interface, so that saves a lot of time. It takes care of a lot of the boring work for you, like unpacking door balls, downloading files, applying patch files, so all of that grunt work you don't have to do yourself. EasyBuild doesn't need any special rights to use it, so it doesn't acquire admin privileges. It actually actively discourages you from running it as root. And as a result, researchers can also use EasyBuild themselves in their home directories on clusters. We certainly see a lot of that in Gantt, where people either just manage their own software stack on top of what we provide centrally, or they actually figure out the installations themselves and then they give us an easy config file and say please install this centrally so everybody can use it. And of course we will review there. But yeah, they sometimes take some of the work away from us. And over time EasyBuild has become a community project, so lots of HPC sites are collaborating on it, not only using it, but also contributing back changes and enhancements to the tool, so that's very nice. The key features, the main one is of course fully automatically installing scientific software, even when it involves interactive installation or configuration tools or applying patch files, so it does all of that by itself. It also generates module files for the installations it does and when you tell it to, it will make sure that all the required dependencies are there before actually installing the package you're interested in. Admin privileges are already mentioned that it's highly configurable so we try very hard to not have any hard coded things in EasyBuild, and whenever something is hard coded and somebody bumps into it, we probably make it configurable as a result. So you get a lot of configuration options, which is a bit daunting at first, but that does mean it's highly flexible. And there's three configuration levels, you can use configuration files, you can use environment variables, or you can use command line options and we will look at that in detail today. You can also extend the capabilities of EasyBuild through a sort of plugin system for different things, for telling it about different compilers, different tool chains, support for other software, or even customizing what it does by default. So if there's something you're not happy with and it's a very specific thing to your site or the way you install software, you can use hooks to customize the behavior and we will cover that in the advanced part of this tutorial. Everything that EasyBuild does is totally logged in log files, even to excruciating detail if you wanted to, it has a debug mode as well which makes it a lot more verbose. But that does mean that after the fact, even weeks or months later, you can figure out what happened and also why it happened and that may be important if problems with installations pop up later. It's also transparent, so you can figure out what it's going to do before it will actually do something through a drive run mode and while it's running you can actually, you can also enable trace mode to get more information about what it's currently running. So rather than just building, it will tell you which command is running in which directory it will give you a log file that you can tail so you can see progress. You can use a custom module naming scheme which we will cover also in the advanced part, even hierarchical module naming schemes and what that means I will explain them. And it also integrates with various other tools like slurm so you can submit installations as jobs into a slurm resource manager. And this is often how we install software for a new cluster for example we just say EasyBuild please install everything that we've installed in the last two years. And it submits jobs to slurm, one job per installation, it sets dependencies between the jobs. And so over the weekend we can populate a new cluster with software that was installed in the last one, two or three years, very easily. It has some support for creating container images as well but I would say it's rather minimal support and it's experimental support so we don't consider it stable yet. You can create packages like RPMs for installations that were done by EasyBuilds through integration with the FPM tool, which is a tool to create WM files, RPMs, all of these things from existing installations. It's definitely not covering everything here so it also talks to other tools. Can you repeat what is it you said is experimental and not yet fully developed support? The support for, so EasyBuild has some support for generating container recipes so a singularity file or a Docker file, which you can then use to build a container that includes the software that you want and it will use EasyBuild to install that software. There's some support for that, it works, but we're not fully happy with it and it's more experimental. So that means if you try to use this, EasyBuild will tell you this is experimental and you have to tell me you're okay with using experimental features. An experimental means that we're not happy with the current implementation. It may change over time so we're very careful with making breaking changes in EasyBuilds, but flagging it as experimental gives us the flexibility to change it as we see fit. It's a sort of warning flag to people who use it and only when we consider it stable enough, we will mark it as non-experimental and you will not have to use this flag anymore. There's currently not a lot of development done on this container support, mostly because of a lack of interest in the community, but that may change at some point. So what is there should work? If it doesn't, we will probably fix it, but finding time to extend those capabilities is a bit of an issue. I must say I have been using this container support with Singularity and I didn't find anything that won't work, so it may be experimental, but it works quite fine. Okay, that's good to hear. So what is there? We properly test this and it's also tested in CI and stuff, so if we break it, we will notice before we do a new release. But there's probably aspects to it that we're not happy with. I think it's only working with a YUM-based operating system in the container image. For example, that's something we would like to change and break it open a bit, which needs a bit more work. But yeah, overall, certainly with Singularity, it should be working darker, it's less mature than the Singularity support. But yeah, good to hear it's working for you. Yeah, just one very basic question about this part before we leave it. So it is the way that you install the software in a container and then distribute this as a container to the systems or you bring the container and provide access to it or to the software through a Singularity-based and easy-build-based module. So what we do in the support we have in EasyBuild is to EasyBuild generates a container recipe for you. So basically the script that you would use to build the container image and inside of this recipe, EasyBuild itself is used. So say, if you tell to EasyBuild I want a container image for TensorFlow, EasyBuild will generate a recipe to install TensorFlow in the container image. It will use itself to install TensorFlow in that container image. And then either you say, I just want a recipe or I want the actual container. And when EasyBuild is asked to create a container image, it just calls out to Singularity, what is it? Build or Singularity build, I think. And it gives it that recipe, Singularity builds the container image for you. So it's basically preparing everything and just passing it on to Singularity, which then should give you the actual container image. Yeah, this is well covered in our documentation. So there's a separate section on how to build container images with EasyBuild. And it has, I think, clear examples as well on how to use it. The main focus points of EasyBuild, so what do we mostly pay attention to when developing the tool and adding features to it. So it was built from the start to install scientific software on HPC systems. Then, of course, the first thing you worry about is performance. So whenever we can, we build software from source. We don't get the option unless the sources are not available or there's a good reason not to do it. The main reason we do this is because then we can optimize the installation for the processors on which the software will be used. By default, EasyBuild will use the MRCH native compiler option when using the GCC compiler or the dash X host option when using Intel compilers and so on. So by default, it will build for the machine on which it is running. But you can change this. So we have a configuration setting to change these flags. If you want to do cross compiling or you want to compile only for AVX2, even if you're on a Skylake system, things like this you can control. But at least by default, it will always optimize for the system it is running on. Reproducibility is important as well. So whenever we do an installation with EasyBuild, we try to make sure that you can do the exact same installation again later. But just redoing what you did before, even if it's months later. So this is reflected in a couple of ways. The main way is there's actually two ways this is done. So we try to take full control over the compiler tool chain. So we deliberately don't use the system compiler, but we build our own, usually GCC compiler and then maybe install Intel compilers on top and then go from there. So we know exactly what kind of compiler we are using, which version, how it's configured and so on. I presume that the system is not capable of doing binary reproducible builds. It's out of scope of EasyBuild, correct? Yes, we don't go that far. So with reproducibility, I would say it's best effort reproducibility. There's other tools that go very, very far to try and reach binary reproducibility even to the extent where they reset the time to January 1, 1970, because time stamps can affect exactly what kind of binary you generate. We don't go to that extreme. So it's like a middle ground between controlling the full installation process and not going to the very extreme of trying to get an exact same binary. All right, thank you. So next to the compiler tool chain, so that includes compiler and libraries like MPI, BLAST, Layback, so all of these are installed by EasyBuild and controlled by EasyBuild, so we know what we are using. In addition, all the dependencies that are specified for a particular software package have fixed versions, so whenever you need Python as a dependency, the recipe, the easy config file that you will use to install that software will say, I want to use Python 3.8.6 and this specific version, and then EasyBuild will make sure that that version is installed and is in place before it starts working on the software itself. So there's some exceptions to which dependencies are installed through EasyBuild if there's good technical reasons to do so. So OpenSSL for example is something that EasyBuild is aware of, you can install OpenSSL with EasyBuild if you want to, but usually we don't rely on an OpenSSL that's installed with EasyBuild whenever it's needed. We have a kind of marker that says this software package needs OpenSSL and it's expected to be there, and EasyBuild can check if it's there, but it will not, at least in the easy config files we use by default, it will not try to install OpenSSL itself. That's mainly done for security reasons, so we want to make sure that you always have a secure OpenSSL that you're using. Other things we typically don't install or can't install are InfiniBand and GPU drivers, since these are closely connected to the Linux kernel. So we only use the runtime stuff like CUDA, the CUDA runtime library that's installed with EasyBuild but not the CUDA or the GPU drivers. The fixed software versions we will definitely get back to in the second part of the tutorial when we start writing our own easy config files that will become very clear. And then community effort, this was initially not the plan with EasyBuild. Even when we released it publicly we never tried or wanted to build a community but it just happened because apparently the tool was interesting and useful enough that people started picking up on it and even sending us back things even though we were not asking for it. And over time it has really grown into a quite big community with hundreds of HD sites around the world. And basically people working on it around the clock from Singapore to the US and New Zealand. We have people using and contributing to EasyBuild all around the world. Over time we have learned that not everybody who is using EasyBuild or can contribute to EasyBuild is very familiar with Git. Maybe nowadays it's a bit better since it's been there and it's a quite standard tool for quite a while now. But certainly several years ago many people were not familiar with Git and were wanting to contribute back were kind of stuck. Because of that we have GitHub integration as well where you can straight from the EasyBuild command line make contributions without even looking into GitHub or clicking around in GitHub or figuring out Git push, Git branch, all these things. So you don't have to run any Git commands or you don't have to click around and GitHub, EasyBuild does all that work for you. That's very good for contributors. It's also very good for the EasyBuild maintainers because the way that contributions are made are kind of uniform. So they always have a certain structure to them and certain things are always okay because EasyBuild takes care of them. And that's just things we don't have to have to look at. Things are in the right place, things have the right names. That's all done automatically and we don't have to worry about that. And hopefully we'll have time to look at some of that in the advanced part of this tutorial as well. Then just to clarify what EasyBuild is not. So we don't try to be another build tool like CMake or Make or Bazel or any of these things. It's basically a wrapper around everything that's out there and whatever the software developers want to use to build their software that's absolutely fine. EasyBuild just automates that the way for you. So whenever software is using CMake, EasyBuild will just run CMake for you. We're also not trying to replace traditional Linux package managers like Yum or Apt or anything like this. You will still need to use those to install some things that EasyBuild relies on, like Python, like maybe Lmods, things like Gillip C will still come from the OS, so we don't install our own Gillip C, at least not today. OpenSSL drivers for InfiniBand and GPU, so those will still be installed mainly through the standard OS package manager. So you will still need that as well and we're not trying to replace them. And also, at least not today, EasyBuild is not a magic solution to all your, certainly not to all your problems and even not to all your software installation problems. So you will still run into compiler errors, certainly if you try to use new compilers or if you try to use software and update it and move it to a new compiler toolchain, things may still be broken there. The nice thing here is that since it's a collaborative effort, hopefully somebody else has already taken care of that for you and maybe done a lot of work with patching it or figuring out the right compiler options to make it happy. And if that has already happened, you will not have to worry about it. EasyBuild needs three things, since we're very focused on HPC systems to use it to its full extent, you will need a Linux system. So the core of EasyBuild and the Python code certainly works on macOS as well, but you won't get very far, at least not with the available EC conflicts we have. So even building GCC with EasyBuild on macOS is probably not going to work. We were not very interested in making that work, since we have a strong focus on installing software for HPC systems. That could change over time, but right now there's very little effort being spent on that. EasyBuild itself is implemented in Python, it's still compatible with Python 2.7 and any Python 3 version newer than or equal 3.5, it actually still works on Python 2.6 as well, but it's deprecated and we will stop supporting that at some point. And of course it's recommended to use Python 3 for running EasyBuild. For example, in Python you also need an environment modules tool, so the L-models is certainly the most common modules tool that's used nowadays, it's also used on PuTTY. But other tickle-based implementations like the revamped environment modules version 4 is also supported by EasyBuild. The hard requirement EasyBuild will not work without having an environment modules tool, so it relies on this for loading dependencies and all of that. The development of EasyBuild is done on GitHub, it's open sourced under the GPL version 2 license. There's an organization on GitHub where there's a number of repositories, the main ones are the ones listed here, so it's broken up largely into three parts. The EasyBuild framework, which is the core of EasyBuild, EasyBlocks which are Python modules build scripts for particular software packages, and then EasyConfig files are the recipes that specifies the versions which tool chain to use with dependencies, and dependency versions to use, and maybe some more things like configuration options. The EasyBuild repository has the EasyBuild website and the documentation, and then the tutorial repository has this tutorial. There's some other less commonly used repositories on the side as well. It's very actively developed, we merge about just short over 3,000 pull requests a year. So there's a lot of stuff coming in every day, and we try to push out regular stable versions of EasyBuild roughly about every six to eight weeks. It's more leaning towards eight weeks recently, but there's certainly regular releases several times a year. And all of these are available on the Python package index, which is the standard platform for releasing Python software. Whenever we do a release, or actually also during the development, we do comprehensive testing, so we have a quite extensive test suite in each of these three repositories, which is run through GitHub Actions. Whenever pull requests come in for new easy config files, we do actual testing of those installations on HPC systems or on the MFARMS, and we report back on that in the pull request. EasyBuild can do that for us, so we can tell EasyBuild, please test this pull request and upload the test report back into the pull request. So the contributor knows that it worked for other people, the maintainer who will merge the pull request eventually has a good view on what kind of systems it works, and so on. And then whenever we do an EasyBuild release, we try to do a full regression test, which means everything we support up until that point, we reinstall so that's typically about 10,000 installations. But since we are all sitting on top of large HPC clusters and EasyBuild can submit installations as jobs, that's not really a big issue. So we just shortly before the weekend we push all those installations in the job queue and with a little bit of luck on Monday morning, we have the full result of the regression test. So we're quite serious about making sure that things that work keep working, and whatever we ship is also working as expected. Kenneth, can I have a question? Sure. Do you have also tested integration with GitLab? In what sense? The answer is no, but maybe you can clarify the question. There are some differences in how this CI pipeline works in GitLab, not exactly the same but very similar as for GitLab. I'm asking because many centers have own hosted GitLabs, and it might be helpful to have also integration in this direction, but I think it's very similar, just some details are different. But okay, if not then someone can just experiment with this, thank you. We haven't looked into this and I don't specialize on GitLab pipelines quite a lot, even though the development is done in GitHub, and I'm sure they have a good reason for that. We haven't looked into this yet, so what we use GitHub actions for is running our CI tests. There were not really building any software. We're running checks on the easy config files, like are all dependencies available, or are things consistent, are there any syntax errors in the easy config files? The functionality in the framework, like running commands, applying patch files, so we have simple tests that check for specific functionality, all of that is done in GitHub actions. Whenever we want to test actual installations, we do that typically on HPC systems or on your own laptop, and that's where the GitHub integration comes in. So usually we tell EasyBuilds, pull from this pull request, do the installation, and then report back whether it worked and if it worked on what kind of system it was working. I will show this in detail in the advanced section of the tutorial. But that's sort of our replacement for GitLab pipelines. It's not fully automatic and we're actually working on making it fully automatic, so whenever a pull request is opened and a maintainer has looked at it and says, this looks okay to me, we would actually want to have bots that say, okay, I have this approved pull request, I can test this on Skylake or I can test it on ARM, I can test it on Power, and then have a better automatic workflow of that. Some of it is automatic now, but usually it's humans triggering the tests. Thanks. Yeah, it's different from what SPAC does. It's less automatic and we want to make it more automatic. That's one of the main things we can win a lot of maintaining our time on, but it's not there yet. Then the community, I don't want to spend too much time on this here, but it has been growing and it's still growing. We're up to a point where we do yearly user build user meetings since 2016. The one we had this year was online, of course, no surprise. It's all around the world. So this is an overview of where the easy build documentation was visited in the last year, and how often so the bigger the circle, the more hits we get from that part of the world. And you can tell, yeah, it's everywhere in Europe and lots of places in the US, even down to New Zealand in Asia. So we're doing quite well. This is a short overview of the couple of sites that use it and have been using it for a long time. Many of these will be familiar like the Swedish consortium or the University of Oslo and so on, but also big companies like Microsoft are using it internally and are even talking about that and making it public. If you want to get help on easy build, there's several ways. So we have extensive documentation, which is part on GitHub, but since recently has a nice URL docs.easybuild.io. This is quite extensive. It's a bit maybe disorganized or not very well structured and that's something we hope to work on in the coming months. We want to move towards the documentation system that we're using for the tutorial here, just MK docs, which not only looks nicer, but it is also easier to structure. And you can actually have some CI for your documentation to make sure you don't have broken links and things like that. So we'll probably move to that system quite soon. Asking for help on GitHub makes sense, especially for reporting problems and bugs, but also just for general questions you can open an issue in one of the easy build repositories. We prefer getting issues in the right place. So depending on what kind of aspect you're having trouble with. So if you're not sure what the right place is just aim for maybe the easy build repository and maintainers can move issues to the right one transparent lease that's on the big problem. And there's a search box there on the left as well, which is quite useful. Whenever we have any problems that pop up, particularly compiler errors or any errors that that pop up during the installation. So I'm sure we mentioned the full error message in the description of the issues, which means if you're looking for error messages you may have a good chance of finding a hit somewhere in one of the issues. So rather than saying something went wrong and I tried this and it didn't work. We want, we try to make sure that the error message is also mentioned in the issue which is very useful over time. So active mailing list, which you'll need to subscribe to to post messages. These days it gets about I think 600 messages a year so it's relatively active but it's getting less active over time. Mainly because we have a slack channel now as well which is growing and growing fast. I think in the last year it almost doubled in terms of the number of people who are in the slack today we have over 400 people there. So we can be awake, which is useful for a quick question or, or getting some quick help. You can ask an invite to join the slack yourself via this link. And we still have an IRC channel as well that's how we started out back in 2012. There's a lot of people there. And there's also a bot there that whenever somebody asks correction in IRC we actually also see it in slack and we can answer them from slack so we don't actually need to be an IRC ourselves anymore. Next to this we also have conference calls every other week on Wednesdays on two different times so in the morning European time and then 5pm European time and we switch between those to cover different time zones. These calls are open they typically take about one hour. We've given overview of recent developments and then open the floor for any questions or problems or thoughts related to easy built and we've been doing these since November 2013. I don't think we've missed more than five over the scope of the years so that's a very nice place to just listen in or ask any questions you may have. So that was the what is easy built part and I don't know if there's any more questions before we continue with some more specific terminology. Could you paste the easy builders GitHub URL to the chat please because easy builders that GitHub that IO just redirects to easy build that something. You want the GitHub organization or the one that you currently have the URL open for the tutorial. Okay I think I already pasted it before. Okay yeah not here in Zoom maybe okay I did paste it in the Rocket Chat. There is a special channel in the Rocket Chat for this tutorial and the link is also in the link to that Rocket Chat channel. Okay yeah I'm retarded sorry got it. That's because then after the tutorial when the Zoom chat is closed we can still continue if needed. Okay thank you. Any other questions or shall we continue to terminology? I guess just a short question about let's say the future of easy build and development plans and so on. Can you say something about that in which direction is easy build. Moving are there any sort of major philosophical changes or what's the sort of upcoming things or is the focus more to improve the features that are already there or the functionality. That's a good question it's a bit difficult to answer that on the spot maybe. There's certainly some things that I'm personally not happy with and would like to find time on to significantly improve. But one thing I'm hoping to find time a lot of time for is the documentation revamping the documentation a bit cleaning it up. Making sure we cover everything rather we know of some things that are not covered or not well covered in the documentation that's really something we want to fix in terms of easy build itself and features. Good question. A lot of it is is like saying ongoing maintenance work and and keeping up certainly with with the changes in the HPC landscape in terms of platforms so different architectures arm and power. Maybe eventually risk five were a little bit worried about one thing we're also working on but that's a side project is the easy project the European environment for scientific software installations. Where easy build is one of the main components. So there we're trying to take sort of take easy build to the next level. In the sense that we're not only working together on a tool to facilitate the installation of software, but actually work together on the actual software stack itself. So here we have ways and you've seen the presentation Peter. There we have ways of making sure that we take away the differences between HPC systems that make it hard to have a shared software stack. And work on that so that's that certainly a focus for at least some of the easy build community not everybody. And that I would like to find time for some cosmetic changes sort of so making the command line interface and the way it produces output a bit more attractive. Things like colors and making it a bit more dynamic it still looks like it was created in 2010 which it was. So that that could maybe be smoked up a bit. In terms of main functionality and whatever is needed to get software installed, I think that's mostly there. So big holes and in there, maybe the container support if a lot of people care about that that could probably use some more work to make it better. But we don't see a lot of uptake on that in the easy build community or at least I don't have a good view on that we don't hear a lot of people complaining or asking for more features there. Because we're always open to ideas so if there's something now an easy build that you say, okay, this could be a lot better or it's really missing this particular feature. Please tell us, maybe we haven't talked about it or maybe it's something that we know about but that is not really on the top of our to do list maybe it should be so any feedback there certainly welcome. So I think maybe support for bliss or change to bliss that was something that's custom and not really a framework change. Yeah, so the tool chains part were indeed considering I consider maintenance work actually and it's not really in a landslide shift and what we do in easy build, but we are actively looking at the bliss library as a replacement for open blasts, since we've seen some some very promising performance results, especially on AMD CPU systems. So the the fast tool chain we have and I'll get back to that as an open source base tool chain so GCC open MPI open blasts, all these libraries open blasts works okay, but what we've seen from bliss is very promising and we may switch away from open blasts and into bliss where we've been in close contact with the lead developer there, and there's an ongoing effort, sort of working group to evaluate bliss does it indeed perform better also on Intel systems. Does it pass all the tests so if we build software on top of bliss is it actually performing better so not just micro benchmarks the actual application benchmarks. Is it working correctly or are there any issues that that could block us from moving to place and so on so that's an ongoing effort, but I consider that maintenance work not really any shocking new feature in the easy build. Okay, if there's no more questions and there will be plenty of time for questions at the end, hopefully. I want to go over some of the terminology we have an easy build so over the years we've built up some vocabulary for specific aspects or specific parts of easy builds. And we use certain general terms like dependencies tool chains and modules in a specific way that I just want to clarify here. So to start things off the easy build framework which I already mentioned, which is being developed in this get hub repository is a collection of Python modules which are fairly nicely organized in packages. The easy build tools package for example has a lot of functionality for things like running bash commands is here in this run module. We have a file tools module for things like applying patch files or reading files writing files. So there's a lot of functionality in here, but all of this functionality is general so there's nothing specific, or at least there's very little that's specific to particular software there. Things like running commands applying patch files, but not like. What do I need to do to install TensorFlow the framework doesn't know about that, but it provides all the functionality for doing that and other Python module that leverages framework. This is definitely the hardest part to get into if you want to make changes here. I've heard good feedback from people who are new to the tool that it's relatively with a bit of work it's well structured and it's not that difficult to find your way around them. The, the Python codes that we use here is not very, I would say advanced in terms of Python features we use so if you're familiar with other programming languages it's probably not too much of a jump to understand what's coming on here and I've heard people use these contributions by copy pasting other code snippets and changing them as needed, rather than really understanding what they're doing from a to Z in Python so I consider that good feedback. Easy blocks are then where a specific software installation procedure is implemented. So running configure make make install that's done in an easy block. We have a custom easy block for installing TensorFlow a custom easy block for installing open foam. So all of these things are softer specific in some sense, either very specific to a particular software package or specific to a particular installation procedure that's standard in some So we have a generic easy blocks for things like configure make make install or scenic make make install or installing Python packages or software specific that are TensorFlow warf open foam, all these complex things, complex things typically have a specific easy block. All these easy blocks leverages leverage the functionality provided by the framework so in some sense they're a plug into the framework they leverages everything that easy build provides in terms of functionality and puzzle things together to get something specific generic easy blocks software specific easy blocks I already explained The exact way that an installation procedure is performed is not fully determined by the easy block it has some parameters that it takes from an easy conflict file so we call these easy conflict parameters things like the version of the software or the exact the exact list of dependencies configuration options so these are not typically hard coded in easy blocks, but they come in through a parameter mechanism. So the easy blocks we have are in a separate repository on GitHub we split out these repositories because they have a different rate of development, we get way more pull requests in the easy conflicts repository that we get an easy blocks and framework is way more specific and maybe a bit more work to get into. So that's again a different repository. There's more about these parameters that that steer the actual software installation and that are used by the easy blocks to perform an installation. Some of these easy conflict parameters are mandatory. So obvious things like the software name and the software version. You have to tell easy build what you want to install. Which tool chain will be used, whether it's a GCC based tool chain Intel based tool chain, or gray or whatever. That's also specified in the easy conflict file through a parameter. Two things we always require are the homepage of the software and short description of the software. And both of these are used in the module file that is generated by easy. These are the only parameters that are really mandatory, which may seem a bit weird because you could say, Okay, if you don't have source files, how you can, how can you install anything. But there are situations where the source file is not really needed so we have a way of bundling things together, which is just a very lightweight, lightweight collection of other things and you end up with a module file that just loads other modules and in that case you don't need the source file. And therefore, there's corner cases or specific use cases where you don't need specific parameters. Of course, many of these optional parameters these other optional parameters are very common, like sources or like the location where sources can be downloaded. The list of dependencies and a separate list of build only dependencies. And for the configuration command options for the build command options for the install command so these are very commonly used as well to have full control over the installation procedure as easy build will perform it. Another one is which specific easy block that should be used to do an installation so this is optional because by default easy build will try to find an easy block for the specific software name. If you say name equals TensorFlow, the first thing easy build will do is look try to look for a TensorFlow easy block. If it cannot find it, or if there's, there is no such easy block then you have to tell it with easy book to use. For example, well, install TensorFlow as a regular Python package that just does the install, which you typically don't want for TensorFlow but you get the idea. And we will see this in the exercises a little bit today as well and certainly in the next section on using easy build and writing our own easy config files. This will become a lot more clear. Yeah, a small distinction here there's general easy conflict parameters like all the ones I mentioned above. But for some easy blocks additional easy conflict parameters can be can make sense as well. For example, when using the Python package easy block to install a Python package, you have a parameter to control whether or not it will use pip. Whether or not it will check on some things that are specific to Python. And those wouldn't make sense in other contexts. Then easy config files are basically a collection of easy conflict parameters that are defined. Key value type of definitions, which are collected in a simple text file the text file is actually written in Python syntax so you can use values like Python dictionaries or lists or things like this that works and to some extent you can and we allow using the Python construct as well like list comprehensions or string, string replacement and things like this so it's not really a good practice to do this extensively so we don't want you to do import OS and start reading environment variables or something like this and easy config files, but to some extent it may make sense to do this. So, from the start we decided not to use our own small domain specific language or use something like Jason or the animal, which were back in when easily was created were not very popular back then yet because they all have limitations that were not happy with Python syntax is, since everything else and usable despite and it's relatively easy to get into, and it gives you some flexibility that comes in very handy in certain situations as well. So, at some point there was a plan to move, move away from this and go to YAML syntax but that's that development halted because I don't think many people were actually convinced that it was a good way forward. So, you know, you really problems with having these easy config files in Python syntax. So they define a set of easy config parameters that then steer the actual installation, as it is implemented by the easy block. The file name of easy config files is important in some situations. So typically you have to dot eb extension for easy config files, just to easily recognize them. The name of easy config files is usually something like this so software name dash software version. If there's a particular tool chain used we add the tool chain in the file name as well. And in some situations you have an additional suffix which we call a version suffix, and the name of the file as well. It's not always needed to have this exact file name so this is only needed when easily able to actively search for easy config files itself, since it does this on a file name basis to avoid reading every possible file it runs into. And we will see that in the basic usage today. So, you know, easy configs are in a separate repository. And this is where we get about 2500 pull requests a year for new easy config files new software updates of existing easy config files or bug fixes in existing files. And a new concept is easy stack files. So, if you want to want to control a full software stack with easy build right now, we don't have a very good mechanism, or at least not a very mature mechanism for that. And this is what easy stack files will become in the future. The implementation is already there the basic implementation, it's still marked as experimental, because we're not happy with the implementation at least in terms of what it supports, what is their works. We want to extend it and because we're not sure whether the additional features we want to implement will require us to make some breaking changes we have marked it as experimental. Easy stack files that the best way to show this is through an example. So I'll jump in the documentation. So this is a particular example of an easy stack file. This is where it says, I want to install the software packages bio conductor, easy build itself grow max open from an hour. I want to use these two chains for bio conductor and in this case they are all the same but they may be different of course for different software packages, and I want to install these versions. This is an example of an easy stack file that's already supported we want to extend this a little bit so for a particular version you can say okay if you're installing bio conductor. In this version we want to change this configuration option to another one and you will be able to do that and an easy stack file. So this will be become a single place where you describe your whole, or you can describe your whole software stack, and maybe you'll have multiple easy stack files for different tool chains or different software packages so you will be able to combine this in all kinds of ways. The reason we do this is this single file will replace a long command line like this. Pass all the easy config file names all the time to the eb command you want to have a single file, where everything is listed that is able to install. So that's, I didn't mention it when Peter was asking the question but this is one of the things we are actively working on to make this better and more capable going forward. Extensions are specific term we, we didn't came came come up with it but we use it in a specific way. Extensions are any software packages that are installed on top of something else. So common examples are Python packages you cannot have a Python package without having Python. We call these extensions to have a single term to refer to all of these type of software packages. And we have, you can install extensions in two different ways I just stand alone so as a separate module. So that module you'll typically load Python or our or plural along with it, or we can install an extension together with another module so if we install Python. We also install, I forgot the exact number but something like 40 additional Python packages from the pi pi library, and we add those into the Python installation so we do with batteries included installation of Python. And for R that's a lot more extreme we install our itself and probably close to 1000 our libraries from the CRAN library and most of the easy config files we include so we see our is just the base and the real value is in the actual libraries so we install lots of these together with R and whenever a user does a module load R they actually get a whole rich environment to play with where most of the libraries that they will use are just readily available as well. So that's that's extensions. Stand alone or as a part of a separate package like separate module like Python, or you can also just have a bundle of extensions where. You can only really use them if you also load something else with them and that's usually done automatically so there's actually three ways of installing extensions. And also this will become clear hopefully in the. In the examples, especially in the second part in using easy build will will work with. A small software package and then adding a Python, Python bindings on top of that we will do that separate or as a part of the installation will play with that a little bit. And also useful for Julia. Eventually yes so Julia also has the concept of packages. And there we will use the same mechanism of having extensions we don't have very good support for Julia yet today. Because we're one of the issues there is we're not happy that that Julia tries to control the whole software stack so it installs its own version of a blast library and all these things. And we want to we want to peel that apart a bit and be a bit more in control. We've been in touch with some of the Julia developers over that we need to pick up the discussion again and see what makes sense for us to control and what they really need or want to have control over. And find them a good middle ground of that. And then on top of that will will look into adding support for installing Julia packages as extensions which is apparently also not that trivial, because they don't have a very good way of doing that in the way we are used to with with the Python R and Pearl. So that's an, that's an ongoing discussion. But yeah the concept may also make sense there. And also for other languages. So camel and there's other examples where we have support for installing extensions dependencies probably don't have to explain to you what dependencies are but just the way we usually treat them in an easy build is like this so of course the dependency is any software that is required by something else, or that can enhance other software. So the technical point of view I think there's there's three different types of dependency there's a build dependency so something like CMake which you only need when building and installing a software package. A runtime dependency so something that you need when you want to use software so if you want to run a Python package you need Python. And there's also another type of dependency a link time dependency which strictly speaking you only need when linking the software so that's part of the installation procedure. But depending on how it's linked it may not be needed at runtime anymore. At least the module for it may not have to be loaded at runtime when you want to use the software. So for example here is open blast if you do a static link of open blast then okay you don't need it at runtime anymore because it's included in the binary. Or if you do an R pot link to open blast and the binary knows where to find the library and you don't need to load the module to have it available at runtime anymore. So that distinction right now is not really made in easy build. So runtime and link time dependencies are thrown in one to into one big bag. But that's something that may that may change so we're starting to pay more attention to that. We have very good support for our part linking now which is something we're using in the easy project. And eventually, so this is one of the things that I could see change in easy build five that we start making this split between runtime and link time dependencies. Then tool chains what we refer to as a tool chain is usually a set of compilers typically CC plus plus and Fortran, sometimes also CUDA for GPU software. Together with additional libraries common things and HPC for MPI blast layback 50 are usually included in a tool chain. Whenever we refer to a full tool chain, it means we have both compilers and all of these MPI blast layback and FFW. That's what we call a full tool chain. A sub tool chain is only a subset of tax, for example, only compilers or only compilers and MPI is a sub tool chain. And there's a special case of a tool chain, which we call we now call the system tool chain this used to be called the dummy tool chain. So whenever you see an easy company file that says I want to use the system tool chain to install this, then you're basically using whatever is provided by the operating system. We try to minimize the usage of this because that means we're at the mercy of the operating system. We could be using DCC for which is still the default 4.8 incentives seven while in real eight for example it's a significantly less version, and we don't have control over what's being used, and that that means the installation, maybe less robust maybe less reproducible because it really depends on what system you're using it, whether it will work or not. To minimize this we typically only use it in two cases to start building a compiler tool chain ourselves so whenever we build GCC, we do that with the system compiler, but when once we have our own GCC compiler from that point on, we use our GCC. And of course whenever we install binary software or things that are just adorable that you went back, then you don't need a specific tool chain for that you can just use the system tool chain because you won't actually be using any compiler or any libraries that come from the system that could affect the installation. And then we also have common tool chains so these are, this is a concept that we came up with several years ago to try and focus the effort a bit in the easy build community. We have support for lots of different tool chains, think about 30 or something so different combinations of compiler, different compilers GCC Intel PGI. We also support for IBM compilers and things like this, and then different MPI libraries so they're going to have a fork of three or four different MPI libraries, combine this with different blasts and layback libraries like open blasts but at some point we also had atlas. We now have bliss, we have Intel and KL, and you can combine all of these of course into different combinations which give you different tool chains. And with that effort a bit we have two main tool chains which we call the common tool chains. One of them is the false tool chain which is everything open source. Currently that's GCC open MPI open blasts scale a pack and ffdw, and the open blasts one is under discussion. We named it false to have a neutral name. And one of the software packages that it currently consists of, because we knew at some point we may have to reconsider open API or open blasts and replace it by something else. So even if we move from open blasts to bliss we're going to keep the false name. And based on the version it will just be a different combination of components. The Intel tool chain, no surprises combination of Intel compilers Intel MPI and Intel and KL. The one API new generation, we're probably going to stick to Intel as a tool chain name. So from 2021 versions onwards it's going to be the one API version of the Intel tools. And roughly every six months we do an update of these common tools. So modules. So modules as a very overloaded term but we usually when we say modules we refer to environment modules. So the files that you get alongside an installation to activate them in your environment. It supports both tickle and Lua by default they use is Lua, you can tell it to use tickle if that's what you prefer. And of course you'll need a modules tool when using easy build by default it will use lmods which supports Lua syntax and module files, but it also supports the tickle based environment modules implementation. And then of course you will also have to tell it to use tickle syntax. Easy build will automatically generate modules for you for every installation it does and then by loading that module, you can activate it in your environment and start using it. So when we just say module, it usually refers to an environment module when we say Python module, it could be part of easy build that implements a specific functionality. And then here to bring it all together. So this collects all the terms in a couple of paragraphs so we use the easy build framework to leverage easy blocks to automate the installation of software, which may include additional extensions for installing the software user compiler toolchain, the easy config file specifies which toolchain to use which dependencies to use through easy config parameters. Whatever is specified easy build will make sure that the build and run run time dependencies are in place and will automatically generate modules environment modules for us. And the new feature easy stack files you can use these to define a software stack that should be installed by easy. So that's everything in terms of terminology. I hope that's a bit clear. There's any questions on this before we continue. I'll now look at the installation and configuration parts. There's several options for installing easy build. I'll try to quickly show all of them, but the recommended way for the hands on is here shown in the slides, which is method two, and on the tutorial website. So it's it's sort of a bootstrap procedure we used to have is still have a bootstrap script, but it's actually not recommended to use it anymore. So step one is installing easy build in a temporary directory using pip. So this is how you would do that, install prefix and make some changes to the environment. Then you can use this easy but installation to install easy build itself in the final place so where you would actually want it. In this case, I'll install it in home slash easy build. And you can use the install latest eb release command line option for the eb command so easy build knows what the latest stable releases, even if you use an older version it knows about the latest 433. So it basically pulls it from GitHub what the latest releases. And then when that's done you have an easy build module that you can load and start using easy build. I will show this hands on as well. I'll make this. Yeah, like this. Okay, so I'll use half terminal have website here. So there's three options. You can just install easy build with pip it's a standard Python package without too much special things. You need Python 2.7 or 3.5 or a more recent the three version Linux or Mac OS at least to install and play around with it but for any serious huge use use Linux, and then environment modules tools so I'm not as recommended. So we'll, we'll make sure we have all of that here I'll check on Python three this looks good. I'll check on the module command, which should tell me it's the Lua based implementation so this is elbow. Easy build is compatible with a range of different versions this is version seven which is relatively old but that's absolutely fine. Easy build will be happy with that. So that's in short, yeah Linux no surprise there are Python you can check with Python minus V or Python three minus V like I did. You can use the modules tool you can check with the module version command which ones you have if it says Lua you have the element implementation. If it says something else you have a tickle implementation and then you will need to configure easy build to use it. So by default, it expects a lot. And how to configure it, we will briefly cover that in the next part configuration parts, but it's explained in detail in the documentation. So easy build is a standard Python package that's good, but if you've seen this XKCD comic, then you know it can be a bit of a mess as well. So that's why here in the tutorial it's totally documented that there's different options and what the pros and cons are of those options. So something that complicates things a little bit is that easy build is broken up into three components framework easy blocks easy conflicts. That's done like that in GitHub. It also reflects in how it's packaged in as a Python package, the easy build Python package actually just pulls in the other three. So if you pip install easy build you will see it pulls in framework is blocks easy conflicts. So that's a little bit special I guess there's not a lot of Python packages that do that. That may be something that we could change going forward, but we don't see too many problems with this. Python two or Python three use Python three whenever you have the option. By default easy build will look for the Python commands to run. In this case, the Python commands actually Python two, so we have to be a little bit careful if we install easy build with Python three will also have to tell it to use Python three. The Python command itself as a shell wrapper and just looks at a range or a selection of different Python commands, it will consider Python, the Python command the default Python command before it looks at Python three. That's actually sort of a bug that we should fix in the eb command, but you can control which version, which Python command uses by setting the eb Python environment variable, and that's exactly what we will do here as well. I will show this with doing all three and just cleaning up again after doing one of them. I'll do the PIP one first, then I'll do the development setup. I'll clean up both because the I would say the recommended setup is the second one so having installing easy build with easy build and that way getting your first module that you install with easy build for easy build itself. There's again different options. So if you do PIP install easy build, it depends a bit on what kind of rights your user has, it may do a system wide installation by default if you have the necessary rights, or if you want to use a system wide installation, you may have to do sudo PIP install. I wouldn't recommend doing that, but it's up to you. If you want to just do an installation in your home directory, you can do PIP install user. That's the one I'll show here. So just to be clear, there's no eb right now and I don't even have a dot, oh, I do have a dot local directory. Okay, I'll clean that up. Really care what's in here. If you do PIP install dash dash user easy build. People do an installation in your home directory and here I think the PIP command is not even there. Let me check on that first. Yeah, it is not there but PIP three is there. So that will make sure we do the installation for Python three. You'll see it pulls in the easy build package and then it pulls in framework, easy blocks, easy conflicts, especially the last one has quite a bit of files. So if you're doing this in a shared home file system, it may take a little bit for it to put all those files in place. Depending on how you do the installation, so sudo PIP install user or PIP install prefix, you'll also have to change some things in the environment. Only if you do a system wide installation, the eb command will be there straight away. If you do a user installation, you may have to update your part to include dot local home dot local slash bin. If you do it in an entirely custom prefix, you'll have to update both Python part and part to get both the eb command and the Python packages behind it. So even though the installation is now complete, okay, dot local bin is already in my pot. And if it's a home installation, Python should just find it. It doesn't in this case because we've installed easy build with PIP three. So we have to tell easy build to also use Python three as the command to run easy build and then it works. This is sort of a bug, we can actually fix that in the eb command itself. But that's maybe also a good example if you're running into this, what the actual problem may be. So the eb command itself was found here but not the Python packages that are needed to run easy build. So this is something to be aware of. This is well explained here. PIP versus PIP three, what's the difference, you can actually if you want to make very sure that you're using the right Python version, you can do Python minus M PIP. So running PIP as a Python module rather than as a command and then you have full control over which Python version is used. So that's another option. You may have to update POT or Python POT to make sure everything is found. And if you're using a Python command that's different from the standard Python command, you have to tell easy build currently which Python command to use. This could be Python three or it could be a more specific version like Python three. And to check on the installation, you can use the eb-dash version or to get some more information about what the eb command is considering in terms of Python commands, you can do eb verbose equals one. That's the version that will tell you, okay, the first one I'll consider is Python three, because that was what eb-python specifies in the environment variable. And this is Python three dot six and then this is the location so it gives you some details. If you unset eb-python and do this again, you will see it's considering Python which is Python two dot seven and then it, yeah, it fails to find the changes because it's, there's a mismatch between Python two and Python three. I'll clean this up again. So I'll just throw away dot local where the actual installation was done. And I'll show you another method of installing easy build. So I'll skip the method three, I'll get back to method two. So if you're interested in developing easy build or making changes to it. An interesting way of installing it is just by cloning the different git repositories. So I'll do this here in an easy build sub directory. I'll clone the three repositories from GitHub in this directory. This is the history here. So this, this takes a while. And again, the easy conflicts repository includes quite a bit of files. So this may take a little while to get all that in place. And then what you do after cloning the repositories is you update part. You just add the location to the framework repository into part because that's where the eb command is provided small shell script. And now to add all three framework easy blocks easy conflicts. The framework and the easy blocks one is for the Python modules that are provided there. The easy conflicts one is because easy build will look for easy conflict files based on Python parts. That's one of the default things it does. So it makes sense to also add that to your Python part variable. And again here because we want to use Python three. In this case, it's actually not needed because there's nothing specific in here for Python two or Python three. It would just work without doing that. As well. So after cloning, we can copy paste the environment changes. We do which eb it finds it in the cloned easy build framework. If we do eb version. And now we're getting the development version of easy build. So not the latest stable release, but the ongoing development version that will become the next release. So everything is now coming from here. And we can start, we can jump into here and make changes and everything that we change here will immediately be picked up by the installation we are using. So if you're planning to make changes and play around with things. That's a good way of getting getting started just clone the repository set up your environment and you're good to go. I'll clean this up again. To show method to lots of IO going on now to clean up all those files. Method two is the one I also recommend in the slides. So this is the three step process bootstrap process where we eventually want to install easy build with easy build and get an easy build module to load. Step one is using pip like we did in method one. We're going to do a pip install into a temporary directory. So we do that just prefix from to some temporary directory here and doing Python three minus and pip but pip three will do the exact same thing. And I'm using ignore installed just in case there's an easy build installation somewhere on the system I want to tell it. Yeah, just ignore that and do as you're told, and install the latest version. So I'll start with that. I'm going to create a directory I think people do that automatically. So it will again pull in the different Python packages, it has them cashed somewhere because I did this before it will do the installation in the temporary directory we can check what it has done. It has a bin sub directory where the eb command is and some other things that could be useful. And then in the lip directory, this is where the Python specific part is site packages, which has the framework and the easy blocks components in here. So both framework and easy blocks are in here. So config files are actually, let's see where they are. Easy config files are probably just in here. Yes. So these are data files. They're not actually Python modules. So pip throws them in a separate place. So that gives us a temporary installation to actually use this. Again, we need to update our environment. Update part update Python part and set eb Python to Python three. And then we have if you check with which this will be a temporary easy build installation. So of course, you can also do this straight install this prefix into your home directory and then you have a working easy builds. But what we want to have here is a module for easy builds. So just like any other software, we want to install it as a module. Now that we have a working temporary version of easy build, we can just use the eb commands to use eb install it is eb release which is an option to pull in the latest release and install a module for that and we tell it where to go. And then with this prefix, we tell it's everything you do, please do it on their home easy build which I cleaned up. So that should no longer be there. Indeed. And if I run this easy build will install the latest release of itself. You see a bit of a scary warning here feel free to ignore that. That's because of the switch from dummy to system. And because we're making sure that easy build three versions can also install easy build four versions. So that's something we need to be a little bit careful with. And this warning here is because by default on putty. In this case, it was set to ICC and CSX variables are defined to something probably to GCC and G plus plus. And whenever easy build doesn't installation with the system tool chain so using the system compilers. It takes control over these environment variables and sets up a minimal build environment so easy build is warning us. In this case, it was set to ICC and easy build tells us I'll set this to GCC instead so I'm in full control over what's being used. So installing easy build it doesn't matter, but it does this always when using system tool chain. So this again takes a little while because of all these easy conflict files that are included so there's over 10,000 files being added to the installation. See here you get a glimpse of the process that easy build takes when installing software it will make sure it has all the source files for doing that it sets up a build environment starts unpacking maybe applying patch files, repairing things in terms of loading the tool chain module loading dependencies. In this case there's nothing here of course, and then doing configure build, perhaps test install stuff so that's where all the work was done for the package. And then some additional steps like post processing that's fixing things like. Yeah, it could be empty in this case, fixing things like permissions is also done. It does a sanity check to make sure it actually, it actually is providing a working installation. If it's happening with happy with the sanity check it will create a module file, and then dean up after itself and complete the installation. What has happened now is in the easy build subdirectory. We have a bunch of directories. So, not everything in here is in the right place like we eventually wanted to be to install real software. So if you have enough for this use case, it has created a build directory modules directory software directory, a source directory, and here in the files repo this is an archive of easy company files that was used to do installations. The parts that we care about are the modules directory. So in here you have a modules all with an easy build subdirectory with a modules file for easy build itself. And the actual installation is on the software. Again here easy build the version, and then in there we have the actual easy build installation. So that complete step two, we can actually get rid of the temporary installation now. So I'll just remove this to make sure we're actually using our final easy build installation. So easy build is not available anymore because we haven't loaded the module yet. So to load the module we have to make sure the location of the module files is added to the module part. So we do a module use for that. Then with module avail. We can check that we have an easy build module and then just loading the module is sufficient to get a working easy build version. So we don't have to worry about part Python part, or even EB Python, because the easy build module will set all of that for us so easy build is smart enough to know what it was installed with. If it knows. I was installed with Python three, it will also set EB Python in the module file, and everything will nicely work. I would recommend installing easy but like this because then it's a software installation just like any other installation you can just load the module and start using it and also end users can load easy build and install software in their home directory on top of what they centrally provided and so on. You can verify the installation like I showed EB dash dash version, you can learn EB dash dash help, which gives you a very long output with all the possible configuration options that easy build supports. You can run TV show config and now we're getting ahead of ourselves a little bit to show the current easy build configuration, which will cover in the next part. The TV show system info will give you some information about the system that you're running on which OS which CPU. And also some basic software like GLC version which Python binary is used for running easy build and what exact version that is. So all of these should work once you have working easy build installation. We'll get to the configuration in the next part. The next thing is, if you want to update easy build depends a bit on how you install it originally you can do a standard Python install dash dash upgrade easy build and it will pull in the latest version and overwrite your current easy build installation. So what we recommend is that people use EB install latest EB release, whenever there's a new version of easy build and then they will get an additional module for the next version of easy build. In this case I can try using it, but it will just tell me that easy build 433 is already installed as a module so this will not do much. And it says okay I found a module for this already. So you're good you have your are. You do have the latest easy build version already available. There's an exercise here, all the exercise says is just get it installed big methods. And I, as mentioned in the slides. So doing the pip installation in temporary directory then installing easy build with easy build and loading the module. Any questions on this before I go to configuration. If not, I'll continue. So here for configuring easy build I'll walk through this and show a couple of things. The recommended way of configuring easy build for the rest of the hands on today and also for the rest of the tutorial up to some extent is mentioned here. So we'll configure easy build to just use home easy build for most stuff, except for build directories, because it's not smart to do that on a shared file system you may run into trouble if you use it on Luster or GPFS, where there's actually lots of things going on. So for this will use the temporary file system. And I recommend as I mentioned at the start to make sure you have a clean environment so module purge to do a module list. When you log in, of course, now we have easy build loading as well. Let me get rid of that. When you log in, you'll see there's a couple of modules loaded into compilers until mkl and then MPI. For some installations, this may actually cause trouble. Because I know for non by for example if it's easy mkl it will prefer using that one rather than the open blast that we may provide through easy builds, which is actually some sort of a bug, because it seems like we're not in full control over over the non by there so that's something to look into. But I recommend purging those modules so you end up with an empty module list and then you won't run into trouble. And the other one here is if you check the available modules. So we'll see the easy build module we have but also lots of other stuff that was installed. So this part seems to be installed with SPAC this part seems to be installed with maybe something else but there's a whole collection of software in here. For the most part this will not be a problem unless there's an accidental match with something that easy build plans to install and I think there's a B zip. Maybe it was on on Marty rather than put he. Okay, not here. But in some cases you may run into surprises so it's, it's actually never a good idea to combine modules that come from different installation tools, unless you're very careful. So I recommend doing a module on use of the entire module part here to start with the blank slate. And then we do on put he have a software stack already available. So in here there's a bunch of modules already available. So I'm going to flip up to my own recommendations and wipe the module part and only have these. And then what module avail you should see an output like this. So this is a small software stack already installed with easy build where the most important components are the fast tool chain so GCC open API open blasts fdw, and some Python stuff on top of the API bundle is not by side by pandas combined and a Python version that is controlled through easy build. That's mostly in place there so you don't have to wait for one or two hours until the basic things are installed so you can actually play around with stuff a little bit. So you can definitely do that by yourself you can probably even do that in your home directory here on on put he assuming you have enough storage quota and file quota. Which may be a problem I think if you're installing lots of stuff, but for the installations we are doing. In the next section that should be okay. So that's the recommended configuration to clarify this little bit. I'll walk through this as well. So out of the box easy build will work. Elmod is available as a modules tool and you don't really have to configure it for it to work. But you'll probably not be happy with the defaults, especially not in terms of where it will install stuff by default so by default it will do that local slash easy build in your home directory so this is sort of a standard. But since it's a hidden directory it's at the very least annoying to work with. And using your home directory for lots of software installations may not be the smartest thing to do either, depending on the system you're working on for the hands on today will stick to that. I haven't run into big issues with that. While testing things, but you may want to reconsider that of course. With all the available configuration settings you can query through EB dash dash help that will give you a list of 240 different settings with a small help message. So that that's probably a bit daunting at first. And not everything is going to be crystal clear just from the help output. But hopefully you can find what you need there and if not the documentation should be able to help you. So through a couple of important configuration settings first and then talk about the actual way of configuring easy build. So the prefix that's the one I was using to do the easy build installation by default it's not local easy build so you probably don't want to use this. But if you set the prefix option. You're controlling different locations all at once. The place where software is installed the place for build directories, the place where sources will be downloaded to the location of the easy conflicts archive so eb files repo. And also also where easy build will throw container recipes and content images if you're using that feature. So that's all covered by a single setting that you can define to have easy control over the locations, but each of these can also be set separately so you can set prefix and then set one or two of these other ones to have more fine grains control. The install path option determines where the software and the modules go. The software goes into the software subdirectory modules go into the module subdirectory with some directories under there and we will get to that. But you also have separate settings for to only control software and only control modules, so you can put these in different locations if you want to. I wouldn't recommend that I would keep software and modules next to each other. Unless you have a very good reason to divert from that. Build path is where build directories go so build directories are where easy build will unpack the sources and do the actual compilation before doing the final installation. You don't want to do this on a shared file system for the number of reasons shared fast systems are typically relatively slow in terms of metadata access that's that's one reason compilation can be fairly io intensive so lots of small files flying around. So using a shared file system may may slow down the build quite a bit. This space is also an issue certainly in your home directory. You probably only have a couple of gigabytes of disk space and you may need tens of gigabytes for some software installations. So it's recommended to change this and the recommended configuration here we're changing this to slash them slash user to use a local file system and have a unique parts for our build directory so we're not stepping on each other stones. You could also consider using deaf shmem so the RAM disk. Part for this and probably even a unique directory in there that can help a lot with speeding up the build process but in some cases you will run into problems as well. Space limitations is one issue or depending on how the shmem is mounted. So if it's mounted within no exact option so that doesn't allow any binaries to live there or to be executable from there. Then you will, you may run into trouble as well depending on what you're installing the source path as well where it is able to install sources into. So whenever you download something it will put it in the source path so it doesn't have to download it again. There are multiple locations so not the single directory but a colon separated list of parts. And then the first directory is where is able to store new downloads, the other directories it will only have read only checks in there. See if something is already available so if you have a shared cache of downloaded files you can add that as an additional part to the source path. Easy config archive is a small. It could become big over time but this is where easy build keeps a copy of the easy config files that were used for installations for easy reference. This can be a regular directory but you can also configure it to do to use a kit repository and do get commits and and also pushes into a GitHub repository, for example, and this is well covered in the documentation. The module syntax is only relevant if you're not using Elmod so by default easy build assumes you're using Elmod as a modules tool. Then using Lua as a module syntax. If you want to divert from that you can use the modules tool configuration option to control something else than Elmod and you can check which tools are supported by easy builds. You want to change the syntax for module files if you're a big fan of tickle for whatever reason, you can switch to the tickle module syntax and through that configuration setting. Unless you have strong feelings about this I would just recommend keeping into the default, especially since putty is the modules tool on since Elmod is the modules tool on putty. And things beyond where stuff goes and what modules tool is used the robot search part is where easy build will look for easy config files. By default, it will be aware of where the easy config files that are included with easy build itself live so it does that through the Python parts environment variable, but you can give it additional locations to search in. There are actually two configuration settings here robot and robot parts. And there's a, they pretty much do the same thing in terms of locating easy config files. But if you use the robot option you will also enable dependency resolution, which you want to do by just using robot parts. That's a small distinction. That's important and we will get back to that so only when you use the dash dash robot command line option. You're telling easy build if there's any missing dependencies please install those as well. That will be the software I'm telling you to install the module naming scheme will get back to that in the advanced part so by default easy build uses a default naming scheme, which closely matches the names of the easy config files. So it will use this name slash version dash tool chain, and perhaps with an additional label attached that is controlled by the version suffix easy config parameter. And this tool chain part may be empty. When it when it's using the system tool chain so for installing easy build itself, you just get easy build slash version because the system tool chain makes this part empty. I will get back to this extensively when covering model naming schemes and all the flexibility easy build gives you there. So that's the most important configuration options you should be aware of. To actually configure easy build now you can do this in three different ways, you can create a configuration file, or multiple ones, a user level or system level configuration file. You can define easy build underscore environment variables, or you can use command line options to be the command. There's a hierarchy between these so command line options always when environment variables when over configuration files and configuration files only tweak the default configuration so depending on what you're doing may make sense to configure easy build in different places. That's what this explains. In detail but I don't think that's very surprising. The configuration files are used the standard any format. So, pretty much key equals value type of stuff. And to get a good start, you can use the EB config help command if you actually have easy build loaded of course that clean this up a bit too much. Okay. Currently I did. You just load the easy build module you won't have that problem. Is there a bash completion module for the for the options. Yes, there is there's something that we include with the easy build installation I think it's in the bin folder. I don't know if we have good documentation on that but it should work. You need to source the script that comes with easy build and then you have, you should have a completion. And the path is under the local slash easy build slash something. So it's next to the eb command I think the script that you need to source. It's typically in this in the slash bin sub folder. Okay. If easy build is in a module you can also use eb root easy build to get quick access. Yeah. So for every module that easy build installs it sets an eb root name of software environment variable to get quick access to the root directory of the installation. So configuration files if you use eb dash dash config help that will produce a long list with all the possible configuration options. Let me find one that makes sense. prefix. So here, everything is commented out so this will all of this will be inactive but it gives you the right syntax to say okay prefix should be slash apps or whatever. So this is a starting point for creating a configuration file. You don't have to figure it out manually. An important distinction here, easy build configuration files and easy conflict files are two entirely different things. And we do, we see some confusion around that. Not surprising I guess because the names are so close. But an easy build configuration file is the way you configure easy build in a general sense, where the software go which modules tools should it use things like this and usually the file names are dot CFG for conflict. For example, an easy conflict file is an dot eb file usually that specifies the details for a particular software installation so not where that software installation goes, but what easy build should install which version which dependencies which tool chain, and so on so that's an important distinction to make. The next level of configuration is the easy build environment variables so every easy build underscore environment variable will be picked up by easy build and it will try to match it to a configuration setting. If you make a typo in the name of the environment variable easy build will complain and say, I don't know what this is probably made a typo or you're using a wrong version of easy build that doesn't know about this yet. So it won't silently ignore wrong easy build underscore environment. The mapping from the name of the configuration option as it appears in dash dash help to the name of the environment variable as done by changing all the dashes to an underscore and making everything uppercase, and then prefixing it with easy build underscore. For example dash syntax that corresponding environment variable is easy build underscore module underscore syntax everything uppercase. So to configure easy build to use tickle syntax you can just set this environment variable. And again, whenever you have both an environment variable defined and something and the same setting specified in a configuration file, the environment variable wins. So if you're on the command line, you can give all these options as well with the regular dash dash options or sometimes there are short hands with a single dash. And this always wins. So when, even if install part is specified through an environment variable or configuration file is able to use this. So no surprises here. If you want to be in full control you can use the command line options. Lots of configuration options three different levels for if you include the default settings. That could be quite confusing quite fast, especially if you're using easy build on different systems that is configured in different ways. That may become a bit puzzling, but we have an eb show config option that produces the current easy build active, the active easy build configuration. It will always include some settings which we consider to be very important mostly about where stuff goes, and where easy build will look for easy config files. So this is what you get with a default configuration, everything in dot local slash easy builds. And it also tells you where it is configured. So D is for default. C is for command line argument is for environment variable f is for configuration file. So, there's a very concrete example here. We'll start off with creating a small configuration file that easy build picks up on. So that config easy build config.crg it picks up on. And there's a, there's an option. I didn't mention it here. That tells you which locations easy build considers when checking for configuration file so there's user user level and system level configuration files. Here we'll use it will tell it to use dash slash apps for the prefix, which controls various spots will also set the easy build build part environment variable to control the location of the build directories. And then when we run show config, and we give you specify the location where software should be installed on the command line. This is the kind of output we get. So build part it says E for environment variable so this is controlled via this environment variable. F is through the configuration file so the prefix slash apps and then some others are derived from that. C is for command line so the install path was specified on the command line option here. And B is for default so we didn't touch the location where easy build should look for easy config files. So in a very summarized view you have a complete overview of the active easy build configuration and where the configuration was done. And when a when a variable or when a setting is being read from a config, from a config file when you have a whole bunch of config files, I presume is the the last occurrence that takes precedence correct. Yes, so there's a certain order. Let me jump back to configuration files that here, maybe show default config files. I'll show you which configuration files are active. And the user level ones will win over the system level once. And here if you have a list of them. The last one wins probably. I would have to check that but yeah it's consistent or it's first or last it's consistent and user level wins over system level. All right, well, if in the user level I have, sorry, if the user level wins and then if I have in the user level again a hierarchy of config files, I can read them kind of recursive list I suppose I can do include. So, the, the most recent include a reset whatever, whatever the previous setting has. It gets over the last thing. Yeah, I would have to have to check the exact order but it will list them here in an order and either the first one or the last one determines the setting. I think it's the last one. I think you read all configuration files and just process them in the order that they are found in. And then the last one wins. It is left to right I recently did it. Is it okay. It is the last one that wins. Well I did it with minus minus coffee files and gave multiple coffee files that way that's clean environment variable but it's the same thing of course. So show config yeah and there's a small exercise here. It basically tells you to configure easy builds, mostly in the way that is recommended in the, in the slides, I actually gave away the answer here pretty much. So there's not a lot of exercising to do here but yeah, I think it's clear how much flexibility you can. So, I'll just make sure before I continue. What I did in the example is actually not fully what I want. I don't want to play in slash apps, since I don't have right access to that so I'll throw away the configuration file, and I'll set the prefix through the environment variable. So like I showed in the slides. And then if you do show config without any options. It should tell us that most stuff will be done in slash easy built in the home directory, except for the built part which goes into slash jump. So that's actually the kind of view you want to get for the next part. Go back here and go to basic usage. We're running out of time a bit. I don't know if there's any more questions on configuration. And the package does not seem to show up. But when using the method, the first method that you showed, which is the method number three. Have I missed a step? Yeah, it's possible if you if you don't specify the prefix so to be clear. Other things are always included here like built parts like install parts. Other things are only included when you customize them. So I think it doesn't show prefix by default because it just says okay that's the default and I'm not even going to show you that if you're happy with the default. If you want to check every possible setting, you can do show full config. And is there a way to say okay I see show me where this where this setting has been set, whether it was a command line where where where is the active value taken from which file which line number. Okay, it just tells you on which level. So default or environment or file or command line option it doesn't tell you in which file. That's something that I guess we could add a to show conflict I haven't considered that. Especially when when there are multiple files I mean there's only a single environment, there's only a single command line, but it's a very good point. Yeah, yeah, files then which all of this. All right, thanks. So we don't have that now but that could be useful addition of these two maybe here on the side mentioned the file name and then the line number, or just a file name. Yeah, that could be useful. Good point. Okay, I'll walk through the example here for basic usage. There's an exercise at the end here actually multiple exercises that basically cover most of the things that I'll show in basic usage. For example, the answers here are in the green box if you click this you'll see the answer to make it fun for yourself don't click the box until you've tried it yourself. And the exercises are different from the example I give so you have to at least think a little bit. It's not very difficult I think. I'll walk you through some examples important here when doing this one is that these this software stack is picked up so if you check module avail that easy build is aware of these modules and that will prevent it from reinstalling everything here so easily doesn't care where modules live if it can see them through module avail it's happy. So you can combine things from different places very easily centrally installed software home directory temporary directory you can combine all of these and as long as easy build can load the module. It's happy. So the way why is it that the being on on login one module avail gives me a meal three pages worth of worth of stuff and when and you are showing just one page. That's because you didn't clean the module part. I didn't clean it all right. If you do a fresh login you'll see this. Yeah. Yeah, and that's because the module part here includes all these directories. If you just do module on use module part and don't make any typos. Yeah, it will unload the modules as well but then module avail will be empty. And if you do module use our part that's when you get only this few. Yeah, got it. Thanks. So make sure at the very least that we have these visible that's probably enough you don't probably need to clean everything else but these should be first in module part and the module use command will do that for you. So the basic usage of easy build is you use the EB command and you give it. You tell it what to install there's actually different ways of telling easy but what to install the most common way is giving it either names of easy conflict files or parts to easy conflict files. There's other options like that has a software name option and a software version option, but that's a bit lots of typing work and that's typically not how people use EB. So you can pass to easy conflict files relative or absolute so relative to the current directory or absolute. The name of an easy conflict file which could live somewhere else and easy build will go and look for it in the robot search box. Or you can also give it the path to a directory containing easy conflict files and then easy build will use all the easy conflict files in the directory. There's an example here. So this is the kind of structure we have we have two easy conflict files here and then a directory with two easy conflict files and another file in it. And then the sub directory in there that has yet another easy conflict file. If you run this command we tell easy build use this local easy conflict file, use this one which is not in here anywhere so easy but will go hunt for that in the robot search box. So this is an absolute part so it will take this file and then this is directory and it will use all the EB files in this directory, and it will ignore everything else so it won't even try to read the dot txt. That's the very basics. So let's try that here with one that's included with easy builds with just to be busy to the full name is it will go and find that. It says okay I found this file here so it takes the first hit. And then it goes ahead and does the installation of a specified by that easy conflict file. So this is a visa to one dot zero six with the system tool chain. File names we've briefly been over all the easy conflict files included with easy build have a specific naming scheme. That's because whenever easy build needs to find an easy conflict file based on, for example only name and version or name version tool chain, it will only look for files with the correct name so it will not try to read and interpret every possible file it runs into that would be way too slow. If you're just listing files on the command line itself the name doesn't matter, even the extension doesn't matter so if you do EB foo and foo isn't an easy conflict file that's available in the current directory, it will just work to search for easy conflict files that are available there's a search option. So look here I'm using open foam as an example. So the search option you can ask easy build which easy conflict files it knows about it will consider all the parts in the robot search part so by default the ones included with easy build but you can extend that. If you have your own stash of easy conflict files. Search gives you a full pot answer the minus capital S shorthand gives you a shorter option that says these files are in this directory, depending on what you're doing it may be useful to use either of those. The search option has support for regular expressions. So you can find grain things a bit. Let's say I just want TensorFlow two for one with a 2020 B tool chain so you can use dot star, and make sure to get a TensorFlow, an easy conflict file that starts with TensorFlow. The reason I'm doing that is because if you don't do that. You may also find other stuff that has TensorFlow two for one in the, in the name of the file, like this Horovot installation. Add a tool chain version you may also find other files like patch files so search also looks for any, any files it hits in the robot search box so that may include patch files. When you do a search you see a line like this found valid index, so easy build has support for creating file search indexes. So, which I think I are hidden somewhere in here. Yeah, TV pot index. So this is a file that includes all the locations of the files included in this directory so I don't know how many are here but well over 10,000 probably 16,000. So this search index just speeds up the search quite a lot it reads this file and it trusts it. That's always these files are in there. We include a search index with easy build itself for all the easy country files included, and you can create your own search indexes for the parts you have with additional easy company files division. Create index option and some related ones that are covered well in the documentation. So when easy build is hunting for easy company files the search index stuff may help you quite a lot. Inspecting easy company files so before you install something I already did it for busy to, but you may want to check what's in the easy company file so you have a show EC that pretty, pretty much does a cat of the file but you don't have to tell where the file is located so the robot can go out first for you and just show the contents. And this is what that looks like I won't go through all of this here. We will cover that in the next section of the tutorial what all this means and how you can tweak it. If there's something you don't like, but that may help you in getting some information about what's going on. Another important part is checking whether dependencies are in place for the software you want to install. And there's two different ways of doing that. There's a regular dry run. There's another dry remote which will cover in a bit. What the regular dry run eb dash dash dry run or eb dash capital D gives you an overview of all the dependencies for the particular easy company file, and also what's in place already so everything marked with X is already installed. So everything that doesn't have an X is still missing. In this case only sound tools itself is missing all the dependencies are already in place. To use the dash dash dry run, you will get the same view but with the full parts to the easy company files with a bit less easy to digest. Usually you're only interested in missing dependencies so we have a separate option for that. In this case it says there's 22 things required in order to have everything in place to install samples. There's one missing samples itself. So that matches with the output we saw before. If you do it for h5 pi and you're using the software stack that's available in the project space, you will see that there's two things still missing. So minus capital M or dash dash missing is the same thing. You have a view of what's still missing and what easy builds will install to get something working. That's good, but maybe you want to have a detailed view or you want to check on something before easy build actually does the installation. For this we have extended dry run so the name is not perfect let's say but we do have an eb dash X for short so that's probably more useful. This will give you an overview of the full installation procedure like easy build would perform it, but without it actually doing something so it will not even create any directories or try to impact a source. Tarball anything like this it just will report on all the commands and all the steps it would do. So if you do this for this boost example. And I'll have to pipe this through less because this will be quite a lot of output, it will run through all of the steps one by one, but it doesn't in the matter of seconds so even if the installation will take hours you get almost instant feedback of what's going on. There's a lot of text here, but you can see it's it's going over different steps. So the fact steps that okay. This is the download URL I would use for downloading boost and this is the file I will be looking at it says it will be downloaded that means it's not there yet. It's, it found a patch file, and then it's walking through all the steps, verifying checksums doing the extracting the tarball, preparing the environment which means loading this tool chain module loading the dependencies. In this case, it knows what that environment would look like because all the dependencies are already in place so it can tell us the module list that it would end up with. It tells us all the environment variables it will set to control compiler commands and compiler options, the configure command that will be run the build command that will be run which is humongous because it has all the parts expanded. And it actually does that multiple times for I think once for this is built, this is installation and I think it is both shared and static libraries or something like this so there's a reason this is being run multiple times. Scroll through a bit it will tell us in the sanity check what it's going to check for. So it will expect to have this file and this file in place and this not empty directory in place. If that's missing after the installation easily will say something's wrong here I'm not finding what I expect to find. So something went wrong. And then it also gives you an overview of the module file that will be created and the parts that would be changed when the module file is loaded. There's a lot of information here it's very very dense. But if you want to look at some particular aspect of the installation you can easily do that with EB dash X. And you will get instant feedback on that. There's a note here that's worth mentioning this is not perfect. So it depends on how well the easy block was written. So if rock expects to find a file or needs the output of a command before it can take the next decision on how to proceed with the installation. Then some errors will pop up will pop up in the output of EB dash X, maybe that actually keep going it will not crash but it will say an error occurred in this part and I'll try to continue best I can. I may not get the full information you're looking for. That's important to know and also some parts of the easy block or dynamic so there could be an if statement in there that says if this file is there then I will run this command or if it's not there I will run another command and EB dash X is not able to show you that. Because it doesn't create any files it just tips over everything, and it assumes commands run always exit code zero and there's nothing actually going on on the fastest. So take it with a grain of salt. And depending on the easy block this may be a little less useful. But yeah, it can still give you a pretty good view of what is most likely to be going on during the installation. So that gives you a whole bunch of ways to check on stuff before anything actually happens. To do an actual installation, like I showed you the zip. You do EB you give it the name of an easy config file is able to find it and starts downloading sources and packing stuff and running commands. So that's very easy. If everything is in place already like it was for Sam tools. The stuff is still missing. Like for this BCF tools example, there's three things still missing two dependencies and BCF tools itself. Then if you try to do the installation so without minus M. Easy but will not be happy. It will do the download step and it will start preparing things, but then it will notice okay there's modules missing for dependencies. So there's no HTS lip yet there's no DSL yet. The default behavior and easy both is to not automatically install dependencies, even if they're missing, you have to ask it to using the dash dash robot option. The error message I think it's very clear it literally says missing modules for dependencies maybe you want to use dash dash robots to make sure these are in place. If you use dash dash robot it will first do the dependencies. HTS lip and this is in this case and then GSL and then it will proceed to the actual software you care about. So if you use dashes robot, it will do the dependencies first and then it will complete all three installations in the row. Now, it's a bit annoying here when installing GSL. Let me interrupt it. When installing GSL that you're not seeing it's building but what is actually going on why is this taking so long you're a bit blind. So we have a trace output. If you use dash dash trace which you can enable with in setting the easy build trace environment option environment variable as well. You will get a bit more information. I was scrolling a bit faster, but when running the configure step at least now it tells you I run this config dot guess small script, then I run this configure command and I give it these options. And now I'm running this build command, and it's using all the course so that's easily built by default uses all the course it has access to, and you can control that if you're not happy with that. But at least you're getting some more feedback and you can actually check or take a look at the output that is being generated by this command by tailing this in a separate session. So, if something is taking too long, or you want to have a better view on what is actually going on while it's running, you can enable trace mount. In this case I did it for BCF tools but it's actually more interesting for GSL because this takes a while, and you can also see, once it completes something it will tell you that the command completed and how much time it took. And eventually when it gets to the sanity check. It will also tell you that what it was checking for and whether it was okay or not so if the sanity check fails, and you have trace mode enabled you can quickly see what was missing and why it wasn't happy. So here it's doing that for GSL. And then here we're close to the end so I don't think I have to explain this to this audience but if you want to use installed software, you of course have to load the module. At this point you may not be sure where easy build is installing software so you can check show config again and install part. This is where stuff goes. This is where you have the modules and the software directory to get access to the software the modules directory is what you care about. There's an all directory in here and then separate ones for different categories of software. You have sim links into the all directory so the all is usually what you go with, unless you want to break things up into different categories. And that's what's being explained here. So to load these modules we need to do a module use on this. And then if we check module avail will now have two sections our own installations in the home directory, and the central installations in the project directory. The modules rely on what's available here so the GCC installation is in here so loading this module will also load this module which then will load other models. So to actually get access to the software. If you do model list. I did a purge so I don't have anything loaded. Of course BCF tools is probably not installed system wide. But if you load the BCF tools module which we just installed. We have a module list. We have BCF tools loaded both dependencies loaded. Actually other dependencies that were already installed also loaded together with the tool chain, which includes pinutils and GCC core and needs zealot for the new tools as I guess. And then loading this module is enough to get access to the BCF tools command and you can get started with using the software. Clean up your environment before you do the exercises so just run module purge. Make sure you don't have anything loaded to do the exercises. If you have modules loaded. And you run easy build it will at least give you a warning that says hey you have a bunch of easy built installed modules loaded and then you're you're running an installation as well that may not work out too well and it will give you a big fat warning. And you can configure easy build to be more strict and even refuse to continue if it has modules loaded. And that's actually recommended to do so because pre loading modules and then letting easy build loads, either the same modules or other modules next to that may affect your installation so it's not recommended to do that. So that's why the module purge here is recommended before doing the exercises. And like I'm showing here you can stack software very easily with easy builds. We've actually been doing that. We're doing our own installations in our home directory on top of what's installed centrally, and that works perfectly well. Easy build just needs access to the module files and then it's up. So I won't run to the exercises here maybe that's that's good to do, at least by the next session so that's in two weeks you have quite a bit of time for that. You can you can easily run through these exercises and I guess 20 minutes or half an hour. And they just cover all the things we've discussed here searching easy copies checking for dependencies doing installation, enabling robots and getting access to the software that was installed. So, everything that was covered in this part. I think that's it. That's what I had in mind. Very close to the end of the session but maybe we do have time for some questions. If there are any might be just the detail about the again about the bash. Bash completion which doesn't seem to be supported by the logins of the default. Yeah, you need a system package installed indeed. So if you do this for that. Yeah, there's a system package missing to support bash completion. I can look it up what it is and I don't know by heart but you need something in the system installed to find out please put it into the chat thank you. I can look that up we probably have. If you look these look up these errors and the easy build configuration and the easy build GitHub repositories will probably find an issue that explains what's missing. Any questions. So for the next session, which is planned for me to go back. Yeah, I would like to have one question. So we do not have I and my friend magic from separate we do not have the putty account and whether it will be important to join the next session to have the putty account on putty or. Yeah, I will I will get I will get you the accounts. The problem is that you just joined in today that's why I didn't. So for today it was it was quite fine because I was I have been able to do it on my own system. Even even if I have checked that this bash completion didn't work out haven't worked out or system probably due to the same problem as at putty but next session will be how to manage the things that do not work and it would be great to have an access to putty. Yeah. Okay, thanks. For the for the next session, I think we'll mostly need the same software stack as we have here now. So if you have fast installed and Python installed, there won't be a lot more that will be needed to do the next session as well. So we keep that we keep that surely minimal. But yeah, it would be easier if you have access to putty of course is there. Okay, that's that's also where I'm testing and where I make sure that what you see on putty agrees with what is in the document. Okay, so I think that we will we will we will sort it out with Nikolino. Okay. Yeah. Okay, both ways. So sorry, I have to, I have to go to another meeting right now but it was great session. Thank you very much for for it because it is very good. It was very good. So have a good afternoon. Bye. Yeah, bye bye. Next next session will go up a little bit deeper. So now we've just used what is able provides out of the box. Next session will try to write an easy profile from scratch for something new. And we'll actually, we have an example that that gives you an easy company file that doesn't work. You gradually have to fix some some problems and so that's a very good exercise, because it. Yeah, it tells you or it teaches you how to do debugging how to look through log files how to, how to find what's wrong. And I think that's a very good exercise, and we'll go a little bit beyond that I'll cover the basics of implementing easy blocks. Yeah, you can probably fill a full day session on that but I'll, I'll try to cover the basics and also do. I'll show you how to in how to submit installations as slurm jobs which I haven't tested on putty but it should work you don't need anything special. Can we ask you to stay around from that time on the chat during these weeks. In case we need something we can drop you the message that. Yeah, yeah. That's also maybe good to mention to do. I'm happy to hang out in the chat for a while, but longer term. Yeah, or is this here longer term, I strongly recommend using the slack channel. Yes, because right now in the rocket chat it will be mostly me and I guess good as well who is quite experienced with easy build. But in the slack channel there's 400 people and there's always somebody active and also all the other easy build maintainers are in there. That's good. Yeah, I'm happy to hang out there for a while. You know you just for these two weeks in case we play and we want to ask you something for the next sessions. Yep, no problem. I don't know, are there other questions we want to ask or before we suggest the suggestion if you would be different craze sites. We could consider just having a special channel for Craig in the slack. And it's interesting. Yes. It's very craze specific like I mean I was installing GCSE and so on but in CSC they just CSCS they just use the group compiler which comes with the craze software development stack. So there are a couple of differences, which for some questions may be more useful to have a separate channel, but of course if you are not the only site who would use that channel. You can set that up yourself. Good. It's very easy to set up a slack channel. I see you probably just click the plus. I can invite the CSCS people in there and feel free to announce it in a general channel. There's now a craze specific channel. And then maybe maybe some other maintainers will join there as well, or people who are just interested to see how things are different. I would make it an open channel just so people can jump in. Okay, if there's no more questions. Thanks a lot, Ken. Exactly on time I guess. And if there's something specific you would like me to cover in the future session do let me know. Because especially for the advanced part. Yeah, here we do. I think we do have some time I need to prepare all three of these pretty much. So we don't have a tutorial part tutorial stuff for this yet. Well, actually we do for the hard model naming schemes. The other tools are other two I have to prepare, but there's probably room for some additional stuff in here so if people want me to demo and cover something specific feel free to let me know. Okay, definitely. We'll discuss when we can lose them and see if we have a special need. Okay, thanks a lot, Ken. It was great. It was really nice. Okay, so see you in two weeks from now. Yep. See you in two weeks and see you in the chat in the chat. Yeah, thanks a lot. Bye bye.