 So let's continue the part about installing EasyBuild, which is the next part in the tutorial here. So here I will be doing some hands-on in the prepared environment we have in AWS. So be ready to jump into the terminal and execute some commands there. Before I do this, I will explain some of the requirements we have for EasyBuild and the different installation methods that we support. And then at the end, we'll actually have a short exercise and make sure you can get EasyBuild installed in the tutorial environment. So don't run ahead of yourself and try to install it already. Please wait until the end of this part, until we get to the exercise. So what does EasyBuild require? It requires a Linux operating system. So that's the main target platform for EasyBuild since we're heavily focusing on HPC systems and installing software for HPC. This is our main target platform. Up to some sense, EasyBuild is also compatible with macOS. So the framework itself will certainly work on macOS, but you will not find a lot of easy config files for software that installs on macOS out of the box. So you will run into some trouble there if you want to make software installations with EasyBuild on macOS. It's not our main target platform. We need a particular version of Python. So it says here, Python 2.7, Python 2.6 will still work if you're in an old CentOS version, for example. You can still get away with that for now, but the support for 2.6 has been deprecated. Any Python 3 version over 3.5 should work as well. So if we check in the AWS environment here, we will see we have both Python 2 and Python 3 installed. The Python 3 is a 3.6, so that should be good enough for EasyBuild. And this is the version that we will be using in this tutorial as well. Next to Python, we also need an environment modules tool. This is a strict requirement for EasyBuild. You currently cannot use EasyBuild without having an environment modules tool. So the default one that is being used by EasyBuild is ELMOD, which is a Lua-based implementation of the environment modules tool, which is a replacement for the more traditional, original, tickle-based implementation. So ELMOD has seen wide adoption in the HPC community recently, and it has become the default modules tool in EasyBuild. To check if you have a modules tool installed, you can run module-x version. Let's copy this command here and paste it in the AWS environment in the container image. And we should see we have the Lua-based implementation, which is ELMOD, and the particular version, which is recent enough for EasyBuild. So any version over ELMOD 7 should be good enough for EasyBuild. It doesn't need a very, very recent implementation. If you see something else as output here, something that says version equals 3.2.10, or modules release 4.x something, you have a tickle-based implementation of the modules tool, which will also work with EasyBuild, but then you will need to configure EasyBuild to make sure it picks up on this modules tool. So during this tutorial, we will focus on ELMOD, which is the default modules tool that EasyBuild expects. Then looking at EasyBuild itself, so EasyBuild is in some sense a standard Python package, so you can easily install it with the standard Python tools, like PIP, Virtual Enth, Setup Tools, all these different standard Python tools that you have. So XKCD did a very good cartoon on this. So if you're playing around with lots of Python software, it will look a little bit like this. And of course, for this tutorial, we will try to avoid that. One thing to keep in mind when installing EasyBuild is whether you want to run it on your own, when installing EasyBuild, is whether you want to run it on Python 2 or Python 3. So in CentOS 7, for example, it's very easy to have both Python 2 and Python 3 available, like we also have here in the container image. So that's something we need to pay a little bit of attention to. Since Python 2 is no longer supported officially, we will use Python 3 in this case. So that's something to keep in mind. We will make sure that we use Python 3 when installing EasyBuild. There's two main ways for installing EasyBuild, so or at least two recommended ways. You can use the standard PIP tool to install Python packages, where you go PIP install. Maybe you give it a location where the package should be installed and the name of the package. That's a pretty, I hope, well-known installation procedure. And this is explained here in detail in this part of the tutorial. It basically boils down to PIP install EasyBuild, but as you will see, if you try this in the container image, you will see that PIP is not available yet. That's a little bit annoying. If we would install EasyBuild like this, we would also end up with... And I'm not sure if PIP version will... Yeah, yeah, it's not there. It's not going to tell me much. But if it will be there, we will be installing EasyBuild for Python 2 in this case, which is not what we want. We want to run EasyBuild on top of Python 3. And make sure that things will work out well. This gives you a bunch of options here. So if you do it like this, you may be installing in a system location where you may not have the rights to. In that case, you may want to use sudo PIP install. If you don't have administrator rights, you can try PIP install dash dash user. Or you can try PIP install dash prefix to install in a specific directory. Now, since we don't have PIP here, we're a little bit stuck. But as you may know, there's also a Python 3 variety of the PIP command, which is PIP 3. And this will actually install software using Python 3 rather than Python 2. So this is what we will be using here in this case. After installing EasyBuild with PIP, you may have to... Or you will have to update your environment. So whether you... Or at least when you use the dash dash user or dash dash prefix option, you'll have to update your environment. You have to make sure that the EB command, which is part of the EasyBuild installation, is available through your $part environment variable. So you will have to update the part variable with the specific prefix where EasyBuild was installed. And either you... This is a directory you specify to dash dash prefix. Or if you were using dash dash user, you would have to use $home.local slash bin here in this command. So keep that in mind. And then next to the part variable, you may also have to update the Python part variable. This is actually only needed when you did the PIP install by just prefix. If you picked the installation where EasyBuild should be installed yourself, then you also have to update Python part. If you used dash dash user, PIP install by just user, you don't actually have to update Python part. You can, but you don't have to. Because Python will look for Python packages in home.local automatically. So you don't have to tell it to look there by updating Python part. So keep in mind, when you do PIP install dash just user or dash prefix, you also have to update your part and Python part variables to make sure that EasyBuild is actually found by Python. And then depending on whether you are using the standard Python installation, so here in the AWS environment, our standard Python is still Python 2 while we actually want to use Python 3. So by default, EasyBuild will use the standard Python command, the EB command that is part of the EasyBuild installation will use the standard Python command and just assume that will work. If that doesn't work, as will be the case here, we will have to tell EasyBuild which Python commands to use. For this, we can set the EB Python environment variable. So since we will be installing EasyBuild with Python 3, this is exactly what we'll have to do. We'll have to tell EasyBuild to use the Python 3 command to run EasyBuild. And if you want to get more information about the Python commands that EasyBuild is considering, we can set the EB verbose environment variable whenever we run the EB command. And this will tell us which Python command it's considering, which version it found for this Python command and whether that meets the version requirements set by EasyBuild or not. So that's the first way with pip install, which as you can tell is already a little bit messy. So it's not just pip install, but we may have to or we will have to update our environment a little bit as well for this to work. Because installing Python packages is so messy, we actually came up with our own installation mechanism in EasyBuild, which we call the bootstrap mechanism. This is the second method that you can use. This has some advantages. So we provide a script for you, a bootstrap script that you can just run to install EasyBuild. And behind the scenes, this will actually first install EasyBuild to a temporary location and then use this temporary installation of EasyBuild to install EasyBuild itself. So it will install EasyBuild with EasyBuild basically. Because of that, because EasyBuild is installed with EasyBuild itself, we will get a module file for EasyBuild. So we can just run module load EasyBuild afterwards, which is a nice bonus. So let me do that here first. So the first step is downloading the bootstrap script. Let me show you how that works in the AWS environment. So we download and we're already running two problems. Nice. We download the bootstrap script like this, okay. Then we have to run it with a particular Python command and a prefix where EasyBuild should be installed. So in this case, we want to make sure it's using Python 3. So we use the Python 3 command and we give it the prefix where EasyBuild should be installed. Based on the previous command that failed, this may actually not go too well. Let's see how this goes. So in the background, the bootstrap script is now first installing EasyBuild in a temporary location, as it's telling you here. So this will take a little while and then if this goes well, it will do a stage two where it will install EasyBuild with EasyBuild which should give us a module that we can load. This, and as I was expecting, this is failing, which is a bit surprising to me. Okay, let me just change it to a different location. So you may be seeing the same problem. I'm not sure why the home directory is not writable but we can just use slash tmp for example rather than dollar home and that should make it hopefully work. I was going to say the syntax already see popping up here that's safe to ignore. There's a small part in EasyBuild that's Python 2 specific but this will only be used when EasyBuild is run with Python 2. When it's run with Python 3, it will use a different part. So this part, the syntax already may see popping up during the bootstrap is safe to ignore this one. So now we did the stage one installation, a temporary installation of EasyBuild and it uses this to install EasyBuild in the location we specified. This gives us a whole bunch of output. It says the bootstrap was completed so this looked good. It was installed into slash tmp slash EasyBuild which also makes sense. And here it's telling us to first make sure that module part includes this location which basically corresponds to this module use command. So we have to tell the modules to where we have installed modules before we can load them. So let's do that. Let's do module use, temp, EasyBuild. This is the, let me do an LS first to show you what's there. This is where we installed or this is the location we gave to the bootstrap script to install EasyBuild. We can see modules in there and there's a subdirectory modules all which seems to have a module for EasyBuild available. So we can use modules, module use, temp, EasyBuild, modules all. And if we now check with module avail which is how we can check for available modules, we see the EasyBuild installation available which means we can load it, we can do module load EasyBuild as is mentioned here in the tutorial. And then we can see whether it actually works. So a very simple way is checking the version. If we run eb-dash-version, we should get EasyBuild 4.2.1. That worked well. So this gives us an EasyBuild installation we can play around with. We have this installed as a module which is useful in some ways but also a bit annoying in other ways. So if we use an EasyBuild installation installed as a module, we will run into some confusion later because every now and then we will run a module purge which unloads so module purge which unloads all the loaded modules which means we'll also lose our EasyBuild installation. So that's a bit annoying. That's why we recommend to use this the first method to use pip install or actually pip 3 install to do the installation. And this verifying the installation is a number of eb commands that you can run. You can run eb-dash-version, eb-dash-help. To get the help output of the eb command, we can check the current configuration of EasyBuild with the show config command or we can collect some basic system information with eb-show system info. So let me do that before we go to the exercises. So let's load back the EasyBuild module and let's for example run show config. This shows us the current configuration of EasyBuild which is everything or almost everything in home.local and the ds tell us that this is the default configuration. We will get back to configuring EasyBuild in the next part of the tutorial. So it's okay to leave this as default for now. We can also run, let me copy paste it, eb-show system info which will tell us some basic information about our system, basic information about our CPU that it's a Skylake apparently that we're getting on AWS and it will also tell us which Python version is being used. Now I mentioned the eb-python environment variable. This is actually set automatically by the module that was generated by EasyBuild. So EasyBuild noticed that it was being installed with Python3 so it knows that it should also set the eb-python environment variable in the module file. If we purge, remember we have an EasyBuild module loaded if we purge all the modules we lose the EasyBuild module and the eb-python variable will also no longer be set so that shows it's actually set by the module file. Once we have a working EasyBuild installation we can very easily update EasyBuild by running the install-latest-eb-release command. So this will figure out the latest version that is available of EasyBuild and install it and generate a module file for it. So this is a separate installation entirely. It will not affect the EasyBuild version you are using to install the latest EasyBuild version. It's an entirely separate installation. You'll get a separate module file for it. So that brings us to the exercise. So here it tells you install EasyBuild in your home directory. As I noticed there are some permission issues so maybe change this to install EasyBuild and slash TMP so it doesn't matter too much. So rather than using dollar home use slash TMP home instead. And the recommendation here is to either use pip or the bootstrap. I already showed the bootstrap so I recommend you to try the installation with pip and make sure you get a working EasyBuild installation. So spend a couple of minutes maybe on this and try not to peek at the solutions before actually getting it to work yourself. Kenneth, do you mind me charming in? Sure, go ahead Christian. You created the working directory outside of the container with your user on the EC2 machine and within the container you have a different UID. So I put something in the slack chat to mitigate this. We just need to use the user ID that's outside of the container. So just add what I pasted in the tutorial slack and then you're good to go. So I need it's just this dash as user part. ID minus U mapped to... So this first one gives you and you lose your ID and group ID. Yes, and that's it. They just mapped the user we're using outside into the container. And then you have the right to write in the... I mean you don't have a username but that's okay. Okay. So if you do who am I then it will say I don't have a user. Yeah so this part I guess this is also shared with the people attending. Or maybe somebody can fix it in the practical information in the tutorial. So it just needs a refresh for people attending. So this mapping is required to make sure the home directory is mountable. Thank you for the help Christian. So let me do the bootstrap again as it was intended to work. Or I guess here. So download the bootstrap script to Python 3 bootstrap. And now it should work in our home directory without too much trouble. So as long as you do this mapping here you can do the exercise as is mentioned in the tutorial to install easy build in your home directory. This mapping is required to make sure the home directory is writeable. Again, stage one. Did I overlook something here Christian? Okay yeah so user ID 501 doesn't exist. Okay well I think it's fine as long as people actually a nasty trick will be doing this right. Yeah either this or you create the user with the user ID within the container or you create a user on the host with the user ID and group ID of the user. I think that a very easy workaround is just doing this in the container image. You just redefine home to be in a temporary directory and then it should work fine. Well okay assuming the directory is there. I'm sorry about this issue. We haven't hit this when we were preparing the tutorials. I'm a bit confused why it pops up now. But wherever it says dollar home in the exercises or in the hands-on you can just replace it by a directory in slash temp and it should work fine. So easy build doesn't require things to be in the home itself too much. It does complicate the pip installation a little bit but keep in mind. Yeah this is not working well. I don't think the mapping is good idea. Maybe forget about the home directories in the container and just Yeah if I do the mapping I'm still running into problems. So this is probably what people should be using. It's a bit nasty redefining home that should work. That's fun with containers for you even though we haven't hit this issue during extensive testing. We're hitting it now but it's relatively easy to work around. Oh there's an easier fix in the select channel like change mod to the easy build tutorial directory outside of the container. Should work as well. Okay that's another option. So this directory like this and then I guess we can try again. Don't have anything here. Meanwhile I hope most people have figured out this issue in their setup. I'm seeing people are not hitting it when running Docker locally. That's how I mostly tested the Docker container even though I also tested in AWS. And so if you're not hitting it you can just ignore the workaround that's being proposed. But this indeed seems to work. So the chmod 1777 of the directory we created outside of the container image that seems to be doing the trick. So now we have a bootstrapped installation in our home directory. So remember this is not the way we recommend for the tutorial. We highly recommend using pip3 to install easy build because that will cause a little bit less issues for the rest of the tutorial. So we're running behind a little bit on schedule. We go ahead and purge this and just jump to the solved exercise. So the bootstrapping we already showed. You download the bootstrap script. You run it with pip3 bootstrap ev.py $home easy build. And after a two-stage bootstrap process that should be working okay. Then you can do module use. You can load the easy build module and you can run eb-version and the $eb-python should be set for you automatically. What we recommend is using pip3 install instead. So we basically need to run pip3 install-user easy build. And we have to make sure we set the part to include.local-bin. And we also have to tell easy build to use pip3. So let's try this. It doesn't matter whether you define the environment variables first or first to the installation. Let's do the installation first in this case. So this should pull down easy build from the pip package index. You can see it's split into three parts. Framework easy blocks easy conflicts. And this does the installation which includes a whole bunch of files. Installation was successful. If we don't have a module loaded we don't have the eb command yet however. Until we run the pod command. Until we update the pod environment variable. That gives us the eb command. That doesn't work yet because it's still using the standard python. But if we do the eb-python export to tell it to use python3. This should work. As a charm. So we did a pip3 install by just user easy build and we've updated two environment variables. That gives us a working python installation in our home directory. So this is the recommended way to install easy build for this tutorial. I also mentioned this in the slides. So pip3 install by just user easy build and export these two environment variables at the correct values. That should give you a working easy build installation.