 Let's move on to the troubleshooting part. So this is the next part in the tutorial. Sorry, we will take a look at easy build error messages, easy build log files, the build directory, and then also do a couple of simple exercises on this. So here we're doing the troubleshooting part in the tutorial, which looks like this from the start. So as I mentioned before, easy build is not magic. You will still run into problems like compilation errors or if you're out of disk space, if the download of source files fails like we have seen up for bow time because there are problems with source force, we can still run into problems. So it's important to be able to troubleshoot failing installations as well. For this, we need to pay attention to the error messages that easy build produces. So they can be a bit verbose, but hopefully what we will cover here will help you in making sense of them. And we have to keep in mind that easy build has a detailed log file of how it's performing the installation. And we may need to jump into that every now and then and to figure out what actually went wrong and in what context it went wrong. And then there's also the build directory. So that's the location where easy build will compile the software from source before actually installing it out to the target directory. And we may have to jump in there as well and see what's actually going on. At the end of this, we'll have an exercise which lets you do exactly this. Look at error messages, log files, and the build directory. This is a fun exercise to do. Try and find some time to take a close look at this. Let's first look at easy build error messages. So when something goes wrong, easy build will try to produce a short and hopefully helpful error message for you. So things that could go wrong are missing source files like we saw with bowtie. Maybe the downloaded source file is corrupted somehow or it's not what easy build expects so that in that case you get a checksum error. Dependencies could be missing like we saw in the example with bowtie where the TBB dependency was not installed yet. Or there may be even dependencies missing in the easy config file which were not declared. So easy build would not know to make sure that this dependency is in place and then the installation may fail because it's a required dependency. Some of the shell commands that easy build runs may fail for a variety of reasons. You may run out of disk space or out of memory during an installation especially when compiling C++ software, this is not unlikely to happen. Or you may run into segmentation faults which happen even for fairly weird reasons. And there's a whole bunch of other things that could go wrong as well. This is by no means an exhaustive list. One key thing that easy build will do when running shell commands is it will check the exit status. So the exit status is a standard thing in Linux. If the exit status is zero, then that's a sign that the command ran as expected. And easy build will in that case assume that the shell command ran correctly and happily continue. If it sees an exit, the status different from zero, easy build will assume or will know that the command failed and then it will stop the installation at that point and not continue. And basically throw an error at you that you may have to look at to figure out what went wrong. So here's a quick example. It is a bit bigger. Here's a quick example of an error message where we're running or installing an easy config file named trouble. And we're getting an error message that says G plus plus error unrecognized command line option standard C, C plus plus for T. So here for some reason the compiler was passed this particular option and it doesn't know about this option, which makes the compilation fail and of course the installation fail as well. There's a whole bunch of things going on here. Easy build will tell us the location of the build directory where the command was being run. It will tell us which command was run. In this case, a simple make command which exit it with a non-zero exit code which means it failed. And in this case, we're lucky to see both the full command and the error message in the output here. So this will not always be the case. It also gives us the location of the log file that easy build created. So we can look into this to get more details about the context and which this error occurred. If we try this, so this is a fictional example but in this case we were looking at what could have gone wrong here. So G plus plus was not recognizing this command line option. So we were looking at what the version of G plus plus was which was 485, which is quite old. And that probably explains why the C plus plus 14 standard is not known yet. So in this case, the problem here boiled down to easy build or rather make running this hard coded location to G plus plus which is the old system compiler which is not aware yet of the newer C plus plus standard. So the real issue here was that this location was hard coded rather than just running G plus plus and letting easy build control which version of the G plus plus compiler was being used. So that's a short example of an error message. We will run into them hands on as well a bit later. Easy build log files which I've mentioned before are a very comprehensive log of what was going on during the installation, what commands easy build was running, which environment variables were set which modules were loaded in which direct readings were being run what output was generated and so on. All of this is captured in the easy build log file. So this will be a very rich information source in case something goes wrong or to check on how an installation was done later as well. During the installation, the log file will be located in a temporary location. And if the installation fails, that temporary location is not cleaned up. So you can check the log file in case of a failing installation. If the installation worked, the log file is actually copied into the installation directory itself. You can check it there later in case you want to see how a particular installation was done. So for example, here we see the log file being copied for H5Pi into the software H5Pi version directory in a sub directory named easy build. So if we check this easy builds software H5Pi this particular version, you'll have an easy build directory here, the sub directory of the installation. And this has next to a bunch of other things including the easy config file that was used. We have a log file that details everything about the installation as well. So if the installation is successful, this log file is copied into the installation directory for later inspection. There's a handy trick here. Didn't add any text to explain this, but the EB last log in case of a failing installation. And I've probably have one here from before. So this is a failing installation of H5Pi. I think it's the one I interrupted before. This is the last log file that easy build knows about of a failing installation. It just prints the parts to the log file, but it's useful because that way we can quickly open it in an editor, let me do it with less here. So we use anything to open files with and we do this as a sub command in the command. So this will print the location of the log file. We grab this location and we pass it to something else which will open. But that way you don't have to copy paste the location of the log file yourself. You can just use this trick to access the last log file produced by easy build for the failing installation, which is quite useful. If you're stuck in the less environment, by the way, just use the letter Q to exit. So navigating log files can be a bit tricky because they're very rich and dense in terms of information. When you're looking into a failing installation, there's a couple of specific patterns that you may want to run in or may want to search for. So error and capital letters or error followed by a number, error directly followed by a colon or other ways of reporting errors like failure or not found or no search file or directory are common patterns that pop up when during failing installations. They may not, the errors may not always be at the end of the installation log. So that's why it's important that you may have to look or search through log files to find the actual error. Also, you typically want to, depending on your strategy for looking for the error, you either jump to the end of the log file and then scroll your way back up. So you're looking at close to where the installation failed or you start from the top of the command and then you look for a particular pattern because typically what you want to look for is the first error message because the other error messages may be just a result of that first error. I mean, we will do some of this exactly during the exercise. Here's a couple of directions for using the less command when viewing a log file, you can hit some buttons on your keyboard to jump to the end of the log file. So a dollar followed by a capital G will jump to the end of the log file. Then you can use our error keys to scroll up or down or you could search for particular patterns using a question mark or a forward slash to search backwards or forwards in the file and to exit the queue. This gives a short example of how the contents of a log file looks like. It's pretty structured. So there's multiple lines and every line generated by EasyBuild will always start with a double equals sign followed by a date stamp. So you can see how far in the installation it occurred. You will get the name of either the Python module that's part of EasyBuild that generated the log message or maybe the easy block. In this case, it's an easy block generating the log message. And you will get some informative information like this is the configure step and this is the method I'm running that's part of the Python module that is the easy block and so on. When it's running specific commands, it will look like this. It will tell you it's running a command and this is the command it was running. And when the command completes, which you can tell here it was six seconds later, it was done running make install, it had an exit code of zero and this is the start of the output. So if you're looking for the output of a particular command, you can run or you can search for a pattern like this running cmd colon until you hit the specific command that you want. And from there, you can start scrolling down and see what the command generated as output. Finally, we can also, or we may also have to take a look at the build directory. So if EasyBuild prints an error, part of the error message will be the location of the build directory, which is shown here. And we may have to dive into this build directory with LS and CD and check what's there and maybe open some files there to figure out why installation failed. One case that often happens is that the configure script for the classic auto tools configure script. And if this fails, you will get a config.log file, for example, and you may have to open this file to look for clues why the configuration failed. In the case of CMake, you may have to look at CMake output.log or CMake error.log, which are in the sub directory of the build directory. And remember when installation failed, the build directory will stay there. So EasyBuild will not clean up the build directory for you. So either you have to do this yourself on a regular basis or you just ignore it, as long as you're using slash temp for build directories, it's probably not a huge issue if you leave it sitting there for a while. As long as you have enough disk space, it shouldn't cause any big trouble. So here's an exercise for you to play around with. I'll give you a small head start on this. So this is an easy config file that's also located in the container image. In EasyBuild slash tutorial slash subread.ed. So this file here, if you open it, I'll just print the first couple of lines here. It will tell you this is an easy config file for the troubleshooting part. So this will not work out of the box. The idea is that you try running this or try installing this software using this easy config file, see what goes wrong and try to fix the problems you run into. How does that look like? You scroll to the exercise here. So this is showing the contents of the easy config file just for reference, but as soon as you try running it, so here we're copying it to the, we're making sure we're in our home directory. We copy the file here. So subread.ed is now in our home directory, so we can edit it. And let me maybe open it in the file browser here. So you can click here, open this directory and double click subread.ed. This opens it in the editor here in cloud nine. This is the exact same contents as is shown here in the start of the exercise. Now we make sure that we have the pre-installed software available and that EasyBuild is active. So if you don't have EasyBuild as a module, you don't need to run the module load EasyBuild command. We installed EasyBuild with pip, so it should already be there. And we can check whether the software is there by checking for GCC, for example. So that two chain is already here. So that gives us a good starting point. You can make sure EasyBuild is properly configured by running these two export commands, which should already be in place, but it doesn't hurt to make sure, and you can double check that via show config. So with the small bit of preparation in place, let's try running or try installing this eb-subread.eb and see what happens. So first of all, what fails is it cannot find the source file. It's very annoying, and that's the first problem you should try and fix. So maybe take a look at the easy config file and see if you can fix this issue yourself. In the meantime, again, I'm happy to answer any questions if they pop up. Maybe if you're done with this first part of the exercise, let us know through the Slack channel so I can know how to quickly continue. Let me set up a quick poll for that. I'm getting a handful of people who are done with the first exercise, hopefully without peeking to the solution. So let me try and fix it myself. And of course, this thing is located on Swarchforge. It's a good thing I prepared a bit for this. So it's mentioned in the solution. So what you should be doing, the best possible solution is to make sure that EasyBuild can download the sources for you. So you can make sure the source URLs environment variable is defined like this. So we define source URLs with the location where the source file can be downloaded from without the file name. So just the URL, the file name will be added by EasyBuild itself. In theory, this should be enough. Okay, and now this is screaming at me because I don't have the right permissions. So I'll just do it here directly. So we need the source URL here, like this as a string in a list, Python syntax. And this should entirely be enough for EasyBuild to download the sources itself. But in this case, because of the Swarchforge trouble, it may not be good enough. It actually is, okay. So it was able to download the file for me because it passed the fetch files step. And now we're getting a different error for the tool chain. So please take a look at that and see if you can fix it. There's a question on the character limit, the 300 character limit. I think there's a configuration option for that, but I'm not 100% sure. Maybe not. So it's possible this is hard coded in EasyBuild. I think we changed that at some point. So maybe take a look at eb dash dash help. There may be a configuration setting for this. Like Alan mentioned, you almost always have to dive into the log file anyway, since it's unlikely that you actually get the actual error in the output of the eb components. This is something we're gonna try and work on to improve this a little bit, to improve the error reporting that EasyBuild does. It can probably do a better job at pinpointing the actual problem for you. It doesn't have a lot of intelligence for this right now. In case people are hitting problems with downloading the sources, they are also located in easybuild slash easybuild slash tutorial. So you can just copy the source file from here and you should copy it into EasyBuild sources in your home directory and the subread directory. So this file and this file here should be one and the same. EasyBuild has trouble downloading the source file for you. You can download it manually through another location and copy it into the source path. Who has figured out the second part of the exercise yet the tutorial problem, the tool chain problem, I'm sorry. How could you fix that easily without having to install the missing tool chain GCC 8.4? I won't stall too long here because we're going off schedule quite a bit. An easy fix for this is to just use the tool chain that we have, which is GCC 9.3. So in this case, we don't really care too much which exact tool chain is being used. So we can just change the version of the tool chain to 9.3 and then we're good to give it another go. Now it's failing in the building step and there's a whole bunch of output here but there's no real sign of an actual error message. So we can tell the build failed during make but we can't really tell why exactly but we do get the location to the log file. We can open this log file and inspect it. The top of the log file probably won't have much useful information. So we jump to the end of the log file and start scrolling up. And even here we can already see the actual error. GCC says you tried dash fast as an option which is a nice try but I don't know about this option. So I'll fail the compilation and give up. And then this results in a non-zero exit code for the build command which is make with some options which eventually fails the installation. So where does this dash fast come from? We look into the easy config file. We can see here built opts is specified as, so these are options for the make command as dash F and a particular make file to use. And it also passes here C flags equals minus fast. So this was a very naive try to try and get this use as a compiler option which is not working. So let's just remove this. Easy build will try and make sure there's a proper set of compiler flags passed to the compilation already. So we probably don't need to do this at all. Easy build will make sure that the compilation is done by default in a good way that is targeting the processor of the host we're running in. So let's just clean up this build commands and get rid of the dash fast instead. And run this again. And now it takes a quite a bit longer. So I guess that's a good sign. So at least not failing straight away. Let's see who figured out 5.3 already. Oh, that was actually 5.3 already fixed. 5.3 was the fast option that we got rid of. So 5.4 is the failing sanity check that we're hitting now. So here the installation actually completed. So we have an installation on disk. We check our home directory. Easy build software. Subread. We have an attempt at an installation which is probably gonna be empty with a compiler version we don't have. And we have one with 9.3 which seems to have something installed. It does have a bunch of binaries but the sanity check is not happy. And it seems like people are figuring that out as well. If we look at the error message we get, we get unrecognized option dash dash version for the feature counts sanity check command. So the sanity check is running a command but this is failing. If we look into the log file here and again, jump to the end. We get the output of the feature count command. It tells us all the options it knows about. And we can tell the very last option it gives us is a minus capital V. So this is how we check the output version. We're not supposed to use dash dash version but just dash V. So again, this is specified here in the easy config file. We can change the dash dash version to a dash V and that should probably be a better way of doing the sanity check. Now we can redo the installation but because we know the installation has actually completed and it only failed the sanity check, we can skip having to recompile and reinstall the software. We have an option that only generates the module file and does the sanity check, which is module only. This will be a lot quicker of course than redoing the whole installation from scratch. You can see that a bunch of steps here are being skipped. The sanity check is not being skipped. Neither is the module creation step of course and because we fixed the sanity check now the installation is completed. So that's good news. And if we load this module manually, we check with subread, module avail subread, module is there, we can load, oops, we can load it and we can run the feature command, feature counts minus V command ourself after loading the subread module. And that brings us to the end of the troubleshooting part of the tutorial.