 I think. All right. Let me move this chair. Okay, so I have a short tutorial on writing and contributing back easy config files. I'm not going to explain it in length how to write an easy config file, but I'm going to focus on some of the important points, which will hopefully be helpful. The slides are there. They're also linked on the event page. So you can certainly find them again if I'm going a bit quick. So as I already mentioned, there's a bunch of ways you can contribute back to EasyBuild. Just letting us know that stuff doesn't work or that you're unhappy with something. Reporting bugs, mailing lists, Slack, EasyBuild, Comfort Call. So it does know what you have to be code that you contribute back. So we're happy with pretty much everything you send us back in any way. Certainly also documentation. What we're looking at now is adding support for additional software in EasyBuild. It could be an update of something that's already there or a totally new software package. Basically for every installation EasyBuild does, there has to be a corresponding easy config file. The easy config file defines is basically key value definition. So for a bunch of easy config parameters to specify to EasyBuild what it should use, and you can certainly use the big stack of existing easy config files we have, especially when you're doing software updates, you basically you usually take an existing easy config file in one way or another and you update it to whatever or you try to update it to make it install the version you need. You can do this manually or you can do it using the try options that the EB command has. Certainly for version updates or toolchain updates, they are quite good. And especially in the latest version of EasyBuild, try toolchain works very well together with dash dash robot. So it's a toolchain aware to mention that so it will pick up on things a lot better than it has before. So if you've tried it before, you wouldn't be really happy with it. So definitely try it again. It has improved a lot, mostly thanks to Alan who has worked on this. The basics of writing easy config files is well covered in the documentation, I think. And I think we have a detailed example as well for Worf, as I remember correctly. For most installations, and I don't have the numbers here, but I think there's about 20% of the installations need a specific easy block because the software is just so complicated to it's maybe even less than 20%. But for everything else, we use a handful of easy blocks which we call generic easy blocks. So for installing a Python package, we have one easy block, one Python script that knows how to install a Python package. And for every Python package, you can just tell EasyBuild to use this rather than re-implementing the same logic over and over. We have one for, yeah, have a list of the most common ones on the next slide. If you, if it gets really complex or you're hacking around too much, trying to convince EasyBuild to use a generic easy block in a hacky way, that's probably a sign that you actually need a dedicated easy block for it. But there's a thin line there. It's not exactly clear when to use what. So writing easy config files, again, set of key value definitions for the easy config parameters. The order in which you define things doesn't matter. What we try to keep things in a certain order just so it's easy to compare easy config files next to each other. Or when you see an easy config file, it has name, version, toolchain in a certain order that just helps a bit, certainly for people new to EasyBuild to get started or to easily recognize an easy config file. A couple of them are strictly required. So name, version, and toolchain, that's not a surprise. We also always require you to give a homepage and a description just to have some metadata that we throw into the module file that is generated. So that helps people who pick up on the module to actually tell what kind of software it is. Other commonly used easy config parameters are these. Most of them speak for themselves. You can specify an easy block which EasyBuild should use. It's not mandatory because by default we will look for an easy block that matches the software name. And if we can't find it only then, are you required to specify a particular easy block? So like for Zlib, for example, you have to tell easy block equals configure make and then EasyBuild will use this configure make easy block to install Zlib. If you don't do this, there has to be a software specific easy block that corresponds to the software. Source URLs and sources, just a list of sources that EasyBuild can grab from these URLs if it can auto download them. Version suffix maybe to indicate certain variants of builds like certain configure options or maybe a different version of Python it was installed for. You can list dependencies and you can get control over which configure options, which options for the build commands, for example make that you can pass down to EasyBuild and same for the install command you get options that go after the install command so make install something something. But you can also put stuff before the command and that helps a bit sometimes to hack around to convince EasyBuild to first do something else before running make install. That helps a bit with making the general or the generic easy blocks to be used a bit more. Extensions list this is for Python packages, R libraries, so the stuff that John was talking about with EasyUpdate go in here and then sanity check parts. This is a list of files and directories that are supposed to be there when the installation is done. So pretty early on in EasyBuild we noticed that even if make exits with zero and everything looks fine based on exit codes, things may have gone horribly wrong. So what we started doing pretty quickly is if you're installing Zlib we expect a lib slash libz.a to be there. If it's not there something went horribly wrong and then we fail the installation we don't generate the module file. So these are just a couple of simple checks that EasyBuild does for certain files and directories to be there. And then the module class is the categorization like bio modules, chemistry modules. We try to keep track of that as well. So all of this is thrown into the easy config file. An important one of course is the easy block that you can use or that EasyBuild should use. So we have an auto updated list of generic easy blocks in the documentation. For every EasyBuild release this is updated with a script so it's always it always matches the latest EasyBuild release. And the most common ones are listed here so configure make is still very common just want configure make make install and through config opts you can add specific configure options that you need to enable or disable certain features. By default EasyBuild will do configure dash dash prefix equals install directory and anything else that it finds in config opts it will add there as well. Build opts goes to make and install opt goes to make install and so on. Then cmake make is very similar but rather than running configure we run cmake with a bunch of options to specify the prefix but also to specify the C compiler to use the C++ compiler to use and recently also to disable fancy features of cmake like looking onto the system locations first for boost we really don't want it to do that because usually if boost is required it's included as a dependency and it should pick up on that one and not the system one. So we have to tell cmake like don't consider the system parts for boost and it has a dedicated option for system parts for boost. Python package speaks for itself how to install a Python package you can either do Python setup.py install which is what the default is now or you can tell it to do pip install which we hope to switch to with EasyBuild 4 but I'm not entirely sure anymore after fighting with pip for half a day last week. I'm less convinced now that's a good idea. Python bundle is basically to install a whole bunch of Python packages together in a single installation directory so sometimes a Python package needs three other Python packages that in their turn need two other Python packages and rather than having all separate easy config files and all separate modules for them you can just pack them all in a single installation directory quite easily. Make CP this is used a lot for bioinformatics software it doesn't have a configuration step just has a make file and when the make is done the binary floats around somewhere in the build directory and then they're happy because they have the binary and they don't really care. While in EasyBuild you want to do the build and then have a proper installation directory bin with all the tools in it so you can list which files and directories have to be copied into the installation directory. Darball it just unpacks the sources and copies everything so this is for like a bundle of Perl scripts or something like this and binary can be used to run a binary installer where you can install the or specify the installation command to use so this is typically for commercial software if it's not an interactive installer because then you're stuck to writing a custom easy block for it. So there's a thin line between easy config files and easy blocks sometimes you can get away with quite a lot by leveraging a generic easy block and specifying configure options, build options, install options sometimes with being a bit creative like I won't try and find an example but in the pre-config opts you can specify a command to run and then do a .coma and EasyBuild will glue the .configure command after that or you can do a double ampersand to glue the commands together so you can get away with quite a lot in an easy config file that uses a generic easy block but sometimes people get a bit too creative and they start rather than gluing commands together through pre-config opts they specify some command and they add a hash at the end so they're basically commenting out the part that EasyBuild adds as a configure command. If you're doing this then it's really a sign that you're just hacking around and some people are happy while the check guy is already gone but they seem to be quite happy but yeah it works and I'm done right if you're lazy then you can get around it stuff like this or get away with stuff like this but in the central repository we will not accept an easy config file like this so this may be a sign that you have to implement maybe a very simple but you will have to implement an easy block that does whatever you need it to do or you may be using the wrong generic easy block that may be a better one that's more suited for that particular installation procedure so some of the reasons here so if you have some easy config parameters that are really important to get right that may be a sign that you need an easy block typically that is something connected to the tool chains if you have to do have to use a different configure command when building with gcc another one we're building with intel another one we're building with pgi you can't put this in an easy config file but you'll have to get it right every time you change the tool chain and with options like try to chain try to change will only change the tool chain and nothing else so if the configure options have to follow it's actually easy to just implement that logic once in an easy block and then forget about it because the easy block can react to the tool chain and use the proper configure option based on that so it's a bit of an effort certainly if you don't know python it can be a bit of an effort to do it but you do it once or somebody does it once and then you forget about it that's the idea now some people disagree with this so it's not like at css they mentioned that they rather have an easy config file that stand alone and it's not um yeah you can go both ways i guess tool chain specific things that you have to get right are to me a good sign of writing an easy block interactive commands so if you have an installation script or an installer that starts asking you questions and you have to pick a number from a list and this is what the this is goes what beyond an easy config file can can handle so then you really need an easy block uh configure options for certain dependencies a bit similar to connect it to tool chains maybe a good sign as well so the the goal is really to have an easy config file that just specify what is needed version dependency dependency versions tool chain um if it if it goes beyond just some regular configure options maybe it's better to use an easy block for it yeah i think we used the interactive commands for plume d you know plume d there because you probably used that in the easy block for gromax but for the interactive reply it worked also in the easy config using the python syntax with the i think the the minor signs so it was taking the option yeah so sometimes that there are yeah there's like a silent configuration way you can convince the script not to ask you questions but to give the answers up front if you can do that then sure you can use an easy config file but not all interactive installers have this option and then if it doesn't then you're really stuck the tensorflow is a good example as well tensorflow has the configure script is a python script that calls something else that starts asking you questions and if you go into the script you can actually find environment variables that you can define if it finds those it will not ask you the questions so if you know about them you can use them but this is not documented anywhere so you have to dive into the script yourself and and figure it out so yeah and anyway for tensorflow all hope is lost to do it only in an easy config file forget about it ask Guillermo yeah well he's not here anymore um you can probably get get around to making it work but you'll be hacking quite a lot and the easy config file will grow a lot and if you change the version you may have to go back in and see why you did certain things in the configure options and yeah it gets hairy and don't get me wrong the easy block for tensorflow is not easy either it's pretty complex because of the way they do things but at least we do it once and there have been several recent tensorflow updates where we just bump the version and it just works so there was a lot of effort initially to implement an easy block for tensorflow but then it becomes a lot easier okay so when you're writing easy config files and you're considering considering to contribute and back there's two very handy features in easy build that are fairly recent it has an inject checksums option and you can just give it a list of easy config files to consider it will post the easy config file figure out which source files and patch files are needed it will compute the shell 256 checksums for those and it will add them back into the easy config file so rather than having to do shell some manually and copy pasting that easy build can do all of that for you so that's quite handy and we have a hard requirement now for any pull requests in the last year or so that the checksums are there for all sources and patch files if one is missing the test will fail and we will not merge the pull request so this is very useful to have and slightly related to that we have a check contrib option that you can run on easy config files it will do some checks to try and make sure that the easy config file is ready to contribute back so it does a pre-check of some things we check for the checksums is one part of it another part is we check for code style issues so no tab characters in the easy config file yeah some silly I forgot which ones pop up sometimes but there's some some yeah we try to have some strict ish formatting in easy config files to make it easy for people to jump from one to the other and if you don't do this you can still make the contribution but the test will fail anyway so it's good to do it manually up front and just quickly run the the check yourself before you do the contribution so suppose you have composed an easy config file it worked for you or maybe a set of easy config files that work together you need a couple of dependencies so how do you contribute this back so we will this is also well covered in the documentation in full detail so the intention is that you open a pull request to the develop branch of the central easy config repository on github we like that you keep the PRs limited in size so just to make it easy to review and test them if you have a hundred easy config files in there and first of all you that costs like one or two hours just to take a look at all of them and do some reviewing but then we may get back to you and say for these 10 please make this change and then you make the change it just makes the process really long so we like to keep them limited to let's say 10 easy config files touching 10 easy config files is a good rule of thumb but usually it's like one or two or three that are back together in the PR once you've opened your pull request we like that people submit a test report and I'll show you how to do that to to basically show that the easy config files work for you and on what kind of system it was tested centOS or your boot to which python version which easy build version which easy build configuration so that is all fully automated you can easily easily do that after a bit of initial setup and then so we have a bunch of small checks that run automatically when you open the pull request these are things like for an easy config file can I find other easy config files for all the dependencies that are listed is the toolchain unknown toolchain all of these things are checked are you not introducing any conflicts in the module three so you can actually load the module and it's built so these things are checked pretty quickly in a matter of minutes but it's not actually doing the actual installation so that doesn't happen on Travis and then one of the reviewers one of the maintainers usually you know other people can review pull requests as well there's a visual check of the easy config file and maybe asked for a couple of modifications and we have a good way of doing visual checks I'll show this as well so you may have to make some changes based on both of these and you also may have to be a bit patient as I showed you this morning we get 1600 PRs for easy conflicts that's like six or seven per working day on the yearly basis that's quite a lot so we won't jump on the pull request minutes after we've opened it it may take us a while hopefully somebody will look at it and if it gets forgotten definitely bring it up somewhere in the Slack channel or hopefully we're not overlooking your contribution and hopefully it gets merged in the end and then your easy config is included in the next easy book release and you get eternal glory from that so one thing you need to know a bit is git like to open a pull request to to github you need to create a brand well a git branch you need to add stage or easy config file in the right location of the easy configs repository do a git commit do get push out to github login to github click around a bit and because most of the easy build users are not software developers people were running into trouble because of that or were getting frustrated because they didn't know git well enough or even github well enough to make contributions so that was annoying so basically first of all you need to make sure you have a github account like some people don't yet you may have to create it you have to fork the easy configs repository so you you sort of take a copy of it under your own name and then you dive into the terminal you clone either your git repository or the central one so you can do both and depending on that you may have to change your workflow a bit and then you start running git commands you create a funny branch name that you're going to add the easy config file in you put the easy config file in the right location so we have this organization easy build easy configs the first letter of the software name slash software name slash all the easy configs we have to put it there with the right naming you stage it with git add you commit to with the descriptive message or the easy config file then you push it to this is probably your fork of the easy configs repository that's part one then you log into github if you're quick enough you get this button here to open the pull request or you may have to find your branch and then open the pull request a couple of details there so this is a window like this a couple of details there is we want you to target the develop branch not the master branch so master always corresponds to the latest release develop branch is what we're preparing for the next release ideally we'd like you to have a sensible title for the PR so we we have this standard format module class toolchain software name software version and maybe some common dependencies like python just to have a when we look at the list of pull request that we have an idea of what's in there and having this standard format helps us a lot maybe you need a small description for your work you better double check how many changed files are in your branch because maybe you added something by accident that you actually don't want to contribute so be a bit careful and then if you're sure you can create the pull request and then we're not done yet so once your pull request is open travis will do the quick checks on it and hopefully everything passes there one of the maintenance will review your pull request if you're not sending a test report yourself somebody else will test it probably the guy who's reviewing the pull request so you he submits a test report for it and the test report is a hard requirement we will not merge any easy complex pull request unless there's a working test report so if you're not doing it we like you to but if you don't that's okay then we will do it ourselves you may get some requested changes and if you do you'll have to get back to the get dance and figure things out so how do i get back to that pull request and make any changes and you may need to do the cycle a couple of times so say you get a remark for your contribution you remember the branch name that was corresponding to your pull request you use vim what else would you use to change the files you commit the changes if you commit all changes you better make sure you don't have anything else in there and you push the branch again to your repository to update it so this is try explaining this to somebody who doesn't know get or get help it's like okay i don't want to contribute anymore right too much work too much manual steps too much things you have to know develop branch easy config file name where it goes i need to get all these tests right and you have code style policy and ah no i'm done and that's assuming you don't screw up with git anywhere which is not that hard to do you can easily shoot yourself on the foot and either add too much changes in the branch or suddenly blow away what you have and then you can't find it anymore so yeah people were not very happy with that together with this situation we saw that this was back in 2015 in the years before we were getting about 500 easy conflict PRs of which a big part were our own so they are part of this for sure but in 2015 it started exploding a bit we're getting lots of contributions from people who are familiar with git and this whole procedure but some other people were letting us know like i want to contribute but this git thing i would have to learn git to contribute to easy build and it's too much for me i don't have time for this so here i give you a tarble with all my easy config files and you can add them yourself which is yeah not the situation we were very happy with and it also clashes with the idea that easy build is used for automating things but then you have to do all this manual work to contribute back so that's not okay so at some point i think it was christmas 2015 i was fed up with this started hacking a bit i started working on this github integration stuff so the idea was to try and automate the contribution workflow so rather than having all these steps manually having something probably easy build doing it for you and even second goal was to try avoid direct interaction with git and github so you don't have to actually have to know any git or github everything is done under the covers so that's what we did we added support in easy build for opening new pull requests updating existing pull requests downloading easy config files from a pull request you can easily test them without having to copy the easy config files manually you can just ask easy build i know there's a pull request for a new version of cmake i want to use it so you from pr you give it the number of the pull request and easy build pulls in the easy config file and builds it and also for uploading test reports we wanted to have a standard way of doing that that gives us all the information we need in the nice format so it looks all standard and we don't have to figure out for every contributor what they did exactly or how for now it's still limited to easy config files but the concept can actually also work for easy blocks or even for framework so we can let you hopefully in the future hack something in framework make a change that you need to fix a bug and just tell easy build figure out what i changed and open the pull request for it that's still not there but in principle we could make it work under the covers this features uses uses git python which is a python library to talk to git which actually sucks i actually want to get rid of it probably an easy build for rather than using talking to git python we'll just straight up run git commands because there's no real value in using git python at least not in my view maybe i'm missing something but i don't really like it too much we're talking to the the github rest api which is very nice and very well documented and that avoids that you we make you click around on github to open a pull request once you get a github token so like a way that you can talk in an authenticated way with github we can actually automate all the clicks that you have to do to open a pull request which is quite nice and that includes things like targeting develop branch all these all these things you have to remember and then we use the key ring python package which is also a bit of a hassle but haven't found a better alternative to securely store your github token so easy build can load it whatever it needs to talk to github so how does this work so first of all there's a bit of configuration you have to do for all of this to work um you have to tell easy build what your github username is so we have configuration options for all of these things or most of these things you have to give it your github token so we have a command line option where i think it's install github token um you can copy paste the the token in a secure way so you won't actually see it on the on the screen and hit enter and easy build will make sure it's actually a valid github token so you'll test it it will do some simple interactions with github to make sure it's a token for your user which is the case here it will check for the git command because it needs it to run git it will check for git python currently and make sure it's a sufficiently recent version it will check whether this token gives push access to your fork of the repository and whether this fork actually exists and we also use this github gist to for the test report so we make sure we can create gist as well so you can you can create a token that has certain limitations on it so that's why you have to check these things separately and then this is actually less of a less of an issue you can make sure that the commands i will show work a bit faster by telling it where where you have a clone of the easy complex repository already in your process so it doesn't have to download the whole thing yeah okay who is joining the brewery visit the second one most people are okay well i can probably stop here because the stuff that's coming have to do it all at once pretty much but we can interrupt here yeah and continue how long does the tour take 30 minutes yeah so let's let's do that yeah does that work Alan yeah okay so we're gonna that's fine we'll stop the session here because there's a a brewery visit so for the people on the stream the people that are in the room are going out for beers so i will be talking to the walls if i continue so we'll interrupt here and we'll continue in about 45 minutes yeah so say five p.m we'll start again with the rest of this tutorial