 I can talk a bit so you can test it before we actually start. How does it work for the recordings? You have to interrupt the stream so it starts a new video? Yeah. Yeah. Want me to keep... Sure, yeah, just start it and let's see if it helps. Still the same issue. So for the people watching the stream, if you have any issues, just join the Slack channel that we've set up for the meeting and report any problems there. Now it should be fine, okay. We're good to start, let me know. Yeah, good to go. All right. Good morning, everyone. Thank you all for coming. I'm very happy to see a big group of people willing to make the trip again. This time to Louvela Neuve. For the Easy Build User Meeting, more people will be coming tomorrow as well and some people will be leaving a bit early so there will be a bit of moving around. But there will be about 40 people attending the meeting which is roughly the same as last year so I'm very happy with that. For the rest of the morning, I'll give an overview of what we've been up to with Easy Build the last year or so. I'll discuss the results from the survey we did and then also give a bit of an outlook to what is coming in the coming year. The slides are here on this link. I also uploaded them to the event page so you see a small attachment sign there. You can get in there as well. So very briefly, the HPC again team I work at at Gantt University. So we're part of the central IT department at Gantt University so we cater to the whole university, not to a specific group of users or a specific domain. We provide the hardware, we do training, we provide support so the whole package basically for everybody at university also companies and research institutes connected to the university and even throughout all of Gantt. So we have the equivalent of SESI here in Wallonia. We have the Flemish Supercomputing Center in Flanders. It may seem a bit weird to people why do we have two centers in a small country like Belgium but the answer is basically where the finance income comes from. So the Flemish government pays for our infrastructure, the Wallonian government pays for the Wallonian infrastructure so you have two separate centers which is not in a country small but it's politics. And EasyBuild was created at HPC again now almost 10 years ago and we are still doing the lead development and let's say the, yeah, just keeping it going. Who am I? So I joined our team in October 2010 and just a couple of months after I joined EasyBuild was thrown into my lap or what has become EasyBuild I should say because it was nowhere near what it is now. It was, let's say, a bit of a mess, a big pile of code that was doing something and it took me a while to actually figure out what it was doing, let alone how it was doing it. We cleaned it up over the years and it certainly got a lot better after a couple of years and then slowly I became the lead developer because Stain who created it originally didn't have time for it anymore and then this community exploded around it and I had to do this community management, release management as well. Like a whole bunch of things including beer and you'll notice this through the coming days and certainly I toss them as well. I like stickers, I have a big box of stickers, if people have stickers to exchange I'm happy. I already have a SPAC sticker and I need another SPAC sticker. There's lots of things I don't like as well and most of them are connected to EasyBuild somehow and some of these will pop up through the talk as well. Last year at FOSDEM I did this funny talk, how to make package managers cry. This was done after seven years of fighting scientific software and I wanted to give a message like most people are doing this very wrong and people connected to it so the room exploded when the talk was going and like 10,000 people have watched it on YouTube. Some people seem to agree with what I was saying so that was cool. For the meeting this week in the end we have 39 people attending. There's 41 on the list but one is remote and somebody cancelled. So we have 39 in the end, 10 different countries, same as last year. Couple of companies as well, some of them already here, others are coming. Lots of local people, no surprise, about 10 people from the SESI. We have another three people from the Flemish Center and then throughout all of Europe we have two people who flew in from the US. I don't know if they're here yet, maybe they're still a bit jet lagged but they will be coming. And also one brave person from the far non-European country, that's the UK. So this is pretty similar to what we had last year in the Netherlands so hopefully we can continue this, to have this broad community attending the meeting. The agenda, so I'll be talking most of the morning. This afternoon we'll have site presentations, so sites showing how they are using EasyBuild and in the late afternoon we'll have the again me talking, showing a tutorial on creating easy config files and how to contribute them back and then we'll try to play around with this a bit hands-on for those who want to. And there's also the brewery visits. So some people will miss, if you attend the first brewery visit, you will miss the tutorial. So maybe you can still change your mind about when to attend the brewery visit but that's up to you. Some people here already know about the contribution procedure, so. Then tomorrow we'll have quite a lot of talks actually, so we try to do 50-50 hands-on and talks throughout the meeting. It's sometimes a bit unbalanced but I think it mostly worked out this time. Grigory who is not here yet will talk about his collective knowledge framework. Adrian will talk about OpenHPC, Victor will talk about ReFrame. Then we'll have mostly hands-on session but with one remote talk in between from Jan in Spain who's not able to join us physically but who wanted to give a talk as well. And then Eduardo who flew in from the US will talk about Singularity in the afternoon and we have three more remote talks back to back. So we try to plan these back to back just for practical purposes. Then a bit more hands-on maybe if there's time. And there's also the tour of the data center so for people who want to and then we'll have the group dinner that is sponsored by HPC again. So thank you very much to Ewald for covering this. And then on Friday we have a couple more talks Xavier will be talking about his modules tool. We have a talk by Olivier, one of the local guys about this Ansible module. Alan will be giving some thoughts on how we can score some funding in the EuroHPC project for EasyBuild. And then Massimiliano will be talking about SPAC so giving updates on what SPAC has been up to the last year or so. And then more hands-on. So we have less talks on Friday, more time for hands-on or people that want to or have to leave early certainly can. At the end of Friday a large part of the people here will be moving to Brussels to attend the FOSDEM meeting as well. So you're certainly welcome to join us. All the talks here will be live streamed and recorded. So we actually started the YouTube channel like last week to collect everything. All the talks from previous meetings were also recorded so we have everything recorded. That's very nice to have a historical record of that. And these are linked through playlists in the channel. And also any talk I could find on EasyBuild they also linked there. So if something interesting pops up you can certainly look at it again later. Then to the survey. This is the second time we did a user survey. It was an idea that popped up last year. Last year we did it the first time, this year we did it again. I feel it's a pretty good way to get a better insight into the community. So what people like, what people don't like, how they are using EasyBuild and so on. So we sent out a bunch of invitations through the known channels. It was a reasonably short survey about 10 minutes to fill it out. We got close to 100 people responding so that's more than last year. That's very good. At least in my idea it gives a fairly relevant view. There are a couple of weird things that pop up in the survey that cannot be correct. So to take it with a grain of salt. And I'll go through all the results here in between other things. EasyBuild community, this is a picture from last meeting in the Netherlands. It has really exploded beyond what we ever expected. So we originally made EasyBuild available with the idea to get some feedback on it from people who were having another tool or had a better way of dealing with software installations. And after, let's say a year or two, we found out that we really broke some new ground. So we had a tool that was helping out a lot of people and lots of people started picking up on it. And the community grew around it so that's very nice. And we tried to be very welcoming and supportive to new people. So people that do not know EasyBuild at all yet can easily find help on the mailing list on the Slack channel or in meetings like this. So you don't have to be an EasyBuild expert certainly to attend the meeting or to start asking questions. In the survey a whole bunch of things were asked. So the first question was what's your primary profile? So what kind of role do you consider yourself to have? Lots of people consider themselves to be system administrators or user support. So there's a clear bias towards that in the EasyBuild community. I think if you would ask the same question in the SPAC community it would look very different. I would assume that there's more software developers like only 6% here consider them to be software developers. So I would think if SPAC ever does a survey like this maybe this would be a good question to ask just to see how different the communities are. What type of organization do people work for? This is pretty much all across the board. And it's very similar to what we saw last year. So the community hasn't shifted significantly in the last year. Lots of people from Europe no surprise I guess because EasyBuild grew in Europe about 30% in North America and then a couple more across the world. We saw a drop here in North America and a bit of an increase in Europe. I think this is due to SPAC because SPAC is very big in the ECP project. I think more American labs started using SPAC or switched away from EasyBuild to SPAC and I think this is reflected here. Who knows? This is just my interpretation. I know it's very much relevant in the ECP project. I guess that may be why we're seeing this shift. I'm just guessing. Maybe North Americans didn't feel like completing the survey. That's also possible. How long have people been using EasyBuild? So lots of people have been using it for at least two years and there's a bit less new people let's say coming in but that may also be because people that have been using it keep using it and are more keen to fill in the survey. I'm not sure. For the people not very familiar with EasyBuild yet I'm not going to really introduce EasyBuild here. I kind of assume you know it already. One important thing to remember is that it's not the replacement for standard tools like CMake or Make or anything. It basically wraps around it. It implements the whole installation procedure for you so you can basically push the button and it does whatever it has to do to install the software. So we're not trying to replace CMake or anything even though maybe it should be replaced. It also doesn't replace your more apt-get so the OS package managers you will still need this but maybe it helps you to minimize the packages that you have to install in the OS through this and EasyBuild allows you to install them in an optimized way for the hardware you are running the software on and so on. And it's also not magic. There are compilation problems. Installations will still fail but hopefully less so than you were doing it without EasyBuild. So if people run into problems there and figure out what magic compiler flag to use to fix the issue it's hopefully already any EasyBuild and you don't have to even worry about it anymore. The way I see EasyBuild is it's a uniform interface with installing software so no matter whether it's a Python package or something that uses CMake as you see like open foam you just talk to EasyBuild and EasyBuild knows what to do. And that way it can save you a lot of time so you don't have to figure out the problems or the installation procedure and it automates things like generating module files and all of this checking on itself to make sure the compilation actually worked and produced the binaries and all these things you don't have to do EasyBuild does it for you. It helps to provide a consistent software stack to your users because no matter who uses EasyBuild it's going the same way according to how it was configured so that really helps. And because lots of people contributed it has really become an expert system for scientific software installation. People have fixed problems, people have optimized things use the right flags for the right packages so it runs well and it hopefully does what it's supposed to do once it's installed. People are collaborating on it so that's very nice. We never expected this to happen but it did it even allows the scientists, the users of the software to manage the software stack themselves if they want to. We actually see that more and more in Ghent especially our biggest group which is also a quantum chemistry group basically doesn't come to us anymore to install software, they use EasyBuild to install their own software because it has become that easy just change versions, change compiler or configure options and it just installs in their shared directory for their group and they are happy. So we get less work and they are happy. That's perfect. Back to the survey, what was the main aspect that made you use EasyBuild? Most people just say the central functionality in the EasyBuild framework. So the whole automation of the installations generating module files and giving you all the control you need to make EasyBuild do what you want to do. That's certainly the biggest factor and then the next one is supported software and that's very similar to what we saw in the last survey. How did you learn about EasyBuild? Pretty similar to last year, it's still word of mouth so people telling other people or giving a talk like you should look into EasyBuild, try it, it's really going to help you. So we shouldn't spend thousands of euros on marketing. It may not convince people, just hand out stickers and let people talk about it. That seems to be the best way. Today, EasyBuild has been stable for over six years now. So it started, as I already mentioned, in summer of 2009. In HPC again, after a couple of years and after a couple of summer interns who cleaned up the code base quite a bit basically turned it upside down which is not something we had time for but they basically redesigned it from the ground up and put the pieces back together and at the end of their internship it was better organized. One of the interns actually told us you don't have any tests at all for this. It was back in 2011, why not? And he said, we don't feel we need it. So he started adding tests to it and then we basically did it for everything, every change we made. We made sure there was a test covering it, every bug fix which has really helped with keeping it very stable. At Supercomputing 2012 we did the first stable release literally the evening before we gave a talk on it and that seems to have worked out quite well. Some people started picking up on it after Supercomputing and so on. The community says here emerged but it really exploded around it over the years. We've been doing frequent stable releases since then as the timeline here shows. And the development has really become community driven. So people contribute stuff, people report bugs that we are not aware of or that we're not hitting ourselves because we use it in a different configuration and that's what drives the contribution or the development rather. Most people are using the latest release like the vast majority is using the latest release that's very good. That means the effort mostly I do or the effort we do as the community to push out stable releases is really useful. So either they're using the latest release or let's say a recent 3.x release that's the vast majority. There's one or two people still stuck at Easy Build 2. I'm not sure why because the changes between 2 and 3 were not that big at all. They should definitely look into making the jump to 3 and it shouldn't be much of a pain to change it. And some people are very brave and are using our develop branch. That includes myself and probably most of the maintainers. Who know that develop is okay if you know what you're doing. You just don't do this in production like I do. It's quite similar to what we saw last year. And people who are not using the latest release so that's the color part here. Some people just haven't looked into it because of time or they're happy whatever they're using now that's okay. A minority of people ran into problems like there was an issue with the latest releases in combination with GC3.pi because of a bug in GC3.pi not because of a bug in Easy Build. So it prevented people from updating so that this does happen and we try to if these issues are reported we try to fix them as soon as possible. The frequency of release seems to be very good now. This is actually a bit better than the previous survey. So it was 65%. Now it's 81%. I'm very happy with this because every release takes at least I would say a day, a day and a half of my time to run all the tests, check on the tests and just puzzle the release together. I've been getting a big amount of help from Miguel who's down in Singapore putting the release notes together and it's not a lot of work but it's fiddly and you have to collect things from several places and that has been really useful. So maybe watching this stream so thanks a lot Miguel for helping out with that. This is an interesting one. It's a new question. I don't think we asked this last year. So on which operating systems are people using Easy Build? Most commonly. So they could answer, they could give multiple answers so we can't sum these or do a pie chart. To do a bar chart like this but 70% of the people are using it using it on CentOS7. So it's maybe not a surprise. It's the most common OS in the HPC community. Most of us know that. There's all the green ones are the red head based distribution. Quite a lot of people are still stuck on CentOS6 for their older systems. That's important because CentOS6 has Python 2.6 in the OS. So it's a bit of an issue. It's not impossible to get Python 2.7 running there but it's an effort that people have to make it's good to know that people are still stuck and hopefully that's declining. That will come back when we look at the Python versions. These are more exotic. OS is maybe some people are bigger fan of Ubuntu or Debian or even Suze and then the great people that's a whole different stack. Yeah. That's going to be an interesting chart. How do you... It's four dimensional. This only gives... This may be one box. It's hard to tell. I expect this has gone... If you would have asked this question last year hopefully this one may have been a bit bigger but hopefully it's going down quite fast. 20% CentOS6, 10% REL6 these people are kind of stuck to Python 2.6 not entirely through but it's certainly a factor. And then here the Python version so most people are using Python 2.7, no surprise. Some people are using Python 2.6 so that's CentOS6 and REL6 probably. Now 10% of the people say they're using EasyBuild on top of Python 3. I want to talk to these people because EasyBuild doesn't work on Python 3 yet. So this is one of the strange results in the survey. We saw this last year as well and the question is not clear enough or I'm not sure. It's just a funny result. The good news here is people using Python 2.6 is going down. That's certainly good. So it's becoming less of a problem. How much of a problem would it be if we drop support for Python 2.6? So this is also going down. Most people stopped caring or certainly nobody said disastrous so they're certainly away around it. Install Python 2.7 for yourself. That installation and you can certainly keep going. So we can certainly consider dropping Python 2.6 support if it gives us some kind of benefit. And I'll get back to that later. And then the elephant in the room maybe a little bit. Running EasyBuild on Python 3 is certainly becoming more important. Python 2 is end of life end of this year so it will not get any security updates or anything anymore. It hasn't been getting new features for years already. So we certainly start looking into that. More people are starting to worry about Python 3 even though it's still not a critical problem in my view because even the next REL version will still have Python 2. So apparently Redhead is not able or not willing to drop Python 2 in REL 8. So they will certainly be maintaining it themselves and probably backporting fixes if they can. So it's not going to disappear but it's becoming a problem. We've been aware of this for a long time. Which is why for the next major release of EasyBuild EasyBuild 4 and there's a tracker issue for this open the main goal is to make sure that EasyBuild can run on top of Python 3. We're done waiting or we're done postponing this so we'll try to be compatible with Python 3. We'll definitely keep Python 2.7 support so that that will be there for quite a while and maybe up until what I've seen now even keeping Python 2.6 support is not going to be much of a hassle. So it's certainly possible to have something that's compatible with Python 2.6, 2.7 and 3. Which is also about SPAC does. So even SPAC is already compatible with Python 3 but it's still compatible with Python 2.6 So it has the whole range so it's certainly doable and you don't even have to jump through a lot of hoops to make it work. The downside is you cannot use any of the fancy features of Python 3 because they're not supported in Python 2. That's the only downside but I don't see that a huge problem and it's certainly not the motivation to go Python 3 only. That would be too much of a push I feel. Another thing I would like to do is get rid of dependencies for the EasyBuild framework. So the EasyBuild framework should be or actually EasyBuild as a whole so 2 and 3 combined here should be a single tarball that you can easily install with pip or whatever and it shouldn't need any external things like VC base or VC install or even setup tools because having these as dependencies has been too much of a pain. People run into installation problems. They can't install EasyBuild with it and it has been a running joke in the community and I'm fed up with it. That's something I intend to fix for EasyBuild 4. It's already done I'll get to the next slide on that but for VC install and VC base it's very easy to fix it's already done actually. Setup tools is a bit more difficult because some features that EasyBuild has rely on having setup tools there to like the include EasyBlocks stuff now relies on having setup tools but there are other ways of doing that so hopefully we can figure those out and get rid of the setup tools requirement as well. Single tarball releases is just also to make the installation easier and 2 is like a prerequisite for 3 so that will help a bit. The different GitHub repositories will stay for now so there's certainly I feel there's a good reason to have them separate. We have 1600 pull requests for easy conflict files which is like 200 for framework if we're going to mix those it's going to be a huge mess I don't want to have that situation I think it makes sense to have separate repositories even though that will come up later in the survey people don't like this situation and I see why if I run into software that spread across repositories I also go what the fuck are you doing that EasyBuild is doing the same thing so I feel we have good reasons for that. There are minor things that I hope to get to for EasyBuild 4 is getting rid or at least deprecating the dummy toolchain and replacing it with the system toolchain so this is just a rename but there's some funky behavior in the dummy toolchain that we want to get rid of and the system toolchain will not do this anymore so it's a detail and system makes more sense than dummy anyway a custom EasyBlock for open API because this has come up with easy build config goals with recent versions of open API it's very hard to have a single configuration that works for most people so we need more logic to deal with that and the only way to implement logic is to have an easy block rather than only an easy config file so that's something and something I've been wanting to do is by default we use setup.py install for Python packages that's the default mechanism you can tell EasyBuild to use pip instead I actually want to switch that default to using pip as default and having support for setup.py install because pip is the recommended installation tool for Python packages but I've been running into a couple of issues with pip that are very frustrating so I'm starting to second-guess this I'm not entirely sure anymore whether it's a good idea I was looking into an issue with pip last year they have a fancy new build isolation feature which messes things up and disables it to make it behave so if they keep doing things like this this may actually be a bad move so I'm not sure about this one yet the current idea to have EasyBuild 4.0 available is somewhere this year but that's very early on and it really depends on whether we hit any roadblocks or any hard to solve problems with these requirements but the way it's looking now I've been making quite a bit of progress with Python 3 compatibility already it seems doable ask me again in May we'll see what the situation is with them so Python 3 support has already worked in progress since end of last year let's say so about a month I've been working on this and it's done in a separate branch a 4.x branch in the framework repository we'll keep working on EasyBuild 3 new releases 3.9 3.91 will probably happen in the coming months so we're not going to stall releases for this but in parallel working in a separate branch to start porting EasyBuild to Python 3 that's the first goal and we're keeping this in sync with develop of course anything that's implemented new or bug fixes in develop will merge back into the 4.x branch and I will take care of that I know the codebase well enough a little effort for me or small effort for me but it's been done already so one thing that kept us from adding Python 3 support for a long time is relying or is having VSE installed in VSE base as dependencies because these are not ported to Python 3 yet and partially because of the decision that Redhead made that they will still have Python 2 in rel 8 and anything based on it I have the feeling that we won't be spending enough time on porting VSE base to Python 3 or we're certainly going to it's going to take us one or maybe two years to actually port this and without having this ported we cannot even start looking at EasyBuild there's no point the option parser, the logging infrastructure a couple of other things are provided by VSE base so without that we can just even start porting EasyBuild itself so that was a blocker but with this, after trying to convince the guys in our team we should start working on Python 3 support we don't have time for it it's not a critical thing yet so it kept being postponed so I decided together with getting rid of dependencies which I mentioned here we've just copied whatever code we're using from VSE installing VSE base into the EasyBuild framework merge that as a pull request and now we have control over the code to Python 3 I dropped a whole bunch of code that is not relevant to EasyBuild so that helps with the porting effort, we don't have to look at that and VSE base will probably be ported separately just for our needs so we use VSE base as a base for lots of our scripting stuff in our infrastructure but again since it's not critical since Python 2 will still be around for quite a while I don't expect this to happen this year maybe not even next year, it depends on how much time and manpower we can use for this so this helps ingesting VSE base into the EasyBuild framework was step 0 another effort that was done we can do an import of all the EasyBuild framework modules in Python 3 so that's just fixing silly syntax errors like print statements and all these things are now fixed there's a large work that doesn't mean the code works but it's a first step you can configure EasyBuild in Python 3 now so you can run the setup configuration function that works in Python 3 and it generates the right configuration as a result and we have some of the test suites already passing in Python 3 as well so we're slowly working towards having EasyBuild compatible with Python 3 it's done in small steps to keep the pull requests small so others can easily review them and not have a pull request that touches 10,000 lines of code, we don't want that we want to keep them short and focused so people can look at it for half an hour and say ok, test pass changes look ok, we merge this and we move on to the next set of test modules in the coming months I hope to find enough time to keep working on this so the first step is definitely making all the test pass in the framework, suite by suite I think we have about 10 subsweets of tests so just work on them one by one but in a particular order because file2 is also like the first one because everything else uses file2 so it has a certain order there and hopefully keep the pull request rather small so they are easy to review by the maintainers once framework is done we should take a look at EasyBlocks now I don't think any big porting effort it will mostly be changing import statements because we ingested VSC base so we shouldn't import from VSC base anymore but from another location in the EasyBlocks framework that will be the biggest change and which is very trivial to make for easy config files I don't think we have to change anything so the syntax we are using in easy config files is Python syntax but it still works in Python 3 so that should be ok I think, I haven't tried it but I don't expect any big problems there. Once all of these three are done it's all just testing actually doing builds on top of Python 3 see if we hit any weird things and then when we are happy with that we can do an EasyBuild 4.0 release with support for Python 3 it maybe will market as experimental just to tell people like be careful if you do this in production because it hasn't been fully tested by lots of people yet I guess it depends on how smooth this process goes people who want to help out are definitely welcome to help out so we have a separate label in framework for these pull requests I hope Victor for example has already offered help to help out with the Python 3 support so if others are willing or can spend some time on this it's certainly welcome ok and maybe any questions on this it's a good time to help people who are happy with our yeah yeah for the screen I just have one question concerning the pip usage for EasyBuild 4.0 so setup actually compiles the code right so you get better performance from NumPy for instance like and for pip you won't have it so how is it going to do? Pip doesn't compile no I think it just follows the compile the wheel? No not the way EasyBuild doesn't EasyBuild downloads the source package and the pip install dot in the impact directory so it also compiles from we are not pulling in wheels we're staying away from that for exactly that reason more questions I was expecting another question haha alright a bit of a jump to the common tool chains or the community tool chains maybe we should rename them a bit so these were introduced several years ago so two let's say standard tool chains that we try to recommend people to use to focus the efforts of it so we can more easily reuse each other's work so we have an Intel based tool chain, Intel compilers Intel MPI, Intel MKL and then a free and open source software tool chain this is GCC, OpenMPI OpenBlast, ScalaPack and FFTW we kept the name generic here to allow exchanging one of the components for something else like at some point OpenBlast development was really slowing down or there were no releases anymore for like a year or the year and a half we were starting to consider switching to a different Blast layback level in the end we didn't need to because they picked up again in doing releases but having a name like this allows us to just change something under the covers hopefully you don't notice it helps a lot to focus the effort if people use these tool chains they can more easily use each other's easy config files that certainly helped the latest version this is a typo this should be A the latest version of the tool chain was included in EasyBuild 381 which was released yesterday so it's very new based on GCC82, the latest Benutils not the latest OpenMPI because that's a 400 which has some changes in API things like that so and people ran into trouble there so we're sticking with the latest OpenMPI 3 latest OpenBlast, latest FFTW and for Intel 2019 A latest Compiler, latest MKL for MPI we're using the latest 2018 version because there were issues with the latest 2019 version on bigger scale if you run big jobs then things just go wrong very badly like if you have jobs of, what was it thousands of cores? when did the issue pop up? even 500 cores things just fell over so about a thousand or even 500 cores you already see the problem and Intel support has reproduced the issue and we just decided we're going to stay away from this version for now until they fix the problem so maybe in 2019 B we can go back to using the latest version of these components so when we define these tool chains yeah sorry? yeah, yeah, this is the typo yeah, I know sorry you reviewed the slides you did so we also asked in the survey which tool chains people are using so full tool chains, not sub-tool chains and things like this so to try and make some sense of the results and it's pretty clear the common tool chains jump out as the most commonly used ones that's very good there's even a shift to using the latest common tool chains so 2018 B which is pretty much the latest that was available at the time of the survey is used by most people so 60% use the latest FOSS and like 45% use the latest Intel so that's very good some people still use an older version which is also okay till the common tool chain so they can still reuse each other's efforts and then a whole bunch of people, a smaller amount of people are using other ones so there's two versions of FOSS and Intel that just add CUDA into the mix as well so some people with GPU systems are obviously using these more people are using FOSS CUDA than Intel CUDA is there a good reason for that? okay I'm not sure if there's an issue there with the latest Intel version I'm not quite sure there is not supported by Avilia yeah so that may be why that may be why people are not easily combining Intel with CUDA some people use a mix of FOSS and Intel which is IOMKL this is Intel Compilers or Intel Compilers or Intel Compilers this is Intel Compilers OpenMPI and MkL or GI MkL so GCC, Intel MPI and MkL so we have like a mix of everything but that's certainly less than the more common tool chains yeah, close to 20% of the people replace Intel MPI with OpenMPI that may be related to problems that people have been seeing in the past with Intel MPI or maybe because of licensing so Intel MPI is not free for long time you get better performance with OpenMPI than Intel MPI sometimes, depends on the code yeah it's good to know what people are using and the common tool chain effort seems to be picked up quite well so that's good how frequently should they be updated so now we do twice a year once in January, once in June, July we try to, sometimes it shifted a bit because we know there's a big or an important release coming of one of the components so I think 2018 was delayed a bit because of a big important OpenMPI release that kept being stalled hopefully we won't make that mistake again or we won't have to do that again but usually try January, July and that seems to work well for most people some people say once a year is enough I'm not sure it is because we also lock dependency versions per tool chain so if we're sticking to Python, let's say 366 that's the Python we will use for that tool chain and if we do it once a year we're stuck to that version for a whole year that may be too long some people don't care some people say more frequently then if they start helping out then maybe how many software installations are people doing some people use it for just a handful maybe they're just getting started with EasyBuild more than half more than half of people use it for more than 100 installations per year we are, our team is certainly in the thousand mark here also because we use EasyBuild for pretty much every install we do which is this question here and this is a bit small, I apologize but so the green is basically good here were people saying yes we use EasyBuild for everything this has some exceptions the yellow says it's the main way so the vast majority uses EasyBuild for most of their installations and then here the people still install manually the green is mostly no and then the the blue is yes sometimes so maybe people are very familiar with the particular software package or they want to be in full control over how it's installed and they don't like how EasyBuild is doing it or they don't know how EasyBuild is doing it so they want to be in control I guess that makes sense this is very similar to what we saw last year so there's no shift in one way or the other easy config files that people use 85% uses the ones we include in EasyBuild so again the efforts we make are certainly appreciated there many people use there or create their own easy config files probably mostly for simple updates about half of the people have their own repository where they keep things nicely organized hopefully I hope a lot of these are also coming back and through the central repository so they benefit everyone and there's actually things we can do there in EasyBuild to make it easier to pick up easy config files from somewhere else that's probably also a feature we should use custom EasyBlocks half of the people have at least some customized EasyBlocks some people have over 5 which I consider a lot so hopefully there also they're being contributed back I guess you're one of the people that has more than 5 more than 15 we're actively trying to get those changes back so I hope that the people have these customized EasyBlocks try to make the effort of putting them back into mainstream side specific customization so this is a bit across the board 20% or 18% have no customizations at all just use EasyBuild like the way it is the biggest part uses custom easy configs probably small updates or things that are easy to do and then a bit less than half actually customize code either in EasyBlocks and framework so again probably people have good reasons they're trying to fix problems they are seeing in their EasyBuild installation first and then hopefully also contributed back but it's hard to get a good view on that I guess people are doing that consistently and it's certainly a time issue it's mostly a matter of time whether to contribute things back or not it's not a matter of not wanting to be being able to then looking at the mailing list still growing over 250 people on the mailing list which is quite good the line keeps going up which still scares me a bit but ok traffic has actually gone down last year compared to 2017 I don't think that's a bad sign I think it's because of Slack a lot of people have jumped into the Slack channel and just ask questions there specifically than on the mailing list because it's easier to just quickly give an answer then type out the whole mail and try to be friendly in the email Slack is a bit more informal I guess that's where the effect comes from and it's probably a good thing as long as people get answers to their questions this is an overview of the Slack channel Slack is a commercial tool so it doesn't allow you to see the growth of people joining the channel unless you pay for it just to get a graph doesn't make sense but they did give me this graph so very recently we passed 50,000 messages sent on the Slack channel and it has only been here for about a year and a half and it's like steadily growing in the amount of people that are active so the green line is weekly active the blue line is weekly active and posting messages so green is only reading blue is also actively posting so that's good so this is certainly a good place to ask questions and get help the nice thing is if you're really not a big fan of Slack that's fine you can join the IRC channel and we have a bot that sends any message on Slack to IRC and the other way around you don't want to use the commercial Slack tool it's perfectly fine and you can still talk to people who are on Slack you just have a bot that in between sends things back and forth and the bot has been very stable so it's running on a VM somewhere that Pablo manages and he has to look at it like once a year so it's very good and if people want to join the Slack channel you need an invitation but there's this app running here that you can enter your email address and it will send you an invitation so you can join it's a bit backwards but if Slack doesn't want to have open an open channel that anyone can join then you just work around it like this now what's important here and we saw the same thing last year in the survey is let's say this number is wrong I know this is a green one so some people are on the mailing list but not reading the posts just filtering it out and then definitely I guess almost 40% of only reading posts are never typing a mail themselves I don't think any questions most people only participate occasionally and some people are not joining or are not interested and in Slack it's even worse so like 60% of the people are not in the Slack channel which is fine, you don't have to be if you don't want to another like 20% barely pays attention to it and only about 25% actually at least reads the post or joins the discussions that's important to know because it means if we're discussing something on Slack channel it's a very small group that makes decisions so we should try not to do that at least push it out to the mailing list where we reach more people but definitely have some official record of it like open an issue in one of the repositories and try to have a relevant discussion there and please document why we made some decision and this is not even covering the conf calls so the bi-weekly conf calls on a good day 10 maybe 12 people joining so that's even a smaller group of people at least the conf calls are very well documented so I'm taking detailed notes on them so that's good but it's good to take this into account if you ask a question on Slack we should never expect to get a relevant answer from the community that's just good to know an easy build documentation has been well, slowly growing we get 400-500 weekly visitors that's unique visitors not visits but visitors that's quite good now what they are doing who knows that just Google telling us somebody was hitting this page already actually reading stuff maybe they're looking for something else that's called easy build this is very hard to view but it will be clear on the slides this is a map of the world that shows where the visits are coming from this is Europe obviously quite a lot in North America here in the middle nobody lives or I don't know what happens there but Central America doesn't seem to have a lot of data centers or everybody uses Slack it's back here, sorry I don't know, Australia but in Asia so it's the sun never sets on easy builds and that's quite cool and this is still the best way of figuring out how many people are using easy build you just look at how many people hit the documentation assume that they are using easy build and trying to figure out something on how to do a certain thing I don't know of any better way of counting users than this questions in the survey related to the documentation, how complete is it so quite a lot of people seem happy fairly complete or like three quarters of the people most of the others are saying okay I'm probably in the one percent that considers it incomplete so I know a lot of gaps in the documentation that we should really cover and this is coming up at the end of the survey as well so the problem is documentation is very unsexy to work on, rather do codes and write text that explains how the code works it's all known but hopefully we can change this if anyone really likes writing documentation let me know, I'll tell you what to put in the documentation how often do people consult it at least three quarters consult it once a month at least people are using the information that is there, that's good and how useful is it the vast majority says at least somewhat useful so it's certainly useful to fill the gaps that we have in the documentation even though people are mostly satisfied with what we have already favorite easy build feature the answers here are a bit different from last year because we gave different options most people seem to like what we call dependency resolution most people seem to like disagree with me a lot on that it's certainly understandable the robot just picks up what it's missing installs it and you can get a whole stack like this what SPAC has is certainly a lot more advanced than what easy build has there try tool chain and try software version it's basically generating a slightly different easy conflict files so you don't have to do it manually quite a lot of people think this is the favorite feature in easy build the github integration I'm in this box here this is what allows you to easily contribute stuff to easy build and I will discuss this this afternoon how that works extended dry run if you use eb-x it will show you what easy build it's planning to do without actually doing it so you get like a report all the commands it will run in a couple of seconds and it won't change anything on the file system so it makes a couple of guesses here and there and it tries to give you a report it's not going to be 100% accurate but it's going to be very close to what it actually does that's quite useful and dash dash trace I like this one a lot as well just gives you more information about what's going on rather than just saying building it will say I'm running this command it started at this time it's running in this directory and you can see the output it's generating in this temporary log file so you can tail it and just see what is going on rather than just saying building I'm going for a coffee and I have no idea what's going on so it gives a bit more a better idea of what easy build is doing thanks for inviting we do? perfect I knew it was a good idea to mention that excellent so which parts of easy build do you not like so people could get multiple answers here a bit of a naughty question maybe but it's good to know and this is no surprise the fixed versions we have for dependencies that's the biggest thing people don't like which is exactly the opposite of what SPAC does I see Massimiliano smiling he will explain on Friday what SPAC does here it's very different from what easy build does and we should try to loosen that up a bit what we currently do I still think the risk is not a good way of it's not a good word for it but it will open up surprises once we start making this more flexible if you say I don't care which Cmake version is used just use whatever is there for some people it's going to work for others it's not so it's a double-edged sword so people have no features that they don't like that's very good lack of manpower I'm in this box so we get a lot of contributions it's hard to keep up if we had some or more dedicated manpower we could certainly do a lot better than what we're doing now even though we're doing okay I think which is one of the reasons that we're looking into European funding to score some funding to have some dedicated manpower for easy build which is not there yet and then a whole bunch of other things a long tail I guess the other includes things like lacking documentation has at least I certainly agree with that no support for uninstalling packages this pops up a lot I think there's even a pull request open for it that sort of works but it's not complete I think we well looked into this and yeah we should just get it in there with a big fat warning around it if you're doing an uninstallation you may be breaking something that easy build doesn't see currently so there may be things floating around that rely on the module that you're removing but if they're not in the view of easy build or not in the easy build prefix then you may be breaking installations in other places and it's just impossible for easy build to know as long as you print a big fat warning and people agree that it's up to them to break other stuff like if you have central installations and you remove it people may be building their own modules on top of the central installations and you will break their stuff and it's pretty much impossible to know that up front so be very careful to do that I guess it's just a matter of warning lack of support for Python 3 came up as well but we're actively working on that already so that's good and other tools and projects that people use in combination with easy builds Lmod, no surprise sorry XFIA but Lmod is certainly the biggest or the most commonly used modules tools modules tools, the latest version or the old 6 version I think Tmod is here that's your 4.x version at least some people are using it so that's very good but more people are apparently stuck to the old one I'm not sure how correct these answers are because it was clear in the list of answers what is what but how certain people are which version they are using I'm not sure I'm not sure if everybody ran module.sh version before giving the answer maybe I'm just guessing to get rid of the survey I don't know so don't rely on this too much but it gives the main conclusion with respect to modules tools is that Lmod is more popular okay yeah that could be an option to add a question of which modules tools do you use and why or why yeah okay I guess if they are really using 3.2.10 it's probably because it works and they don't want to touch it I mean I know and you will talk about this with your features so does Lmod compared to this really really old version right now but if people don't know about the features or they don't think they need them then it works and they don't touch it that's probably a big part of it lots of people use slurm I'm not fully sure whether they actually use slurm in combination with easy build so if they use the distributed installation feature so if you use dash dash job you can tell easy build just spread the installation for each dependency as jobs to the cluster and any installation that can be done together at the same time will be done as long as the job starts and easy build make sure that things are done in the right order so that that's very useful like if you're populating a new cluster buy a new cluster you have to install the same thousand modules that you have on the old cluster you can basically do this overnight with easy build just send it to the cluster as jobs and next morning hopefully most of them are done how I do the regression testing of easy build we have in the latest test because I skipped all the stool chains that are deprecated there were close to 8000 jobs being submitted to the cluster and I was lucky over the weekend there were like 10 or 20 nodes free and just ran through the whole thing over the weekend so that's very useful and certainly a lot better than having a single box doing all these builds just one thing about the the minus minus job apart from the GCG PI which is all configurable the other methods are not really configurable so the slurm support at the moment doesn't take any options there's no way of putting some options there for our system that's essential there's no way you can run something without doing that it gives you a couple of options like you yeah there's some that you can't put there's some things that don't go into environment variables and also there's no documentation that you can do that some people might not know it's not mentioned in the documentation so you would just want an option to say just pass this to SBatch yeah just take this take this option and put it in it's very simple to add it's very easy the slurm support now is pretty basic lots of people are at least using Singularity maybe in combination with EasyBuild so that's not fully clear here normally Eduardo already made it here it's a little jet lag so that's quite good and it's growing as well compared to what people were mentioning last year I don't like this one 16% also uses KONDA I hope you really know what you're doing if you're using KONDA first of all it may give you installations that are not optimal for the hardware you're running it on that's one part and once you're using KONDA you basically have to do the whole stack in KONDA things get hairy so hopefully people know what they are doing when also using KONDA or using KONDA for certain things I'm not saying you really shouldn't but at least be aware of the issues there commercial support we asked this question last time as well just to see if there's a market for commercial support we're not really considering adding it right now and 4% is definitely until they have to pay for it maybe 20% or 21% would at least consider it and then the others say no I don't know if there's a market for it there are some consultants and there's certainly one guy in the room where's the glue room person there's some consultants that do help out companies with using EasyBuild for their installations so I guess that's some form of commercial support I know Pablo has done work like this one of the maintainers has done work like this Fortis has done work like this so you can certainly get help just not from the not from HPC again using a custom module naming scheme so maybe this is not well known enough and part of the reason is this is not covered in the documentation which has been annoying me for a while but not enough that I actually fixed it you can get full control over how EasyBuild names the modules it generates so by default it will do software name slash version dash toolchain dash version suffix if there is one but it's very easy to change you write a small Python script that takes all the information name, version, version suffix, toolchain even other stuff and you can just puzzle together your own naming scheme the way you want it EasyBuild comes with a couple of naming schemes out of the box which you can check and you can specify which one you want to use or you can write your own that is derived from one of the existing ones or an entirely new one if you want to obfuscate everything or I don't know what you want to do, you can and you can easily tell EasyBuild about this new scheme you're writing and there's a this detailed option that may be relevant to you as well now this also was covered in the survey the vast majority of people is using just the defaults maybe because they don't know there's actually an option of taking control of what EasyBuild does maybe they are just happy now the default scheme does generate quite long module names sometimes so which can get confusing for users and it's a flat scheme so everything is together you module avail you see everything all the time certainly for users it can be misleading this is a small customized version of basically this maybe lower case or something like that and then there's a minority of people about 18% that use a hierarchical naming scheme where things are organized in a different way and I guess some people are not familiar with how that works so briefly explain that here this is one of the let's say big early features that we implemented in EasyBuild is support for having hierarchical scheme because it's not trivial to do this by hand to know how this works and to convince EasyBuild to do it in the correct way was not easy we had to work with Robert McLean from Elmott to really get this right Elmott was built especially for hierarchical module naming schemes and there was a lot of back and forth figuring out the details like what do you do in this situation in this combination of things it was not that easy to do so in a flat naming scheme the default does everything is available all the time so if you can ask me looking module names if you do module avail you see all the blue ones then you just pick whatever you have to load to do module load and then that one becomes green in a hierarchical naming scheme things are organized differently so initially when you do so the modules are organized in a tree sort of when you do module avail initially you only see a very limited handful of modules typically only compiler modules you have multiple levels in the hierarchy which is flexible you can have as many levels if you want to but usually the typical ones are the core modules so the ones you see at the start the modules that are dependent on the compiler and then the lower layer is the modules that are dependent on an MPI library and through Elmott even though module avail only shows these it has a separate command module spider where you can just look through the whole tree if you have to how does this work you pick one of the compiler modules you load it so it becomes green here and if you then run module avail you see new modules popping up so these two open MPI versions were built with this version of GCC then you can load one of those and you get modules that depend on this GCC and this open MPI now the nice thing is here you get shorter module names because you don't have to have this toolchain crap in there anymore it's like implicit in the way you reach the module and everything that's not compatible with the compiler and the open MPI you've loaded is not available you cannot load them straight away you can see them through module spider but not in module avail so people get a cleaner view and they only see things that are compatible with each other and it has fancy features like if you would do a module load of this one which you can load at this time Elmott will actually unload all of whatever you have loaded and then load them in the right tree again so it will swap things in an intelligent way which is not trivial to implement at all and you can just tell Easyable to generate the modules in this way you just change the module naming scheme and Easyable knows what to do it knows what is a compiler what is dependent on a compiler where it goes in the hierarchy it knows to use the short module names and there's lots of magic underneath that you don't need to care about one downside here you have to tell your users what a compiler is and what an MPI library is it sounds silly but for bioinformaticians it's like why do I have to pick a GCC thing I don't know what GCC is and why do I have to care give me access to my Blast or whatever software so sometimes to me it feels a bit backwards you would actually have to turn it upside down see the software first they pick a software then you somehow magically also load what is needed maybe that's something that can be implemented as well Robert has told me it's not possible to turn it upside down I'm not convinced it is but I haven't tried it myself I know what he said so rather than pass the mic over and say you can always have a default set loaded so you can always have a default view that does include a lot of software and somebody else asked the question here as well can the hierarchy scheme and a default be used at the same time I can answer this it's currently not really supported but it will be fairly trivial to do so the way the software is installed it's installed in a generic way basically using the the easily naming scheme for the software installation it's important to use this option though the fixed install their naming scheme when you're installing multiple things you have to use this one and this by using the fixed install then having different trees for different formats is completely possible it's possible if you install with naming scheme first together with this option enabled and then you do another one with module only that's possible the detail there is module only is not perfect so we can actually do a better job and just let easy build here take multiple schemes at once and then it will always get in terms of something we could develop in the future that could be something that we could handle it also popped up in one of the suggested features if you do something like this the robot will get more tricky maybe because you may be in a situation where things are available in one naming scheme and not in another things could get hairy which may be why this hasn't been implemented yet but you can make easy build opt out and just check it's available for everything first if not the combination robot may be tricky that's true depends on how you do it but as a first pass you could say I'm gonna assume that if it's available in one naming scheme it's also in the other which is in general as long as you use this consistently it should be the case and then you can see how bad it is in practice you can do things step by step things can get hairy as soon as you yeah I've been caught a lot saying that this shouldn't be hard to implement and then when you start looking it with like oh okay it wouldn't be the first time but yeah let's say it's on our radar to look into this the video also mentioned to me that people at supercomputing were asking about this specific thing like being able to install the different naming schemes in one go and not afterwards because it's sort of manual work and you have to keep an eye on things okay yeah so I hope this is a bit clear so EasyBuild supports this quite easily at least in this form core compiler MPI and if you want to you can try to turn it upside down or have an extra level maybe for Python so pick a Python version and only then you see the Python packages actually we have a better way of doing that another aspect of the modules that are installed is how long are they available to the users most people said forever which means as long as the system is there the installations will be there and people can load the modules this is also what we are doing now but I'm not happy with this on our older system I would have to count but we probably have like 3000 modules installed so people do avail and start scrolling and then they are shared I like this option better so you make them available for a limited or actually the green one you could make them available for a limited time and then hard remove them or you could make them only available indirectly so if people know that they are there they can still use them but you like stop recommending or stop making it visible to people so you can hide the module or what they do in Ulich is they have like these build sets to actively swap to an older build set to still be able to load these modules so there are things you can do there and we should probably also document different approaches you can do here and I just have to find time to start hiding one very easy thing you can do with Elmot is hide things with old tool chains hide modules that were installed with old tool chains that would be quite easy to do just have to find time to look into it okay and take a look at contributors so both in the survey and in the well I collected some statistics on contributors as well and this guy is very enthusiastic I like this so one of the questions in the survey was how do you actively contribute to easy build multiple answers were possible over half of the people say they open issues so they ask for features or they report bugs it's one way of contributing it's a very easy way of course just ask for stuff and let other people fix it but it's certainly better than not reporting bugs like if you're not telling us that something is wrong we don't even know that we should fix it we certainly appreciate people doing this as well 20% is active on the mailing list less so is active on Slack these are pull requests so about 40% of the people open pull requests for easy conflict files that's quite good that's close to half of the community or half of the people that answered the survey or at least willing to contribute back through pull request and there's a good reason why this is a lot bigger than the other ones I'll get back to that this afternoon also easy blocks and framework at least there's a will now contributing back here is hard you need to know a bit of Python maybe even need to know a bit of how things fit together in EasyBuild so you need to work yourself into EasyBuild first and you need to know Git you need to know GitHub a bit you need to know how we test things so it's certainly harder here to contribute but at least we're getting contributions one thing I didn't like here but I don't know if it really is if you should make any big conclusions from it is that we're seeing a decline in the amount of people that answered these questions so are people less willing to contribute or do we just have people that are happy with how EasyBuild works and they feel they don't need to contribute I'm not sure now I'm certainly not complaining we get a lot of contributions so sorry how many survey answers did we get last year we got 77 this year we got 93 that may be related to this yeah so again it's very hard to draw major conclusions from this different things could be happening are people actually less contributing or do we get more people answering the survey that are not contributing how to tell making observations so looking at actual contributions we get this is number of pull requests I should have added an access here number of pull requests basically since we've released EasyBuild 1.0 this is the number of pull requests per year the blue bar is me I've been mostly most active in framework since it was thrown into my lap the red bar is other people in HPC again team so they have disappeared from framework mostly due to lack of time also due to one of the maintainers moving team so he moved to the VUB so he's not in the red bar anymore in the yellow bar in the green bar actually and you can see it going down in the last two years that's because of lack of time with me so it's mostly the blue one going down not the others others are actually increasing so that's good so there's less pull requests less features less activity in framework maybe we're also getting to a point where most of what we need is already there and it's just details it may be that as well blocks is a little bit different so you get roughly the same amount of PRs per year but it's less relying on me so we have more maintainers and other people outside of maintainers contributing back there which is very good it's a fairly low amount of PRs here because most of this is updates to existing easy blocks something changed for a new version of software you need a different configure option or they moved from configure to CMake or things like this the generic easy blocks are actually rarely touched they seem to be working quite well and you get all the flexibility you need so it doesn't need a lot of extra logic that's good and now it gets scary this is easy complex the last three years we've been getting over 1500 pull requests per year that's quite a lot which is also why since summer 2017 I think I have it on another slide we have 10 people that help out with getting these things merged because it wasn't possible to keep doing it by myself and you can see the amount of people not in the group of maintainers is increasing which is good but it means more work for the maintainers I will explain part of the reason why this explosion happened this afternoon as well now it has been stabilizing so around 1600-1500 PR per year that's more than enough it's a bit of a not a problem but it's a struggle to keep up with things that come in so I guess we should be happy about that not complaining and close to half of contributions are by people that are not an easy-built maintainers and that's quite good and this looks at how many contributors we have so how many different people have been contributing over the years for Easy Conflict we've reached the 200 mark recently for Easy Blocks we've reached the 100 mark recently so over 200 different people have contributed to Easy Conflicts over the years and that's quite good and if you look at it per year so framework is green Easy Blocks is yellow blue so the number of different people contributing back Easy Conflicts has been steadily increasing and it looks like this year if it keeps going up we'll have over 100 different people making pull requests for Easy Conflicts into the central repository so that's quite good it also means you have to talk to a lot of different people expect them to respond to your feedback on their pull requests so it's about a 10 ratio of maintainers to contributors for Easy Conflicts which is challenging now we also have maintainers thankfully so different group of people dancing along so since the summer of 2017 I started actively asking help to the most active people in the community like are you willing to spend some of your time looking into contributions because it's impossible for me to keep up I was noticing that other real work sort of was becoming more and more demanding and it was hard for me to keep up by myself so now we have 10 maintainers then very active people in the community who are willing to spend some of their time looking into contributions and since May last year we have a maintainer of the week so one of the nine maintainers so it doesn't include me says okay this week I'm gonna try and spend a bit more time looking into contributions try to do one hour per day looking at what's coming in answering questions testing pull requests reviewing pull requests and so on just to make it clear that there's always somebody who's actively looking into what's coming in it's sort of a goal so that we don't have any silent periods or things don't pile up if people are a bit busy and we just plan this maintainer of the week actually have to still do it for next week or next month I think people in green are here so Alan Damien myself and okay are attending the meeting here people in red will be attending false them as well so you can run into them there so definitely maybe give them a thank you for spending some of their time helping out with keeping easy build active and maintained then this is one of the last questions overall quality of easy build excellent grade or okay so 100% at least said okay there were two other answers could be better or pretty bad but nobody knows so that's quite good quite happy with this and then there was another open question with suggestions for features the one that was mentioned several times and came up already in the talk is support for uninstalling packages so there's one big feature we should add for 3.9 maybe it could or should be this and then with some kind of check on what easy build sees like is this what you're removing being used somewhere else if not I can remove it or at least I can warn you like this maybe somewhere use that I'm not seeing so be careful so you probably want to have some interactive thing that makes you hit yes like yamdas are you really sure and yeah if they then answer yes it's up to them or we could refuse if we see it being used somewhere else we could say I'm not going to remove this it's used as a dependency for something else so either use dash force or remove dash force but that's only if it can't see them it's not I'm not a big fan of that in general but we could also use the reproducibility thing so we could keep a place for uninstalled packages like we can pull the re-produ re-prod folder out so that if they needed to put it back in place that would be straightforward to do a trash bin of at least the easy config and the easy block that was used so at least it's recoverables in some way that sounds like a good idea there's an issue or PR open somewhere find it and put it in there better support for filtering and hiding dependencies so somebody mentioned to me that some people here want to talk to me about this so please do I want to hear more about what the problem here is maybe we're not aware of some issues or some somebody said containers containers containers I guess this is the singularity guy not sure somebody was asking for support for skipping the sanity check that's already supported since easyboot 3.7 so in the skip steps easy config parameter you can now mention sanity and we'll skip the sanity check we kind of refuse to do this for a long time because we know the sanity check is there for a very good reason but now we give you the option to skip it if you want to it's just something we will never accept in the central world positively we will not accept easy configs that skip the sanity check it has to be there but yeah it's something you can easily add once the installation is done you can check on it manually and say okay these things should be there put them in a sanity check and then just do an eb module only to regenerate the module which does the sanity check as well several or a couple of people mentioned there's a steep learning curve to pick up on easy build this probably comes back to missing documentation I think we have missing documentation on common workflows like how do you get started there is a page there but probably needs a bit of updating a bit of work common workflows best practices so these two I think are linked together just a comment I mean I think it goes further than just documentation actually all the bells and whistles that are there for you there's an awful lot of them right so having somebody guide you who's more familiar with all of them because you're never going to learn them all from just they're not going to read the documentation from front to back right and so they're not going to be aware of a lot of things and the typical workflow that we see with people is that they try things and then they go oh I should have done it like this from the beginning I mean I think it's hard to get around that without doing like you know like having like a stop in support or something like that like where people can actually like on my system how would you configure it or why would you make a certain choice why do you make certain choices in your configuration what's the impact documenting best practices and getting started which is something we don't have right now but it's probably there's more to it than that there's two and four right people want things their own way too right they don't want it they don't necessarily want it the way I have it or I can hazard or something like that other things that were mentioned this is something that's relevant for the common tool chains maybe okay mention this one have a list of version locked dependencies for this generation of easy concrete files or you've certainly brought it up before maybe you didn't answer it in a certain way so if we're sticking to Python 366 for the 2018 B tool chains then that should be documented or listed somewhere if you need to pick a Python version you know what to do and we could probably automate this for at least partially loosen up the restriction on dependency versions at least one person mentioned this I'm suspicious that Masimiliano was answering the survey as well no then it was taught not you I'm afraid from that another one that was mentioned semi-auto creation of easy conflicts based on existing ones so that's kind of what try tool chains version does there's an open pull request for eb-dash new which doesn't do this yet but it could and it's different from what try tool chain does this is one I hope to finish for 3.9 and yeah one thing that's definitely missing now the implementation is a bit hairy but it's definitely missing documentation so this is not one we will merge without having proper documentation for it because nobody will know how to use it and there's no point in adding it documentation first and then merging the br installing multiple packages in a single directory more easily that's kind of supported already you can define a bundle of things easily install the bundle that's how we installed x11 for example we already have good support for it installing bundles of Python packages which all go into the same directory maybe we have eb install this and this and then an option like back at all in the same install directory and give me a single module for it maybe that's a the problem with bundles if you start it may have changed in recent versions but in all the versions we checked all the parameters and so with an easy block define new parameters which you used or tried to use in your bundle you got an error message that those parameters were not recognized so in those fields for a bundle it only seems to recognize a number of standard parameters I think that we fixed it so I should try it again you mean like configure options for one specific Python package yes or things like built-in CMake I think was even refused but now that's picked up anything you can put in a normal easy config file like configure options or build and install all these things you can also do for extensions so also easy block specific ones yeah that's all recognized I'm not sure which version it was fixed in but the 3.6 or 3.7 you can find it in the release notes but it's something that should be dealt with now so if it's not yeah it hasn't maybe in the last six months or eight months it's something we dealt with yeah better error reporting and readability and this is has been a pain for a long time I've done some work here related to the robots the robot should be spitting out better error messages now in the latest versions but the build log themselves are still big and confusing if you don't know what you're looking for maybe we need some logic and easy build to help you to quickly zoom into the actual error like known patterns for errors and extract them or I don't know what but I think this is not an easy one to fix I think SPAC has some logic for this for zooming in on actual errors for installations SPAC so one of the things that popped up in the survey was we should have better error reporting or a better way of when a build fails to actually zoom into what the actual problem is like where is the compilation error or so what we did with SPAC was basically to steal the already awesome way that CMake has to scrape on build blogs and whatever CMake has awesome features that can be unexpected but yeah and basically we use the same regular expressions to scrape blogs and we provide comments to I don't know look after build blogs and then just like show to the user what is exactly the errors to F and we could also search for warnings even after the fact we store I don't know to build outwards or the config log yeah we do the same thing we store it but we have no facilities to then we have facilities to spray them if you don't I'll have to look into a CMake feature crap no that's good feedback take the mic first that might be also something useful for the minus minus job because I actually have no idea where the logs are written in that case because I have no idea even on which machines the logs are written so that's also where we can improve if the build succeeds the same way as the normal the log when the build doesn't succeed I don't know but it does it probably dumps them wherever you run the job command it dumps them to where you are being you point to the the problem is the temp you have to pass it by a global temp is that covered in documentation maybe it's covered in the temp so it should be a section there that says how do I get logs for failing and then support for using multiple naming schemes which has already come up which seems easy at first but after what Damien said I'm now scared about implementing it we'll see so all these things are on our radar if anyone is interested in working on any of these it will certainly help it will certainly help to let us know so we can kickstart you with it then a little bit of future work I'm almost done talking here I haven't kept an eye on time at all but okay we're doing quite well one thing that's now a bit of an ongoing discussion for the people who have been following the mailing list and the conf calls in the recent weeks is we're looking into some ways to change how we deal with Python and Python packages and that's even separate from running easy build on top of Python 3 which is a whole different issue there's two things we're looking into so right now Python itself is being installed with a full toolchain like FOSS or Intel which is a bit stupid and weird because you have two Python installations which are basically the same thing so the interpreter whether it's built with Intel compilers with GCC has an impact but it's quite small and only a very specific cases it actually matters in terms of performance so the idea would be to like pull down the Python installation to the GCC course sub-toolchain and have one Python installation that you can use with multiple different full toolchains how the best way of doing that there's a bunch of options there and Mikael who has been working and thinking about this has opened a very good issue with the different alternatives that we're considering about the pros and cons of each of them you can find it somewhere in the issue I should have linked it here but it shouldn't be too hard to find and then related to that so we pulled on Python to GCC core but for some Python packages that we install with Python side-py, side-py, pandas who requires numpy and so on so some of these need a blast or a lay-pack or even an MPI like MPI4py so these should be separated and be installed with a full toolchain because GCC core does not provide this so basically you have to split the Python interpreter with some basic packages like pip and setup tools and then have a separate bundle with Python packages that need a blast lay-pack so the question is a bit how do we do this where do we use the Python name, do we use it for the interpreter or for the bundle if not how do we name the bundle and how do are there ways to make it transparent to users so you don't actually notice that something changed or you think that's something we can do we can have like a wrapper that gives the same name as before but then underneath actually loads something else so there's a whole bunch of options there we're looking into this now and this will certainly be in the coming month calls and coming weeks we will look into changing this maybe even already for 2019 A so the Python installations and the 2019 A toolchains it depends a bit on how quickly we can make progress on it so it has to happen in the next couple of weeks if we want to do it for 2019 A but certainly for 2019 D we will try to do this and we'll try to do it in a way that doesn't impact people that are happy with how things are now we sort of have like the two options and you can pick whether you want to do a hard switch or make it a transparent switch for your users so we're not fully sure about this yet but keep an eye on the mailing list, keep an eye on the notes for the conf calls if you don't follow the conf calls and we'll try to make this a very transparent process and be very vocal about it on the mailing list ask people for feedback, tell people what the kind of impact was probably also have a dedicated page in the documentation for it like what changed and how can you make sure it doesn't impact your users and so on doing this is going to make the maintenance of Python packages a lot easier because having one Python module also means that for a lot of Python packages you'll also have one Python module rather than now two like H5Py for example you have now two separate modules one for FOS, one for Intel which probably doesn't make sense one that you can just combine with other full tool change so it will limit the number of easy config files and modules we have for Python packages hopefully but it's something we have to look into and actually see what kind of impact it will have and then separate from this so both of these can be done orthogonally we can do one and not the other or the other way around or both or none we can have support in EasyBuild for installing Python packages a particular Python package in a single install directory but for multiple different Python versions at a time so we could install let's say NumPy or let's say Mappletlip which is already a separate easy config we have one Mappletlip 2.x installation that is compatible with both Python 3 and Python 2 so we have one module we do module with Mappletlip and depending on which Python you load it will pick up the right part of the install directory this is exactly what they're doing at ComputeCanada and this will be Bart will explain how they do this in his remote talk tomorrow afternoon so this again this means you'll have less easy config files for Mappletlip you'll have one rather than two for two different Python versions the module names will be shorter because you don't have to mention the Python version anymore in the module name you just load Mappletlip load whatever Python and it works together so there's a couple of benefits from this as well it simplifies things quite a lot and they've been quite happy with it at ComputeCanada now they're doing more stuff in ComputeCanada than only this they also built their own wheels and let users install those wheels in their own directories I think so there's more going on there but at least the part where they install a single package for different Python versions in one installation directory this could be integrated into EasyBuild quite well and the things that have to change are actually quite minimal we think but Bart is working on this himself so and then one other thing that certainly I would like to look into and I've heard that other people have been looking into it or planning to look into it as well as to test contributions more in isolation so right now a lot of the testing is being done just on our cluster we pull in the easy config file so reviewing it we build it and if it installs there it's called test report and we say tada it works now in our system we have things like ZLIP and we even have an old boost version installed on our OS which sometimes make builds pass even though they may not pass on other systems so that's something it's an issue in the testing setup we have it's mainly relevant for easy config but for other stuff as well now the problem here is it's not just enough to test contributions in a minimal environment so something that's entirely stripped like no ZLIP no boost no maybe not even a G plus plus anymore in the OS because you load a module anyway that provides GCC so we certainly need to test in this environment but we've recently discovered you also need to test on a system that has as much as possible like boost ZLIP maybe Python 2 and Python 3 installed so that's as packed as possible so some problems only pop up when you do have an old boost on the system apparently CMake has by default will look onto the system first for boost if it finds it there it will use that even if you include it as a dependency in the easy config file so that means if you don't have boost on the system you will never see that problem and the test may pass in the middle of an environment and then if you try it on your system where you do have an old boost it may fail so you have to do the testing like both environments and maybe even things in between that have only boost and or only ZLIP so it explodes but we can certainly do a better job there and I think Singularity can help we can have a repository with recipes for Singularity images that we share at least with the maintainers the maintainers build those images and use them on their test system and it also helps we can test for different Linux distributions even though we are using CentOS 7 through a Linux but through a Singularity image we can test for Debian or Ubuntu or ZLAS if we want to it kind of explodes the amount of environments we may have to test things in but we can pick I guess the test environments depending on how important the easy config files are like for common tool chains we will test maybe a bit more to have more or to have a better idea that things work in different environments as well while for other PRs we may test testing a bit less it depends that's something I would hope to find time for I'm doing some of that already I have a Singularity image for CentOS 6 that I used to reproduce problems that people report for CentOS 6 that has been helpful because it helps me to reproduce the problem and then figure out a way to fix it properly but we should do that more and in a more organized way hopefully and then I guess we're almost done so next year will be the fifth easy build user meeting well next year maybe this year leaving the option open to collocate it with supercomputing in November that may be a good idea to fifth edition do it in the US for once rather than all the ones we've had in Europe so if people are interested and if Davide sees this I think he will go haha so Davide is in the US and he has been quite active in the community that's maybe an option to do it or yeah either at supercomputing so that we make it this year in November or early next year we'll see so if somebody is interested let me know definitely planning to do this again somewhere that's all I have for now and I think I'm pretty much on time alright any more questions I'm sure lots of people have questions in their head but yeah is there any updates on the plans of using Yammerl easy complex I was expecting that I didn't include anything in the talk here because nobody has been working on it for the last year no you did work on it did I see it? yeah at least there has been very little things so for the people that are not familiar one of the things we've been trying to do or been working on a little bit is that the easy comfy files we have now are in Python syntax and are basically evaluated as Python syntax just key value definitions we execute it in Python and we get a dictionary of things that we give to easy build and it goes like that not very nice way of doing it because we're running a Python script what to do another option is to use a proper configuration syntax like Yammerl or JSON or I don't know what there has been some work on that to look into transitioning to this new format and how that would work and together with that also rather than having an easy comfy file per different version different tool chain to sort of collapse things together in a single file and we could do these things separate perfect mapping of what we have now in Yammerl syntax and then what I call fat easy configs later I don't have time to work on it at all it would also complicate things a bit like how do you make the transition what things do you need to change on the easy build command line if you have these fat easy configs how do you tell easy build use this version because you have all these options so what first thing I did was to create something that dumped from the Yammerl format into the current format the first thing would be to just to be able to parse that file the parser is already in there maybe it needs to be updated a little bit and I have a pull request open for being able to dump that out into an easy normal easy config format and then you can do your dependent I mean it's a very indirect way of using it so first you take it you deconstruct it and then you use it so but that would be for me that would be the first way thing you would do and then eventually you remove this deconstructing part because you don't really need it but that's the easy let's say while making sure it works I think that's the easy part the hard part is do we want to support this as a separate format do we make this the new format how do we transition from A to B making this change is going to affect a lot of people unless we have tooling around it like transit existing easy config files to a channel yeah it's hairy so I'm a bit scared of that process if you see some people are still stuck to easy build 2 even though the changes haven't been that big in easy build 3 but they feel they can it's too much of an effort this is going to have a big impact also we have 10,000 easy config files right now there's going to be those that are a problem for one way or one reason or another so it's a big effort it still makes sense to keep working on it so if people want to work on it definitely do but I have very little time to spend on it myself for me the big benefit is that in the case of easy blocks we're collecting knowledge right but in the case of easy configs things are getting lost right because people contribute something important to one tool channel but not to all or maybe people don't pick up on it because people don't pick up on something important because they already have their own custom version right so they're losing information but just moving to YAML is not going to fix that you need both YAML and the FAT easy configs for that oh yeah I mean it would be FAT I mean it wouldn't just move to YAML I mean FAT to YAML questions popping up and I hear any plans on better handling of filter depths and easy blocks yeah that was mentioned in the whoops here so I just need to hear from these people what they mean exactly it's not entirely clear are there open issues on that or if not open those issues in the room so they're on the slide so the people who are talking about this maybe mention it in the chat what you mean exactly how we come you can see what the problem is and how to fix it I guess they're talking about that you sometimes check for something that something exists right and then if it's filtered it doesn't falls over and that's actually a hard problem to fix I'm not sure how we can be robust against that the checking at the moment just checks the environment variable or does more than that yeah just does that when you filter it you define the environment but sometimes the location in the environment variable also matters how do you figure out the location for something installed in the US then you have to have a way to tell easily build where it's installed and configure it correctly so it picks it up it's doable but yeah we need more details on this so either look into an issue that's open already and spit your guts there or open a new one so we know what the exact issue is no more questions for now I guess we're pretty much on time I'm very happy with that so now we have the lunch break right for the lunch we have to submit it there is one thing for vegetable with the orange type of system that you have to know that you are one for the people on the stream we'll be back at one so about one hour from now