 Welcome back to the last talk of the day. This will be Kenneth Hoss from HBCU-Gent, giving us the easy build state of the union talk. Thank you, Simon, and welcome everybody to the traditional state of the union presentation at the user meeting in a little bit different setting this time. So for the people who do not know me, I'm Kenneth Hoss, I've been working at the HPC team at Ghent University over 10 years now. And I became responsible for user support and training. And as a big part of that, I'm also responsible for the software installations we do on our HPC clusters. And as a part of that, I became the lead developer, the release manager and the benevolent dictator for life as they call it, of the easy build project. I did not create easy builds, it was already there when I joined the team in Ghent, but it was quickly handed over to me as the person who did create, didn't have time anymore to maintain or develop it. A little bit more about HPC again. So we are the HPC team that is central to Ghent University. So we help out anyone in Ghent University or by extension in Flanders who wants to do scientific computing. We have a fairly small amount of resources, about 20,000 cores in total currently with a handful of GPUs on the side. And our latest cluster is AMD Rome. So that's getting interesting. And we have a big or bigger cluster coming up, just named Hortons, just currently under construction. So that's about well over double the amount of AMD Rome cores we have compared to our tier two setup. And it will also include 80 A100 GPUs. We're also a member of the Flemish Supercomputing Center, which is a collaboration between the Flemish universities on everything related to scientific computing. So as usual in this talk, I will cover most of what has been asked in the easy build user survey. So the survey is something we do every year. This is the fourth time in a row that we've done the survey. So it's starting to get interesting in terms of trends, looking back a couple of years, what has changed recently, where things are going or yeah. It's becoming interesting to take a look at these results and see what's happening. We got good participation again, close to a hundred people as we did in previous years. And I'm not gonna cover all the questions here because there are some questions where we keep seeing the same answer. So it's not really very interesting to discuss these, but the full survey results are available on the easy build website, on the user survey part of the website. You can scroll through the raw results, including all the answers to the open questions are available there and pick PDF file. And I'll cover the highlights here in this presentation. So to start with the demographics of the easy build user community, so what kind of profile do people have? What type of organization do people work for? This is actually pretty stable over the years. We don't see big shifts here. So most people are identified themselves as a system administrator or as a user support, as a member of the user support team. And then there's a smaller part to our scientists, researchers, managers, software developers and so on. Also the pie chart on the right, which shows what type of organization people work in, pretty much we get the same distribution over the years. So this is not maybe not very interesting to talk about. The next one also in which part of the world are you located three quarters of the community, at least based on the survey are located in Europe. And that's also what we see in terms of who is active in the community. So this is also, again, very, very similar to previous surveys. So why do I mention it? Because it's interesting to take a look at the differences with the SPAC community. So SPAC has done a user survey as well very recently. So I think they wrapped it up in November or December of last year. I wonder where they got the idea to do a user survey. But if you look at the results, so many of the questions are actually very close to what we have in the survey. And two of them are also what kind of user are you? So what kind of profile do you have or what country are you in or in which part of the world are you in? And if you look at the answers for the type of user profile, you'll see there's a big difference between EasyBuild and SPAC. So SPAC is more used by, or at least a bigger part of the community as software developers and scientists. While in EasyBuild, it's more system administrators and user support team members. So you can see that it's almost a night and day differences, maybe a bit strongly worded, but you get the idea that the user communities are very different. In terms of where people are located as well, you can again see most of the EasyBuild user communities in Europe and a smaller part is in North America, which includes Canada. So that's important to know in the comparison. While on the SPAC side, the biggest chunk of people is in the USA. And then the smaller part is in Europe, which doesn't include everything in the rest of the chart because that's country-based, which Japan is separate, Australia separate, things like that. So again here, you also see a big difference between the different communities. And I think a big part of the reason for that is the impact that the ECP project has on the SPAC community. So they get a lot of funding through ECP and hence also a big part of the user base comes through the ECP project. If we look at how long people have been using EasyBuild, this looks very similar to previous years, but we do see a bit of a drift towards people that have been using EasyBuild for a long time. So two to five years or even longer. That seems to be growing. That's that ratio of the user community. But we do still see new people coming in, even though the new people are less relatively speaking compared to the experienced people. So three quarters of the community have at least two years of experience with EasyBuild. So that's very important to know also when we ask for feedback like in this survey or just on the mailing list or in Slack that the new people are getting, let's say a lower weight in discussions than experienced people. So it's very important to be aware of that. Then how did you first learn about EasyBuild? The majority is still here work of mount. So that's people telling each other about it is about one third of how people learn about EasyBuild. But maybe the more interesting thing to notice here is that it was already in use when I joined the job that I'm currently doing or when I started the job that I'm currently doing that has been increasing over the years and it's now up to a quarter of people that got to learn EasyBuild through that. So because the site where they started was already using EasyBuild. So to me, that's a very good sign. People who start using EasyBuild stick to it because they're probably happy with how it works. Looking at some of the highlights that we had in 2020 in the last year. So 2020 was a bit of a weird year of course, but for EasyBuild specifically, there were several highlights. We see the community keeps growing. We see also growth in the amount and also in the quality of contributions. That's very good, but as maintainers, we need to be able to keep up with them. So make sure we take a look at incoming contributions, make sure we process them, review them, try to work together with the contributors to get things merged properly. And up until now, it seems like we're managing to do that. And I'll get back to that towards the end of the presentation. We also saw a big increase in the number of supported software packages. So about one quarter, yeah, we have about one quarter more software packages supported than at the end of 2019. So that's quite impressive. We have a proper EasyBuild tutorial now. So I'll have a separate slide on that. We also have our first big security bug that was fixed and which has a proper CVE identifier. So the issue was that the GitHub tokens that you can give to EasyBuild, so it can interact with the GitHub API. That was leaking in an EasyBuild log file under very specific circumstances. So it was not always leaking. You had to do two or three things in a certain way for it to actually be leaking. But yeah, this was noticed by one of the EasyBuilds maintainers. We quickly fixed the issue and pushed out a release straight away. So we didn't wait with pushing out a new release. And we took the proper steps and reporting the issue and getting a proper identifier for it and so on. So that was a bit of a new experience with us. But to me, it shows that we're doing well. So we know how to go through that process. We're taking steps we were supposed to take. So I'm actually quite happy with how that was handled. Not happy about the bug, of course, but yeah. We've phased out the use of the Travis platform for running our test suites in the different EasyBuild repositories. We're now fully using GitHub Actions. For multiple reasons, Travis was being a bit of a pain. It was quite slow. Some of the tests were failing for connectivity issues or problems not related to the EasyBuild code at all. We're seeing a lot less of that in GitHub Actions. And then also somewhere, I think around summer 2020, Travis made some big changes in terms of how many resources it provides to open source software. So things got very tricky there. So we were very happy that we were already actively working away from Travis. We made an effort to make all the Python code. We have PAP8 compliance. So PAP8 is a code style standard in Python. And we also included the code style checks in the CI tests as well. So any new code or any code that we change will also be PAP8 compliant. And we have a new domain for EasyBuild. So EasyBuild.io is now the homepage and we try to make things easier in terms of the URLs we use for documentation. And so looking at supporting software, as I already mentioned, the number of different supported software packages keeps coming up, so no surprise there. And it's been fairly stable over the years. So it's almost a linear increase, which is good in some sense where we don't have many problems with adding more or supporting more software packages, but it's also a concern that it keeps going up. So there's no sign of slowing down or anything like this. Maybe even it's likely speeding up. So yeah, having to support more and more software is obviously a problem, but if the users or if the researchers keep asking for new software, then that's what we have to support. The EasyBuild tutorial. So this is new. So having a proper EasyBuild tutorial is something we didn't have up until mid last year. We have made a proposal or we submitted proposals for an EasyBuild tutorial at both supercomputing and the ISC conference several times, but it was never accepted, even though the tutorials usually got pretty good reviews. Somehow we never made the cut. That changed in 2020. So we did get accepted for the ISC conference, the European Supercomputing equivalent, let's say. So that was very good news, but then of course the pandemic hit and ISC was in trouble. They were not able to do a physical meeting. They ended up actually postponing all tutorials. So there were no tutorials at ISC 20, including the EasyBuild one. They were postponed to this year, but we figured we were already preparing things and we figured to just push through and organize the tutorial ourselves online. So we did similar to the setup of this meeting. We had a limited Zoom meeting, which was live streamed to YouTube and we handled all the questions via Slack. We ended up with over 100 people registering for the tutorial, so that was quite good. And since it was the first one, we were maybe a bit over ambitious in terms of the number of content. That we had in mind and also the amount of time we had planned to cover it. That didn't really work out very well, but at least the tutorial is there. It was fully recorded. People can get back to it. If things were going too fast, they can scroll back in the recording and so on. And we did an evaluation afterwards and we still got a 4.3 out of five in terms of score. So people still liked the tutorial, even though the most common remark was definitely having too much content or maybe going through it too quickly. So that's something we'll take into account for the next installment of the tutorial. So as mentioned, accepted tutorials were postponed to 2021 at ISC. So that means we again have an easy build tutorial or at least hopefully this time it will actually happen at ISC 21. So we'll use the tutorial that we have. We'll fine tune it a bit, maybe trim the content a bit so it fits in the timeframe we have in mind. And hopefully you'll hear more about that, about the easy build tutorial at the next ISC. Then the new domain, there's not a lot to talk about here, but we now have a proper homepage which just redirects to the homepage we have on GitHub, but that URL was a bit long and maybe hard to remember. So now it's just easybuild.io, which is very easy. And also the documentation now lives on the same domain. So docs.easybuild.io is easy to find. And yeah, we should have done this years ago, but somehow it never happened. Then more highlights in 2020 in terms of changes we've made to easy build on an additional features, the try update depth option, which is still experimental as something we spent quite a bit of time on or at least Alan has spent quite a bit of time on getting that properly implemented. We think it's a very useful feature, certainly not perfect, but it can help you a lot with updating existing easy config files to another tool chain and updating all the dependencies along with it. So that should automate a big part of that process. We also have proper locks now. So if you're running easy build on two different systems and installing in the same file system, they will not step on each other's toes anymore. So one of them will run into an existing lock file and it will just cancel what it was trying to install while the other one will continue. We have lots of speed ups in easy build itself in terms of stuff it was doing in the background over and over again without it really being needed. And that was causing significant slowdown in terms of responsiveness of the EP command. So several of these issues were fixed. Last year, we have better GitHub integration. So the new PR option, which you can use to open pull requests on GitHub straight from the easy build command line. That also works for framework and easy blocks. So not only for the easy config files and sort of vice versa, you can now also pull modified easy blocks from an open or merged pull request from GitHub. Very similar to how you can pull in easy config files from pull requests. We added support for installing software in an alternate system root environment. That there's a lot of background to this but this will become more clear later this week when we talk about Gen2 prefix and the easy project. And we also improved the Rpot wrapper script. So the script that we have to do Rpot linking of libraries rather than regular dynamic linking was improved a lot. There were some issues that it wasn't taking into account or some bugs left and right that were fixed. And next to that, we also have a new concept called easy stack files which I'll provide some more information about later. So these last three things, the sysroot support, the improved Rpot script and the easy stack files were actually a direct result of the easy project where easy build is one of the components. So we were using easy build or we are using easy build there in a slightly different context than we are used to. And that was the main driver behind these changes and these enhancements. Next to that, there's also significant improvements for especially for TensorFlow and PyTorch. So Alexander from the University of Dresden has spent a lot of time in applying patch files to TensorFlow, figuring out problems with the tests, figuring out issues with both TensorFlow and PyTorch on power systems, for example. So there's lots of changes in how stable the TensorFlow installation is when using easy build same with PyTorch. Also runtime problems, bugs in the software itself that we have patches for that are maybe not merged upstream yet or only coming in future releases. So there's a lot of these things that have happened in the last year. And more general, we also have improved support for both ARM and PowerCPUs as well as a REL-8 based operating systems like CentOS 8. So there were some issues, small issues there left and right that also got fixed last year. So looking into detail about some of these things especially the easy stack files. So that's a new concept. It's still an experimental feature in the latest easy build release. So it's not fully stable yet or at least not fully capable yet with what we have in mind. Easy stack files are a way to describe a software stack that you want to install with easy build in YAML format. So there's an example shown on the slide on the right where it says, okay, I wanna have bio conductor, easy build itself, Gromax open format are installed. And I wanna install these with this particular tool chain. So in this case, FOS 2020-A or for easy build the system tool chain, this particular version. And for Gromax, it actually has two versions with FOS and one version with FOS CUDA. So you can see this is a really condensed way of expressing what you want easy build to install. And then when you give this easy stack file to easy build and you enable the robots, it will make sure that all the dependencies, all the tool chains that are required for these installations are installed. So rather than having to give the EB command all the names of the easy config files that correspond with this, you can give it just a single file and it will make sure that everything is in place. Currently, it's a software first kind of structure in these files. So you list the software and then for each software you listed tool chain. We're probably eventually gonna support the other way around as well. So tool chain first and then software if that's what you prefer. The example file I'm showing here should already work in the latest easy build version, but we have some enhancements in mind, which make, we wanna be a bit careful there. We may have to change some of the behavior while implementing these enhancements. So that's why it's still an experimental feature. So we have to configure easy builds to allow you to use experimental features so you can use it. The enhancement we have in mind are support for also specifying easy build configuration settings. So for example, always enabling the robots when using a particular easy stack file or pulling easy config files from a pull request with Trump PR. So you can specify these things in the easy stack file. We also wanna add support for filtering where you can say, okay, grow max with Foscuda should only be installed on a system where easy build is configured with a GPU label. So where you can say to easy build, look, this is a GPU system and any software that's relevant on a GPU system should be included. Other things should not be included or the other way around. So we'll build or implement a way of expressing that and making sure you can specify that in the easy stack file and in addition, we also wanna add support for regular expressions so you can give it, you can use a wild card for version, for example, and say, okay, any grow max version you know about with this tool chain, please go ahead and install it. And then when there's easy build updates or new releases of easy build that may include easy config files for newer versions of grow max, you will get them automatically when you do another run with the same easy stack file. So all of these enhancements are ideas currently but we actually have somebody who's actively working on implementing these. Then the easy project, we have a separate presentation on the easy project tomorrow I think. So I don't wanna steal too much of Bob's thunder here but I do wanna highlight so I think this is a very important project. I was invited into the project by the people who started it, mainly from the easy build side, but I've been quite active in it and I've been helping out a lot. It's a very nice active bunch of people who are working towards a common goal where easy build is one of the key components. So I think it's a very important thing to learn about. And next to easy build, we're also using CERN VMFS, Gen2 prefix in this project, the Lmod modules tool and so on. So several of the talks during the easy build user meeting this week are actually motivated by the easy project. In some sense, I see easy as the next step after easy build where you don't just work together on a tool to facilitate software installations but actually work together on the software installations themselves. And projects like CERN VMFS make that possible in an organized and good way. So again, please take a look at Bob's talk tomorrow about that new new TC to learn more about this project. Then back to the survey results. So one of the other classic questions we ask is what kind of operating system people are running easy build on. CentOS or REL based operating systems are still the most prevalent here that has been the case over time and is not a big surprise. I think so it's pretty well known that REL based systems are most prevalent in the HPC community. CentOS 7 usage is slowly going down. So people are starting to switch to CentOS 8 and REL 8. The people that switched to CentOS 8 after the news that Red Hat will be making this CentOS 3 which is more like a rolling release maybe scratching their head a bit after that after hearing that news. So it's gonna be interesting to see how that works out. And there's already a new REL based Linux distributions being developed currently, the Rocky distribution which is not available yet. So you cannot install or use it yet but I think by summer of this year it will be usable and it may be a good replacement for people who started using CentOS 8. One thing that was striking for me here is that about almost 10% are still using CentOS 6. So this is probably on old and hopefully dying clusters. It's been going down but a lot slower than I was expecting. So 10% is still quite high, I think. People who use Ubuntu seem to be more aggressive maybe and into switching to a newer version of the operating system. And we also have Gen2 now as a new entry. So this was not listed as an option in previous editions of the survey. You could add it as an other category yourself but now we've added it here and about 5% of the people were using EasyBuild on top of Gen2. So that's probably people who are either involved with the Easy Project or who work in the Compute Canada Consortium where they have a very similar setup to what we do in the Easy Project. And overall this has been pretty stable. So REL based systems are still most popular. Next ones are Ubuntu and then there's a long tail of other operating systems that EasyBuild is being used on. Nobody answered Raspberry Pi OS. So maybe we can see that change as well next year. Then sort of related question is which Python version rather is being used to run EasyBuild. So this is not about what you install with EasyBuild or what you use to run EasyBuild itself with. So here we're expecting a shift to Python 3 since Python 2 is officially dead. It was dead end of 2019, sorry, even though they still had a last bug fix release in April of 2020. But still most people are using Python 2.7 as the Python version to run EasyBuild with. That's probably not a big surprise because Python 2.7 is still the default Python in CentOS 7 for example. So if most people are using CentOS 7 they're probably also using the default system Python which is why they're using Python 2.7. There's an increase in adoption of Python 3. So about one third of the people are using Python 3 to run EasyBuild. But that pickup seems to be relatively slow, especially if you know that Python 2 is officially un-maintained. So that's maybe a bit concerning. And as expected there is splintering across the different Python 3 versions. So Python 3.6 and 3.7 are more prevalent. But the more recent Python 3.8 and Python 3.9 are already being used and are probably gonna be increasing over time as well. So we're coming from a couple of years where basically everybody was using Python 2.7 and we're going towards a new era where there's a lot more splintering in terms of Python versions. That's a bit concerning in terms of EasyBuild development. But as long as we do a proper job in CI and making sure we're running the test suites on all of these different Python versions, we should be okay. While working on these slides, I discovered that I did some predictions last year. So I predicted that most people would still be using Python 2. So that's the case. Predicted a significant increase in Python 3 use. That was maybe an easy prediction to make. But that's also the case from going from 16% to 36%. And I gambled that Python 3.6 would still be the most prevalent Python 3 version. I got very close to that, but it's actually Python 3.7 is the winner here. So just for the fun of it, I made new predictions for this year. If we do the survey again at the end of 2021, I think we'll still see that Python 2.7 is actually the most used Python version to run EasyBuild, even though it's a dead language, it's a dead version. We're probably gonna see more splintering across Python 3 versions. And I think or hope that Python 3.8 will be the most prevalent Python 3 version, even though it's now pretty small at 3%, I think that's where we're going towards. Then related questions we ask is how troublesome would it be if EasyBuild becomes incompatible with Python 2.6? So EasyBuild today is still compatible with Python 2.6, just because it's pretty easy to do that. If you're compatible with 2.7, you're basically also compatible with 2.6. So there's very little difference between these versions. It's clear that most people wouldn't really mind that we dropped support for 2.6. We've actively deprecated Python 2.6 support in EasyBuild 4.0 already, which is September of 2019. And yeah, we're happy to see that the need for Python 2.6 is declining fast. So even people who are still using Central S6 probably have already updated to Python 2.7, even though 2.6 is still the system Python. So yeah, we're ready to let that die finally. Then how much of a problem would it be if Python of EasyBuild went to Python 3.0 only? So only compatible with Python 3. That seems like it's maybe a bit too early to do that. So people would be concerned about doing that, about 15% says, yeah, that would be an issue for me. We currently still don't have any serious plans to drop Python 2.0 support. So support 2.7 rather is not a big issue. And in addition, there's probably not a big benefit in only supporting Python 3. So in terms of code base, I don't think there's a whole lot of changes we would make if we only had to support Python 3.0. There's very little incentive from our side to go Python 3.0 only. Maybe the only advantage would be being able to run the test suite and less configuration so not caring anymore about Python 2.0. That's maybe the biggest advantage that we would see. So short-term, no plans to drop Python 2.0 support. If we do it, we'll probably do it in EasyBuild 5.0, which is currently not in view either. So we don't have any big plans to make big changes that would warrant an EasyBuild 5.0. And just to make it clear here, how things work currently, so if you run the EB command, it will pick a Python command that will work when using EasyBuild so it checks the version of the Python commands it runs into. Currently, it will consider first the Python command, so the default Python on your system. If that doesn't work for whatever reason, so if it doesn't like the version or the Python command is not there, it will try Python 3.0 first. And if that doesn't work, it will fall back to Python 2.0 in the hope that it finds a 2.7. If you don't like this order or commands that it checks for, you can define the EB Python environment variable to tell EasyBuild what to use, and then it will at least try this first before it looks at any of the other three. So you can just set the environment variable and run EB Show System Info, and that will tell you which Python version it's using. So I would recommend that people opt into using Python 3.0 to run EasyBuild through this mechanism. And if you wanna see which Python commands EasyBuild considers and whether it likes them or not, you can use EB verbose equals one to get more information about. Then back to the survey, we also always ask on what processor families people are installing software with EasyBuild. So here Intel is still king, and Intel Skylake is the most prevalent, or I should say Skylake X, I guess the one that supports AVX 512, so the server version. That's still by far the most used CPU. The older Intel CPUs, even very old ones are still around like Sandy Bridge and even Harper Town and the Halem and Westmere are still being used. And especially Sandy Bridge is still close to 30%, which is quite high, but at least it's going down. So if we compare to last year, last year about 50% of the people still had the Sandy Bridge cluster, now it's only about 30%, and also Haswell is slowly going down. No surprise here, AMD Rome is now at 36%, so it has seen a big increase compared to last year when it was only at 14%. So the second generation Epic AMD processors are becoming quite popular in the community and there's no real surprise there. So just again, for the fun of it, my prediction for 2021 is that we'll see a strong increase in AMD Rome. So I think it's slowly gonna be a good competitor for Intel, for several reasons, performance, price, all these things. And we'll probably also see a strong increase for ARM processors. So right now they are barely visible in the plots. So if you look at Graviton 2 and Thunder X2, only a handful of people actually install software with EasyBuild on these platforms, but I think that this will change dramatically in 2021 because ARM is becoming a very interesting platform to use for HPC. Then Accelerator, so same question, but for basically GPUs. Of course, Nvidia is still king here, no surprise. The Nvidia V100 and the Pascal series are very popular. The Capular, older GPUs are also still quite popular. And there's a quick adoption rather of the new Ampere, the A100, Nvidia GPU cards. So they're already at about 25%, so that's quite high. There's currently very little adoption for AMD GPUs and Intel XE, I guess these platforms are just too new currently to already be used very actively. But that's probably gonna change in 2021. We'll see more and more systems coming up with AMD GPUs. And 2021 will probably be a bit early for Intel XE, but also there we'll see systems that start popping up that use these Accelerators. Then more EasyBuild focused, so which tool chains do you use for installing software? So I tried to organize things a bit more here than I did in previous years to get a better view on it. So all the green bars are versions of the FOSS common tool chain. Blue is FOSS CUDA, yellow is Intel, red is Intel CUDA. And then the last box is a big mix of things. So patterns I see here is that the 2019 B versions of the common tool chains are most popular, are still most popular. That's probably because of the amount of easy config files we have. So we were using the 2019 B versions for a long time because 2020 A was delayed for several months. So yeah, we had the longer lifetimes, there are more easy config files for it. So it makes sense that more people are still using that. 2020 A is a close follower and at least we see pretty good adoption. So the most left hand graphs in each box are the most recent versions. So they are getting good adoption. So at least people are following along, are not getting stuck with very old, or yeah, very old tool chain versions. There's a big gap between FOSS and Intel. Again, no surprise there, Intel was licensed. That this is changing with the one API versions. At least those are freely available and free to use. So maybe we'll see changes there as well in the coming year and years. The IO-MKL tool chain. So the first one here, that's Intel Compilers Open MPI and MKL library is slowly increasing in adoption. So that's currently at 20% while it was lowered the years before. Yeah, and there's a long tail of different tool chains. Then here another prediction and we've had some discussions on this already in the recent easy build conf calls is that we'll probably see an increased adoption of client-based tool chains. This is a bit of a very easy prediction to make because most of the new tool chain versions like Intel one API or the AOCC compiler or the NVHPC compiler, which used to be PGI are basically all client-based. So this is almost a given it's not really prediction. But it does mean that we should probably start paying a little bit more attention to client-based tool chains and easy build going forward. Then which easy build version do people use? So the good news here is that 90% of the people relies on easy build, on proper easy build releases. And this is I think slowly increasing over time as well. So that's good. That means all the effort we spent in pushing out releases regularly. So about every six weeks is definitely worth the effort. And people seem to be happy with the releases that we push out since they are certainly more stable than the developed branch, which is, it can always have bugs that we haven't discovered yet. Also here, 80% is using easy build 4.3. So very close to the latest release. And over 50% is using the latest release which was released somewhere mid-December, I think. So that's also a good sign. There's quick adoption of the new easy build releases. So that means the jump to a new version is quite easy to make. Nothing breaks, usually when you update to a new easy build version. About 10% is using develop. I would guess most of those people are maintainers. So who pay close attention to what is moving the developed branches or are happy with running into a bug left or right and then try to maybe fix it themselves. People who are not using the latest release, so about four out of 10 people mentions they were not using the latest release. So that's the split up here. So some people just say the current version works fine for me. So that's about one third. Another third said I should probably upgrade but I don't have time to do that even though the upgrade should be straightforward. And then other reasons either using develop or I've made local customizations and work myself into a corner that takes time to get out of or then a variety of other reasons. Some people only update easy build when they jump to a new tool chain or things like this. So overall, this is pretty similar to previous surveys. In terms of frequency of releases, we've also asked this question every year and here more and more people seem to be happy with the frequency we have, which is about every six weeks or two months somewhere in between that range. We tried to push out a new easy build release. The hope was to push out a new version before the easy build user meeting this week that didn't happen, but we'll probably work on that either during the week or immediately after it. So in terms of frequency, people seem to be quite happy, that's good. Then another question we always ask is which part of easy build do you not like? And we usually have a pretty good view on what that is, but it's good to see that confirmed here in the survey. So about 30% of the people said they don't dislike anything. So that's pretty positive, that was nice to see. The most unliked aspect is the fixed versions that we have for dependencies and tool chains. We know that this isn't an issue or something that people don't like about easy builds, but yeah, it's a blessing and a curse. So it's good because that helps with, if somebody gives you an easy profile that works for them, it will probably also work for you. A big part of that is because we fix versions, but it's also a curse. So it makes it annoying to update to new software versions and update the dependencies along with them. That's currently mostly a manual job, even though there's things like try update depths that are a big help in automating that. So yeah, we should probably work on that and try to get this number down a bit. The error reporting. So yeah, in this survey, it comes forward as another aspect that people don't really like. It's also the main type of question we get in the Slack channel, for example, so people who tried something, it didn't work and then they're kind of stuck because they can't seem to figure out themselves what the actual problem was. So either maybe the compiler ran out of memory or out of disk space or yeah, they have trouble finding that the easy build log file. Yeah, we can probably do a better job there by being smarter about how we report errors about extracting more information from the output of the command that we were running, look for patterns. Yeah, we can do a lot better there and we have some ideas on how to do that. We need to work on it, but it's definitely in the back of our minds. Then the next most popular thing was the lack of manpower either for dealing with contributions or doing easy build development. That has been also raised a number of times basically in every survey. We currently don't have any dedicated funding for easy build development. So that's something we're trying to look into also actively looking into that but it hasn't worked out until now. So it's pretty difficult to find funding for ongoing projects so that are not research or that don't have, let's say, a huge impact on certain aspects of science. So it's quite difficult to motivate funds for that. We currently try to remedy that by making sure we are organized with the manpower we have. So one example of that is in the group of easy build maintainers, which is about 15 people, I think. We have a rotating maintainer of the weak role where we ask, okay, who has time, this particular week to pay more attention to incoming pull requests and then those people try to spend at least one hour for working day and actively looking into stuff that's coming into easy build. And that really helps. We see shorter turnaround of pull requests and more questions being answered because we know one person is paying more attention to things. Then other tools or projects that people are using in combination with easy build. No surprise here that most people are still using LMOT as the modules tool. And I think the tickle-based ones are actually going down a bit in popularity even though the 4.x version is being actively maintained. Interesting here is that singularity is, the popularity of singularity is decreasing. Maybe that's just because slightly different people answered the survey compared to previous times. So about a hundred people, it's maybe not the best possible coverage across the HPC community, but it was still jumped out for me a bit. Similar for SLUR, small increase there, but it's probably not to be overstated. Condal, Docker, OpenHPC and SPAC are pretty much a flat line in terms of how many people are using it. That's for most of these projects without naming them, that's good. That it's, the adoption is not increasing. The one that still puzzles me here is seeing about 30% of the people saying they're using both EasyBuild and KONDA to install software. So that's a bit mind-boggling to me. I'm not sure why that is or how people are actually doing that. And new here are CERN VMFS. So that's about 12, 13% are using EasyBuild in combination with CERN VMFS. Again, that's probably people in the Easy Project or Compute Canada, but maybe also other side. So like I know CSCS is using CVMFS as well. And the ArchPEC tool which EasyBuild can leverage to detect what kind of CPUs you are running on is a small but very useful library as well. Then communication in the community. So are people subscribed to the mailing list? We still have about 280 subscribers to the mailing list, but the activity is going down there. That's not a big surprise. We're seeing a shift to Slack for that. And this also comes forward in the survey. So less people are interested in actively participating in the mailing list and more people are interested in actively participating on the Slack channel. So things are shifting from a traditional mailing list to Slack and that's not a surprise. Oh, and also something that has been fairly consistent over the years in the survey is that we noticed that the significant part of the community remains silent. So maybe they're reading the mailing list or reading on Slack, but not actively participating in discussions. So that's very important when we try to pull in Slack, for example, for the opinion of people we should be very aware that a lot of people are just not speaking up at all. So it's not because we ask something there that it covers the whole community. That's very important to realize. The mailing list is going down in terms of traffic. So we still have 280 people subscribed there. I would guess many of those are not actually looking at the posts at all, but we still see it actively being used for questions that people are still getting their questions answered there. And I would say usually in a fairly quick turnaround time. So it's certainly still useful. And there's no point in stopping the mailing list. I can imagine some people still prefer it over Slack as well. And then the Slack channel has grown over time. No surprise, even in 2020, it almost doubled in the number of members. So we have 160 people extra compared to the end of 2019. And also the number of messages that were sent over time since the existence of the channel of the easy-built Slack in 2017 has almost doubled in 2020. So we're close to 200,000 messages sent now even and yeah, end of or early 2020, that was about 105,000. So it's almost doubled in a year time. So yeah, it's the main way that people interact with each other about the easy-built projects. Then as a bit of a sidestep, something new that has emerged last year is the easy-built tech talks. So that was an, it's actually an indirect result of the previous user meeting where I invited one of the lead open MPI developers to give a keynote talk at the last easy-built user meeting. Somehow he overlooked that email and then five months later, he mailed back apologizing for not responding to my email and that he was very much willing to give a talk like that for the easy-built community. This was months after the user meeting, so that wasn't possible to organize then. But we figured we could still set up a talk like this, a sort of keynote talk, but very technical and in-depth about the open MPI project. We set that up and while we were working on it, so back and forth with both Jeff and Ralph who did the presentation, it was clear that there was absolutely no way that they were gonna cover it in a single hour. So they ended up actually doing three sessions covering different aspects of the open MPI project where it is now how it is structured, where it is going and so on. So those were very interesting talks which are now also a key part of the open MPI documentation themselves. So if you look at the open MPI website, you'll see a video section if you go there. The easy-built tech talks are right there. They see this as a very good way of documentation, the open MPI community. We had a second easy-built tech talk as well since there was a lot of interest in ARM, especially when Nvidia announced that they wanna buy ARM. So yeah, we were wondering how mature is ARM for HPC and so on. So we found a good speaker for that, Chris Edsel, who works at an HPC site in the UK where they have a big ARM HPC cluster. So he gave a very interesting talk about that as well. And we hope we can do more of these tech talks in the coming months. So if people have suggestions for specific topics, I'm glad to hear them. And especially finding good speakers there who can give a technical presentation really in depth. So nothing marketing or anything related to that, but just promoting the project, really going pretty deep in a technical way is interesting. Then also in terms of communication within the community, I wanna highlight that even though we're pretty vocal about this on the easy-built mailing list at least, is that we have easy-built conf calls via Zoom every other week, which are open to anyone to join. So anyone who just wants to listen in or who wants to see what is being actively worked on or who has some questions about easy-built is free to join the conf call. So the Zoom call is open to anyone and we share the link through the easy-built mailing list and also through Slack. We've been doing these conf calls actually since November, 2013. So I was a bit surprised when I looked back how long we've been doing these. So yeah, over seven years now and yeah, we've barely missed any of them. So pretty much every other week we have a conf call even if I'm not around as the lead developer then somebody else takes over and usually there's a good discussion going on. So we've had over 160 of these conf calls up until today and they've been very valuable. The conf calls are now organized at two different times. So at 10 a.m. European time, Central European time and at 5 p.m., so two weeks later, Central European time and then again it flips back and forth between 10 a.m. and 5 p.m. to cater to different time zones. So that was by request by some people more East who had trouble with joining the conf call because of the time of course. That has worked out quite well. The attendance has been pretty good over the last couple of sessions. Sometimes even with 10 plus people hanging out there and joining the discussion. I usually try to create an overview of recent developments in the last two weeks. So it's clear that what we are working on, what will be included in the next release, what is still blocking the next release and so on. And there's always time for questions that anyone can raise. We keep detailed notes about the conf calls as well on the EasyBuild Wiki and that seems to be a valuable source of information as well. Then looking at contributors a bit and I cannot resist with including this GIF in the presentation again like I've done in previous years. So looking at EasyBuild contributions starting with the framework, the core, the heart of EasyBuild, which is in a separate repository. So how things are changing over the years. So the graph shows it's missing a proper labeling on the Y axis, but the graph shows the number of pull requests to the EasyBuild framework repository and by whom they were created. So the blue part is myself. So I've opened just above 100 pull requests in 2020 for framework. There's a small part HPC again which is actually mostly consultants. We pay to work on EasyBuild. Then another part maintainers and a significant part coming outside of the EasyBuild maintainers. So that's good. It seems that people who are maybe less familiar with the insights of EasyBuild are able to make useful contributions through pull requests. So over 50% of the PRs in 2020 was from people outside of the HPC you can team. So that's an important metric to mention. And you see kind of peak here in 2019, that's mostly because of the effort that we put into porting EasyBuild to Python 3. So that's why there were significantly more pull requests than compared to previous years and compared to last year. And similar view for EasyBlocks. So here we see an upwards trend in the number of pull requests in total. And my involvement in EasyBlocks stays more or less stable. So that's good. We're getting more changes in and I'm actually less involved. So I'm happy with that. Yeah, so about 30% more pull requests which I guess is the best unit of contributions you can get compared to the year before. And over 65% by outside contributors. So people who are not even in EasyBuild, outside contributors meaning outside of HPC again and about 40% by people who are not an EasyBuild maintainer. So they make changes, they propose changes and then one of the EasyBuild maintainers reviews, tests and branches those. And then for EasyConfie files the numbers are a bit more spectacular. Also an upward trend which is luckily a bit slowing down last year. So we have two years in a row with over 2,000 pull requests to the EasyConfie repository. So 2,000, that's about, if you think about let's say 250 working days in a year, that's about eight pull requests per working day. So that's quite a lot. So the growth you see here since 2016 is partially or probably mostly thanks to the new PR functionality we have an EasyBuild where you can very easily create a pull request from the straight from the EasyBuild command line without running any Git commands or without directly interacting with GitHub. So it makes the bar to contributing significantly lower. And also here we see a growing ratio in terms of contributors from outside of HPC again. So I'm the blue part again. The red part is our consultants for the most part and my colleague in the team. And then everything else above that is people outside of the HPC again team. So we're still very active but we're getting relatively more contributions from other sites, which is very good. And then looking at these EasyConfie PRs again but in a different way. So these are the same numbers per year but the colors are different. So this is more in terms of processing at least the left graph is in terms of how or who processes the contributions. So the dark green is myself merging pull requests for EasyConfie files. So I was involved or I merged about just under 500 pull requests last year which is still quite a lot. The light green is other maintainers. So they collectively tackled certainly the biggest chunk of pull requests. So if you look at the split between the dark and the light green in 2020, about three quarters of all merged pull requests were handled by somebody else that needs. And that's a quite good metric or yeah, you could say one quarter is still way too much. It probably is but I'm of course very active in the project. And then the right graph is showing how many of these pull requests were created via the new PR feature in EasyBuilds where you don't need to leave your terminal to make a contribution. And there we see an increased adoption. So over 85% of people are creating EasyConfie pull requests through straight from the ED command line. While a low minority is still doing that manually which is okay. But it's very good to see very good adoption of this new PR or the GitHub integration in general. Then looking at the number of unique contributors so different people making contributions to EasyBuilds. So that's still going up. So we're still getting new people making contributions to the project for EasyConfie files. We've crossed 250 different contributors for framework we crossed over 100 unique contributors to the project in the last year. So that's very good. And the lines are still slowly increasing. The increase is more is quicker for EasyConfie that makes sense because it's a lot easier to make a small contribution through a new EasyConfie file than it is to start making changes in easy blocks or framework. And then the same thing but looking per year because the previous graph was showing the total since the start of the project. Here we look at things per year. That's important because you wanna see how many people are actively contributing recently to the project. So for EasyConfie files, we see just over 100 people, 100 different people made a pull request to, or I think that's even a merged pull request to EasyBuild in the last year. And also at least for framework and easy blocks we see pretty stable numbers over the years. So that's very good, around 30, 35 people are contributing every year to framework and easy blocks. And then the final question in other than an open-ended one in the survey was how people rate the overall quality of EasyBuild. So how happy are they with EasyBuild in general? The excellent and great numbers are actually going up there. So if you look at everything that's sort of positive, so at least okay or better, we're hitting 98%, which is the highest we've been up until now. We see significant increase in excellence. So close to half of the people rate at excellence so that I'm not sure what happened there in 2020. But yeah, there's a significant increase there compared to previous years while it was pretty stable in previous years. So I guess this can only go down in 2021, we'll see. And yeah, there was another option pretty bad and nobody answered that as we go. So yeah, very happy with the overall evaluation here. Then looking forward a bit. So some things we want to work on in 2021 is somewhat fueled by the results of the survey, but we were, for most of these things, we were actually discussing already actually in comfort calls or in the Slack channel. So the better error reporting is something I certainly want to actively work on and make it easier for people to diagnose problems, actually pinpoint what the issue was, why installation failed, and hopefully that enables them in fixing the problem themselves, or at least they can more accurately report the problem either in the Slack channel or through the mailing list or through a GitHub issue or so on. So we get a more focused effort there. We want to make it easier to reuse existing easy config files and port them to newer tool chains or yeah, automate that process a lot more so it's less of a manual effort. So to answer the biggest thing that people don't like, the fixed dependency versions, we can probably remedy a big part of that by yeah, providing more features or better support in Easeable to make that more automatic. The improved regression test for Easeable, that's actually more to help myself. So before every Easeable release, we do a full regression test of, or at least we try to do a full regression test. It's becoming harder and harder to actually run it to completion. Of all the easy config files we include in the release and we try to make sure nothing is broken, certainly compared to previous releases. So if something is suddenly broken, it means we have made a change in develop that affects something that worked before. And then we try very hard to fix that first before we actually push out the release. Right now, there's some crude scripting that I do and have been using over the years to run the regression test. I basically push a whole bunch of jobs to one of our clusters, whatever is the least busy at the time, and then I have some scripting to evaluate the results of those jobs, but it's all very manual and very hacky. It's not very clean. So we wanna really start using Reframe for that. So as Vasilius showed in his talk, Reframe is the ideal tool for running regression tests, which is what we wanna do with Easeable. And maybe we can even pull that open a bit and have a public dashboard that shows these things worked with this particular version of Easeable. And maybe there was a regression, but it was not serious enough for us to block the release and we wanna make them more public, more visible. And we also want to run the tests in a more controlled environment. So using container images so we can run in both CentOS 7 and 8, for example, maybe also a recent Ubuntu version, whatever cluster we have access to, we can test on those operating systems. So containers are very good for controlling your test environments. And hopefully we can also do this on, we actually already do this, but not extensively enough, I think on different processor architectures of different generations of Intel, hopefully also AMD Rome, but also more exotic things like power and arm. We wanna get better test coverage on for the regression test. And the Easeable documentation. So even though most people are fairly happy with the documentation, and I didn't discuss that here, but you can see that in the raw results of the survey. Over time, people are fairly happy with what we have in the documentation. We know there are some missing parts that are still not covered yet. But what I wanna do is, I think one of the reasons we are not working enough on the documentation is because it's too cumbersome. So you have to write, make some changes in the plain text file which has a particular format, but you can't really preview the changes you're making in the rendered way locally, at least not easily. Even though there's a very good tool for doing that, MKDocs, which is what we used for the Easeable tutorial. That's really a dream to write documentation and you change markdown files locally. And as soon as you save the file, it re-renders the documentation locally on your laptop. You can do even offline work on the documentation. And I think we should move the current documentation into MKDocs, so in a different format, that's gonna be a pain. But once we make that jump, and maybe along the way, we can make some improvements to the documentation as well. But once we make that jump, I think it's gonna be a lot easier to make changes or make improvements to the existing documentation. So that's one of the things I also wanna work on in the coming year. Then challenges that I see in the coming year are somewhat, some of them are similar to previous ones. The top one certainly is. So the scientists and researchers seems to be always finding new tools to install. So the quarter increase in the number of supported software packages is certainly an example of that aspect. That's gonna be an ongoing challenge. I don't see an easy way of fixing that other than collaborating more and making sure you share easy config files if you have them, even if they may need a bit of cleanup before they can be included centrally in EasyBuild. Hopefully, I think getting more contributions or making people contribute more is the only way of dealing with this in a good way. A big issue I'm seeing, which is also slowly coming forward in the survey results as well as the increased diversity, first of all, in hardware. So we're coming out of an era where Intel wasking, maybe still isking, but it's slowly changing. So AMD Rome is already a headache in terms of which software, which libraries, which compiler you should use to install software. But there's also other platforms on the horizon like ARM, for example, which I think is gonna be, yeah, it's gonna see increased adoption and increased interest in the coming years. And there's other, power has never been away, but it's still a bit of a niche thing, but it's very relevant and certainly relevant enough to care about. And on the longer horizon, there's projects like the European Processor Initiative, the EPI project, which is looking at risk-5-based accelerators. So that's yet another platform that we may have to deal with soon in the coming years. And then somewhat related to that, there's also software, there's an explosion in different compilers. Intel 1API has both the classic Intel compilers, but also the new ones. There's NVHPC, which used to be PGI, is a relevant, very relevant compiler, AUCC and so on. So there's a lot of increased diversity and a lot of challenges coming there as well. So we need to find a good way of dealing with that in some way. And then as I showed, the number of contributions we're getting, we're very happy with those. Also the way in which they are being made. So through the new PR option also makes the life of maintainers easier because things are coming in in a structured way and they look similar, they're fairly easy to review if they were done that way. So if people want to contribute to EasyBuild, I strongly recommend using the GitHub integration features because it helps put yourself and the maintainers. But with the number of contributions, we're getting 2000 pull requests per year. Certainly in combination with the previous point with the increased diversity, I think we need to be a lot more aggressive in terms of automating things. So we already have a small bot running that's very helpful and we can probably make it smarter, make it more capable and give it access to additional hardware so it can fully automatically test things in pull requests after a human maintainer has at least looked at it and submitted a review that he's happy with the contribution before it's being tested. So there I think we need to do more and try to factor out the human as much as possible because that's where most of the delay is. But this dives a bit more into the increased diversity in hardware and software. So CPUs as I mentioned, GPUs as well. So NVIDIA was very popular, is still very popular but AMD GPUs and Intel GPUs are on the horizon. So that's gonna make the diversity in hardware only worse in the coming years. And tied to that, the rise of different compiler suites, different compilers, different libraries, especially for the AMD GPUs, the Rockham stack is which we currently don't support in these build yet but we'll probably need to start looking into that. Yeah, lots of challenges here in terms of diversity. Maybe we need to react to that a bit and the common tool chains we defined. So the FOSS and Intel ones we have currently maybe that's no longer enough. Maybe we need something that's Clang-based, so C-Lang-based in addition to FOSS and Intel or maybe as a replacement to one of those. So we can only make changes like that after revaluating things a bit, benchmarking and checking how many of the big applications actually work with these other compilers. And there's been a lot of discussion recently as well about merging the FOSS and FOSS CUDA tool chain. So we have the split, which doesn't make a lot of sense because the only thing where it really differs is that OpenMPI is built on top of CUDA and CUDA are rare and that's causing a split in the tool chains and lots of duplications and easy config files. So we are already actively discussing how we can merge these two together in a sensible way. And I'm hoping that's what we'll do in the 2021 A common version of the FOSS and FOSS CUDA tool chains, but that's to be worked on. And then related to the diversity at least in CPUs, certainly Intel and AMD. We've seen some reports from Ulich mainly recently that they have seen very good results with the Bliss Blast Library, which is also there's a presentation on the Bliss Library by the lead developer and Wednesday, I think, so later this week we'll learn a lot more about Bliss, but the results we've seen especially compared to Open Blast, but also compared to Intel and ML, very promising when there were a benchmark against each other on the AMD Rome CPUs. We need to look if we see that same pattern on Intel CPUs, especially compared to ML, wanna benchmark that more, but currently it's looking like we will probably switch away from Open Blast to Bliss and the FOSS tool chains, but that's also to be confirmed after a bit more of experimentation. That's what I had in terms of slides until this morning, where I got an email from somebody in the Lumi project. So I was involved in discussions or they invited me into a meeting shortly before the end of last year because they were actively looking into which software tool they would use to install software on the big Lumi supercomputer that's coming in Finland and which will be accessible to lots of European countries. It's a new praise tier zero pre-exascale system. So the two major contenders there were no surprise, EasyBuild and SPAC, so they were comparing those in-depth and also talking to both myself and to Todd Gamlin. And the final decision which I made last week was made public today, so they sent me an email that are going forward with EasyBuild for at least the primary build tool of Lumi. So SPAC will probably also be used, will be provided by their users, but most of the core installations that they provide on Lumi will be via EasyBuild, which is interesting because it's a very prestigious project, that's one reason, but also Lumi is fully AMD based, so it has a big AMD CPU partition, but also a very big AMD GPU partition. So yeah, we'll need to work on supporting the AMD ROCOM ecosystem. And hopefully we'll start seeing contributions from the Lumi support team towards that direction. So that was a very interesting news item that I was able to include last minute in the presentation. That wraps it up for me and now I think we still have time for some questions. Yes, thank you to Kenneth for the talk. If you have any questions, then please use the raise hand feature in the Zoom or raise a question on the Slack chat. Sorry, I think I was muted there. Apologies for that. If you have a question, then please raise your hand if you're on the Zoom call, if you're just watching through the YouTube and raise it on the Slack. Yeah, raise it on the Slack and we'll pass across to you. See the first question, so I will. Okay, so Kenneth, I thought that you announced that Lumi will use EasyBuild. What do you think about the support for Cray2Chains and Cray recipes that happen with EasyBuild? What do I think about it in general or? How do you think it's gonna work? That's a good question. You probably know that better than I do, since at CSCS you're the biggest user of the Cray support we have. But you guys have plans to go into Lumi and support it to change and modify it to change or do you have any plans for that at EasyBuild? That's mostly up to the Lumi people, I would say, how they wanna use EasyBuild. And I think they already know or they know their plans on how they will use EasyBuild since otherwise they wouldn't have made the decision to go with EasyBuild. But I know they took a very, very good look at what EasyBuild can do, what it can't do, what aspects they may not be happy with, like the fixed versions. They probably also took a very good look at what CSCS is doing on their Cray system with your own easy config files and how you're handling that on top of what Cray provides. So I think it's more of a question for the Lumi people themselves. I currently don't have a very good view on how they are planning to use EasyBuild in practice. The meeting I had with them was more about them asking specific questions like can EasyBuild do this or if this happens then how can we work around it or how can we fix it? Those EasyBuild provides features that will help us in this aspect or in that. And I just answered their questions. I didn't tell them what to do or anything like this. But it seems like they see enough value in what EasyBuild can currently do or maybe they have a good idea of what they would need to do to support the parts that are missing that they don't see it as a big hurdle to adopt it. Okay, thanks. Any other questions? I don't know if there's something in the Slack channel or maybe my presentation was too long or too clear. There's no more questions, I'm happy to wrap up. It's getting late also for me, but yeah, we'll certainly be around throughout this week in the different sessions in the user meeting, also in the Slack channel. So if you have any questions, don't hesitate to raise them there either in the breakout room or in Slack. And we're happy to answer them. Yeah, there's nothing else I guess we can wrap up here. So tomorrow the first session is the CVMFS tutorial at 9 a.m. UTC and then I would have to check the schedule myself, but you can find the detailed agenda on EasyBuild.io slash EUM. So we hope to welcome you back tomorrow. Thank you everybody.