 So there we are, we'll pass over to Kurt to talk about Lumi and EasyBuild. Okay, so good afternoon, good evening, good morning or good night depending on where in the world you are. I will present you Lumi, which is one of the European PX skills systems and our early experiences with EasyBuild on it. So I am Kurt Lust, I work at the University of Antwerp, but from my position in Antwerp I work half time for the Lumi user support team, which also happens to be abbreviated Lust. So maybe first disappoint a few people, what this talk is not about? Well, don't expect a talk about sophisticated workflows with automated quality testing because we've only been on the system since October 13, 2021. So we don't even have a finalized directory structure yet and we don't have test nodes yet. That's all still under construction. Also don't expect a talk with a lot of information about how we deal with the AMD GPUs, basically because we have almost no GPUs at the moment and not yet the final ones, let alone that we have any information about the finished programming environment. The first information that we got was like last week, basically from the documentation side of Crusher, which was published Crusher, is the early access platform for Frontier, which is an American access skills system and the big brother of Lumi and architecture. So Lumi is a European project. European projects often means that funding partly comes from the European community and partly from countries that participate in the project. So that's no different for the pre-access skills systems. There are three pre-access skills systems being built. So ours is Lumi, which will be housed in a data center in Kajani in the middle of Finland. Then there is Leonardo built in Italy by Chinica and then there is Marinostro 5, which will hopefully ever be built in Barcelona if they manage to finish the procurement successfully. So it's a joint investment of EuroHPC joint undertaking for 50% of the money and then a consortium. Each of those three hosting countries has built a consortium of countries to fund the other half. So in the case of Lumi, we are 10 countries, essentially the Nordic countries. So Finland, of course, Sweden, Norway, Iceland, Denmark and Estonia, and then Poland, the Czech Republic, Switzerland and Belgium. The resources on Lumi will be allocated proportional to the investment. So half of the resources will be allocated by the EuroHPC joint undertaking with a similar program as praise tier zero access. And in fact, the first world is being managed by praise. On the other hand, each Lumi consortium also gets its own share depending on how much they invested. They can essentially set their own policies for the national access program. And in fact, even user management is distributed. There is no central user and project management. They've built a distributed system where each country enters and its own projects and manages its own users. So Lumi is the finished word for snow. It's also a character in one of the fairy tales of Henry Anderson, but it also stands out comes the marketing talk for large unified modern infrastructure. So it's essentially a tier zero Jeep you accelerated supercomputer, and it aims to enable the convergence of traditional high performance computing artificial intelligence and high performance data analytics. To that end, the main computer partition is called Lumi G is a GPU partition with around 500 petaflops vector performance FP64, coming from over 10,000 AMD MI 250 X GPUs. I say around because, okay, I'm kind of allowed to say the exact number if I would say the if I would know the exact number. The tender was like just over 500 than when we saw the first information about MI 250 it was just below. But then if you look at the specifications of Crusher than that again gives a number which is higher, as if create is allowed to overclock so I don't know the exact number that's vector performance the thing also has matrix units. So for FP64 which are twice as fast. But again that's a very theoretical peak speed because the system simply doesn't have the memory bandwidth and large enough caches to keep those units fed. So we expect the link back with four months around 375 400 petaflops from that partition. Then we have a supplementary CPU partition we have on the order of 1500 nodes again within these CPUs to sockets 128 cores in total per node so that makes for about 200,000 AMD epic course. So two main compute partitions, then there is what we call the Lumie D or data analytics partition, which is mostly meant for more interactive work so it has eight nodes with four terabytes. No GPU and then there's eight nodes with God better instead of AMD GPUs they've put in Nvidia GPUs just to make it a bit more fun for us to install software for different architectures. So this is a small partition which is called Lumie K where the case stands for Kubernetes, which is really a container cloud service. It's mostly meant to run containers that service. The rest of Lumie like a galaxy portal and so on it's not really meant to be your big cloud resource for bioinformaticians or so if they were hoping to be running only on that or to get projects that only use Lumie K that will not be possible. So that's for the compute resources. Let us also consider the storage resources we have three storage systems. The first one is called Lumie F where the F stands for flash, which is like seven petabytes of flash based storage with an IO bandwidth of around a second and hopefully good IOPS also. Of course it's running a parallel file system Luster. So that means that I mean you don't have to hope that on a single thread you can open and close thousands of files per second that's not meant with IOPS here of course. Then we have Lumie P which is a traditional disk based parallel file system metadata on flash. These are really 420 petabyte systems rather than a single 80 petabyte system so if one falls out we still have the other three to run on. And then there is Lumie O which is an object storage service with 30 petabytes of disk space may be extended in the future running self but we're still setting up that system. And so all those partitions are connected via a high speed interconnect, which in the case of Lumie is the new slingshot interconnect from HP and Cray, which is an Ethernet based interconnect but with various various extensions for better performance on small messages for better performance if the system starts to saturate and so on. And then there is room for experimental extensions also so the hope of CSC is to also acquire a small quantum computer which would be linked to Lumie, which we know already called Lumie Q. Let me quickly go over the compute node configurations of the main nodes. Lumie C is pretty boring that's just your standard two socket AMD nodes with per socket a chiplets with eight Z3 cores, 32 megabytes of share till free cash each core of course having its own L2 cache. Lumie G is the more interesting part. This is really like, I would call it the GPU first system rather than a CPU with GPU accelerator. The four sockets is not exactly the right word, maybe I should use packages GPU packages and my 250 X. Now each my 250 X actually has two GPU dice in the package with each die, having 64 gigabytes of HBM memory so 128 package or 512 gigabytes of HBM memory for the note as a whole. The two dies are connected to each other via AMD's infinity fabric so the same fabric as they are using in the epic CPUs, except that it's already running at a higher speed than the current Z3 generation epic processors. The main interconnect is also used to connect the GPU packages, and it's also used to connect the GPUs to a special version of Z3 CPU so it's special in the sense that they don't say much but I think it's really not different IO die, because that it runs at a higher speed than what's currently possible with the regular Z3 CPUs, and that has all the RAM attached to it, but so since everything is infinity band, infinity fabric, sorry, should I say I'm used to wrong word a few times connected. You have a single memory space cash coherence across the whole note and so on. Also interesting to note is that the interconnect there are four network cards per note are not connected to the CPU but are directly connected to the GPU one per package. So this really starts looking more as a GPU system with the CPU accelerator for running the operating system and sequential parts of code than the CPU system with the GPU accelerator. The third interesting element is the organization of user support on Lumi. So we have centralized virtual help desk run by CSE Finland, but it's also run by a distributed Lumi user support team. So we have like a network of expert dedicated Lumi experts, where each partner country except for Iceland which really has the smallest share contributes one FTE. I mean that I'm paid by Belgium to work for Lumi I'm really paid for project funding so my university pays me but reimburses gets reimbursed by CSE Finland. Besides, first and second line user support, we also provide end user training we maintain the software portfolio, we develop user documentation and so on. The support comes from various channels so Lumi is hosted, not just by Finland but by a consortium of countries and all consortium members are kind of expected to contribute a little bit to the support also mainly the higher level support, like application enabling methodology support so that can come from the local HPC centers that can come from the Euro HPC competence centers, and like all big supercomputer projects of course we also have a team at the manufacturer. We also have an HPE and AMD to assist in porting in advice and so on. Besides that, since the allocation of compute time and accounts is completely done on a national basis. Support of that should also be done on a national basis, simply because in the central system you don't even have access to enough data to help those people so it's really like 11 computers in one one for each participating in the HPC part. And of course as you can imagine Belgium make things even a bit more complicated by having to access portals, and then the Danes decided look we can do even better we can have even more. Now, given that so we have a number of boundary conditions. We have a fairly experimental and inhomogeneous machine with a new interconnect, a new GPU architecture with an immature software ecosystem, and I'm not only talking about developer software but also applications, because it's CMD so it doesn't run CUDA but it could run a whole which is far less capable. It hardly runs OpenACC we only have one compiler that partially supports OpenACC. Instead you should use OpenMP and so on. So video GPUs thrown in the mix and we have a mix of Zen 2 and Zen 3 CPUs to make it complete fun of course those CPUs are very close together with on a machine which that expensive you may want to do a processor specific optimizations when you install software. We've got users that come to Lumi from 11 different channels which does not count sub channels like in Belgium, we have two channels and in Denmark I believe they have like five or six access portals. We've got a small central support team, considering the expected number of projects and users were really only nine FTE with a lot of tasks. Of course the consortium should contribute a little bit but still it's like support wise, we have an extremely small team, especially if countries would start to use their share as a tier two machine rather than a tier zero machine. The Cray programming environment is a key part of our system it stands front and center and it's that environment that we get support for from Cray and HP. Of course it's not only with the Cray compilers we have the new compilers in it also and we have the AMD compilers also in it. There's a little operational issue also they decided that they were too concerned that running the whole software stack from a single file system would create both a point of failure and a performance problem on a machine that size. So they decided to design a machine in such a way that the software is fed from four different copies. So each quarter of the node is connected to a different copy of the software stack, which again creates management problems. So what are we looking for, given all those constraints. We want the framework that allows collaboration and testing software as a regular user because none of us people in lust let alone people from local support from the countries have high elevated privileges. We run as regular users. Yet we want to be assured of course that whatever we prepare can easily be installed centrally and we'll run. We also want an easy way to pass software installation instructions to the user. As I will argue later, we don't want too much in our central software stack, because that creates too many management problems. However, on the other hand, neither us people from lust to use the support team nor the national support teams can write in the project directories of regular users, unless they will decide to add us to their project of course. We also want an easy way for developers of software or application experts for instance in the national support teams to contribute software to the Lumi community. And of course we want the tool with the community for support and continue development. We don't want to do all the development work ourselves. So then of course comes the question should we go for easy build, or should we go for the 900 pound gorilla in the room spec. There is a very, very big argument for spec on Lumi, and it is simply spec is better supported on Cray and AMD. Thanks to Frontier and El Capitan, the big exoskill projects in the US with Frontier having essentially the same hardware as Lumi has only much more of it. And El Capitan is then has the next generation, AMD CPU and GPU. However, there's a lot to say for easy build also. First easy build definitely seems more popular in Lumi consortium countries. And don't forget that Switzerland CSCS is also in the consortium. Second, the whole point of Euro HPC is that they want to develop a European supercomputer ecosystem with lots of technologies developed in Europe both hardware wise and software wise so it will definitely be appreciated that we have chosen for a European product there to work on. Community engagement I think it's also easier with easy build simply because so many people are in more or less the same time zone as we are. There is also the argument that spec may be more a developers tool and easy build may be more a support person tool and having looked at both I kind of have the same feeling I also feel like spec. In the days that I was working on a code in my PhD would have been a wonderful tool to quickly create all possible combinations and try them out. But that's not what you want the user to confront with you don't want to confront the user with 40 variants of the same program. Another argument for me for easy build one that you don't often hear but easy config files offer a very easy mechanism to document modules to add help information to modules. And that for us is the most efficient way to spread information. Of course it looks nicely in a web based documentation, but keeping that up to date and you are editing software installation recipes in one system, editing a website in a different system and you can see that it's not going to happen. Developers are just much more tempted to adapt something if they are if they can do everything in a single file in a single system. And last but not least, I think as long as things are done in easy config files. It's very easy for different teams to work on the same software package in different configurations becomes a bit harder of course when you start doing things in easy blocks, because then you have to merge to different implementations of the easy block back in one. But so easy config files to me are very easy to do a lot of that. Let me quickly tell you about some policies that we have so which software do we decide to install wealth so far we've been working both proactive and reactive proactive we've installed a large set of libraries. I'm sure I'm not using all of them yet but I've based myself on this stuff for instance that CSS to make a large set of libraries already available for people to install software on top of it. And then also we've prepared our easy build recipes for some packages that did prove popular in our service. Currently we don't install them in the central stack. Most of those packages, because we want to test them further. Too many experiences and several of us have had too many bad experience with packages that you compile everything goes well until you start running complicated cases and users start reporting all kinds of problems. Reactive would then be based on tickets and of course there we first let the user install it in their home directory before project directory before we even consider installing it centrally. A few caveats when we talk about conda and likely much of Python and are we will almost force users to do that in containerized builds or similar technologies, simply because we don't want. Many, many users each have 150,000 of small files of a conda installation on our parallel file systems. Since that means I'm really concerned that this may hamper the performance of the parallel file system as it starts to fill up. And it's not like easy built or the highway know, we also consider spec for some software packages, certainly for user installs manual install if it's really needed to find no way out. Second policy is that we go for a fairly limited central stack. And in fact, in many of the tier zero systems in Europe that I've worked on for a praise benchmarking project. I noted that there is not that much software installed centrally. Every one reason is avoid having too many variants and versions of the same package like you have lamps with different sets of plugins that can possibly conflict with each other or then you have a user who says I know this constant in the source file needs to be changed for me obviously if you have too many of those variants users don't know anymore what to take. There's also a management problem, which software should we carry on to the next version of the stack. We collect quite frequent updates from the compilers and definitely that it will be needed to follow some of those updates rather quickly, especially on the GPU nodes because that software is so immature, which means start building a new stack. Probably they are the right moment to update versions but there is no way that we on Lumi can know our users the way that the university to support team can know their users. I mean, when Antwerp, when I thought there's someone in the group who has been there since the beginning, and he kind of knows okay this package that's going to be this group I know this guy has left you can remove that. On Lumi with people coming in from all those different channels we don't much communication with with us. There's no way to know that. Now user installs do lead to duplication, but it also needs to automatic removal. If a project is finished, the project that actually three months later is removed. And so the package is removed, but overall Lumi we build storage, not financially they get an allocation by the hours so there is a slight incentive for users to clean up every now and then. Another thing is that removing a faulty package is hard. There's four copies of it. Users may be using it on a big system it may work for other users. So you would have to add a corrected version next to it and you quickly start to build up a pile of trash that way. So there's a slight management problem since at the moment we have four copies. At the moment we still do not have a dedicated master. There's still system services admins are still figuring out how they will realize that for us, which means that if I know install software it appears first of like one login note, and a quarter of the compute notes, before I can push it or our sync it to the other copies, which is kind of annoying a package may be available for a few hours. But I think it's there already while it's not yet there on most compute notes. And it also eases license management as it becomes the responsibility of the principal investigator and members of the projects. For most software we will actually work with the bring your own license model. We do have a considerable software budget. We ask people what software they want you get such a long list that you can spend basically any budget you can think of on buying software and of course they mentioned the most expensive packages first, because they see that as the way to get those packages. Also more and more codes come with restrictions even for academic use, even if they are technically free. So you can first register as a user before you can use that. So we want to avoid that management burden also for fast, for instance, if you would have centrally one installation of fast with a big group then how are you going to manage that group that has access to fast if people come in from so many different channels. So we organize our software in software stacks. We basically have the cray and software stack which is really just a cray environment as you get it on a typical cray system, except that we install some additional software on top of it, not via the operating system but I push it in via a backdoor by easy build. And then we have our Loomis text, where each one corresponds to a particular release of the programming environment. So we do have to work with the native cray programming environment modules. There's no way to install the cray P through easy build. Of course, we also stick with our four versions because we have Zen two nodes, Zen three nodes Zen two within video GPU. And then Zen three with MI 250X so unfortunately we stuck with 64 sockets of Zen two and then there's 3072 regular CPU sockets of Zen three, and then 2560 of that special purpose version for the MD GPU. We may install some software outside those stacks like ARM Forge will be used both in cray and in all the Loomis stacks so we may unsighted a bit on the side. And in the future, we hope to start later this year with a stack based on the common easy build to change the way they are. The big problem there may be MPI to get it to work with the slingshot cards. And actually with our current slur installation there also seems to be a problem it doesn't seem to work together well very well with open MPI. So now that we have our software stack we can talk about modules. Now you would say strengthening the European ecosystem so you must be using environment modules five. Well, no we are not cray only supports the prehistoric environment modules three, or a very old L not 8.3 version. So we decided to stick with L not because we like the hierarchy cray uses the hierarchy in a very unconventional way but we also prefer writing Lua modules over TCL modules. In fact, so the whole Loomis stack is implemented as a two level hierarchy. There is the Loomis module with the Loomis stack version. And then there are the four partitions that correspond to the four hardware configurations. And then I have three pseudo partitions that I have no time to talk about, but these are for certain special installations. And then on top of that we run easy build in a flat naming scheme. The whole implementation is relocatable so for test purposes and so on we can just put the repository somewhere. Initialize a few environment variables to tell Elmott the root module and to tell Elmott where to find the site package, and we run an independent software installation. The modules are integrated completely transparently in that setup so use all user has to do we set an environment variable that points to the place where they want to install software. We have them to ask the default is actually in their user directory, but better places the project directory which is shared by all users of a project, but we have no way to know the project directory of user, since a user may have multiple project models. The structure of the module three is of course imposed by us, but it mirrors the central stack, but easy build takes care of that. And after that, the user gets an almost unified view of both the centrally installed modules and the user project modules that they have installed themselves so whenever you wrote a Loomis module, you automatically get both the centrally installed modules and the modules that are in that module repository pointed to by the environment We also provide several presentations and more user friendly one with nice labels or just the bear directly as most people are used to. We by default we hide a couple of modules that are irrelevant for all the expert users, but you can unhide them and rather than having to remember all the Elmott variables that can do that I do it via modules. And this is a generic implementation so I have just a single implementation of the Loomis module just a single implementation of partition module, but I link to them in different ways and I use Elmott introspection functions, so that the module can determine what it should do. A single implementation makes it much easier to make consistent corrections on the whole system, rather than having four copies of a module after installation, and having to make the same change without this button is in all four. So now we have the modules we can run easy build on top of it. What we do is actually fix the version of easy build for a given version of the software stack, and each version of the software stack is bootstrapped from scratch so that it can be installed completely independently in the test copy of the Loomis software stack. We stick to a single version because even minor updates of easy build, sometimes introduce compatibility problems with easy configs that you had for or that you have already installed. And if we ever need to quickly reinstall the software stack then that's not the moment that you want to figure out which version of easy build you used for which package. We actually build a module that configures easy build through the range of environment variables that you have available. This is again a single module for the Loomis stack that we link to with different names like easy build production, if you want to install in the central stack and easy build user is the user version. Again, single piece of code is definitely more complex, but it's easier to ensure consistency so if you make a change to something to a location of something in the system part of our easy build configuration. It automatically also appears in the user part while otherwise you make a change to one module, you forget to adapt the other one actually started with two different modules, and I run into exactly that problem. It picks up where to install software simply from its name so whether it's linked with easy build production or easy build user and its location in the module three. So we maintain two repositories Loomis software stack you will recognize the structure of that one because it's very similar to the repository that CSS provides for their system. It also contains both all new code and easy build setup, and our easy config files for what we install centrally, and then we have a second repository Loomis easy build contract, which is for all software that we do not yet want to install centrally, or maybe never want to install century like you have those packages like open form yambo that are quite popular but you always have problems with them. So how do you install grow max in that system set up well at the moment, there is no central installation but it's really easy for the user so you load the Loomis module. 21.08 is the version of the great programming environment that we use at the moment. The partition module will actually be loaded automatically but if you want to override or cross compile, you can also specify it you load the easy build user module, and then all you need to do is point easy build to the configuration file that we have for grow max and dash r for the robot, so that it goes looking not only in the current directory but also in easy build contract and Loomis software stack and that it also installs the dependencies. This one actually has a plume of dependency. When you type those commands you first see you load the modules by easy build module actually print some information about where it will install so that you have an additional control that all your variables are set correctly. Then you run the eb command easy build starts okay I've put a lot of dots there because I don't want to show you all the output and hopefully it will end with build succeeded for two out of two. So module three module avail and what do you see of course you get the crane modules I didn't print them all and then you see your easy build managed user software and their your grow max and plume it module appears right away. And then you have the central stack infrastructure modules I have no time to talk about those but that's a trick I needed to get to make it possible to also install software which is common to all four partitions where we say look it doesn't make you really optimize for the specific CPU. Let's install just a single copy of it, create the target modules that's something very great you set options the architecture options for the compiler by loading a module. You can also do it via command line arguments, but that's the great way. We have our partition modules, we have our software stack modules, then we have those module color module and so on are modules that you use to change the presentation, and then some default initialization. I do use module extensions but not extensively, because the element implementation of prey has no way of hiding them implementation is too old, and because module avail doesn't always show the right ones. If you don't know these are ones that I have generated by hand. I don't enable it in easy built yet. And then of course you have the last page which is like all the help information that element prints at the end. So if you work with all this, well we have our crazy base tool chains, and they are a further development of the ones used at CSCS. We made various improvements like a more elegant way of defining the module file we don't need Lua in our easy config files and so on. We have several compiler, some additional compiler options, better implementation of the AMD one and so on. For our easy config files, because they use different tool chains of course CSCS is our first source over two chains are modified from CSCS but remain compatible. And then we also use the easy builder ones and our experience is that the first easy configs are often very easy to port to the new environment of Craig except if they have all kinds of I mean some, some installations assume that a compiler is called GCC and so on and that can cause some problems. Our biggest problem is actually that easy blocks often don't support Craig. So we follow the CSCS approach, most of the time and use generic easy blocks with more complicated easy config files, simply because adapting the easy blocks can be problematic. We have a maintenance problem there, we can contribute it back to the community of course which I know would be appreciated. But then there is no guarantee that if someone else in the community make changes to them, because the Craig system is not on the list of systems that easy build can easily test it before release that they will remain working. We can keep them for ourselves but then if you update easy block centrally with new elements that may be good for us also, we still need to backport them so we're a little bit, we have a little bit of a maintenance problem there. We do tend to add and that's something that I've already said more documentation to the module files and we do pay a lot of attention to making software more discoverable with module spider and module keyword which are commands that not enough people know. We did consider using Elmwood extension functions for extensions of Python, R and Perl modules but we have too many problems with them, partly because of bugs in this specific version of Elmwood that Craig uses that ancient version partly also because problems that even in the most recent version are not solved yet. We're also not a very great fan of too many small packages. Because you get very long path so that's something that whenever the shell does something. If you're unlucky and it's at the end has to run through the whole path which is again quite a bit of metadata operations. However, we haven't done much about it in Antwerp I actually have combined a lot of basic libraries in a single module, but I have not yet done that in. Because that might be too confusing for many users what I did do though is have a build to modules which bundles all kinds of build tools for more predictable built environment. If you look at the easy build it has nice easy conflicts for things like meson ninja CMake autoconference so on but the make for instance is always the system make. I use a make which is in there also to be guaranteed of versions of practically all built tools I could come up with. The components remain findable. Thanks to manual use of extension function and thanks to the use of a lot this line in which I also put them. So with the extension function you can find them with modules fighter with the what this line module keyword works also on that. We do all development in user mode so using our easy build user module, the way a regular user would, and then import them to the contributed or main software stack repositories. Like me and one of my colleagues have a private setup of the full software stack for development and testing before we even try to roll out on the system, but we're still pretty much working on our test setup. So what do I see as upcoming challenges for us in the next year. First AMD GPUs, they have an immature development tools immature applications. So actually, for information about how cray plants to do things we essentially had to look elsewhere to learn about the plans of cray for AMD support. So we had to look at the early access platform for frontier which we could go published its documentation. So really only know for a week now that we for instance have an aiming conflict in over easy build tool changes we will have to change something there. We will still open up in the air how installation procedures of applications will support the MD so how support will appear in auto tools in CMake and so on. Another big issue is what we've sickle sickle is not officially plan part of the plans of cray or AMD. We have a lot of push hip, which is a coulda clone from AMD and open MP for programming AMD GPUs, but you do see that sickle might become quite popular with a number of packages and like the grown backs people are working on a sickle code path that would replace all code paths except their code path. Of course they will still write optimized kernels for different GPU architectures and so on but they will do so in sickle, except for the Nvidia GPUs. But that's currently for us a third party tool so it remains to be seen how this will integrate. And for instance how this will also integrate with pray MPI, or whether we will have to do something different there. And then the other issue is we're only a small team so what with those packages that require a lot of development, like AI tools and I don't think I have to tell the easy build community how hard it is to write a working easy block for TensorFlow. Or bioinformatics simply because they have a zoo of small tools typically with broken installation procedures that assume that each compiler is called GCC that minus MRT was native is a good option, wherever you run so break cross compiling and so on. So these are the three main three main challenges that I see for us in the next year. So I'm already way over time so rather than showing more details about how we do things I would like to conclude so so far I think easy build deployment has been quite successful for me, but there are as I showed in my previous slide several challenges ahead. Of course it is a very a typical setup, because a crazy system is just different in easy build. You work on top of an already installed programming environment rather than managing everything to easy build you have different tool chains, even though you are using the GCC compiler you're using a different tool chain for that. I was very skeptical about my idea of not installing too much software centrally but instead making it very use very friendly to users to install software themselves, but I must say in the first few months of LUMIDIS has worked very well for us. There are several successful installations by users, simply based on build recipes that we provide to them and tell them look you have to run this and this command. And I mean so far we've got several people for instance, who have installed VASP already that way it just works for us. And of course Elmot is also an important building block of our setup. So, when you get the handout of the slides, there will be links in there to our repositories in the LUMID software stack repository there is a lot of documentation about how we did things and many of our easy config files are also documented to have a file in each directory that tells you why we did things certain ways. We may even add some documentation later on how to run on LUMID. We're thinking of doing it in that repository again so that the developers can do it and then pulling it automatically to the regular documentation but that's working progress. And then this is the kind of mandatory slide with our sponsors, mainly mentioned in Europe, and then the region of in which Kayani the data center where the supercomputer is located resides also seems to be funding quite a bit so that's the one at the bottom right. And of course I have to show a picture of the machine to conclude, if not only the CPU section of the machine and the storage build up so that's what you see in the left picture, and it's actually built in an old factory hall from a paper mill. So they build the data center inside the hall, which is that structure which is meant to be by the architect to resemble a snowflake. So inside that structure that you see in the bottom and you see the computer which is in the left part of the picture. So thank you for your attention. Hi, thank you Kurt for the talk. If anybody has a question, if you can, and you're in zoom can you raise your hand and we'll ask you to unmute. Okay, thank you Kurt. One question you mentioned that you're, you're heavily customizing easy blocks and that creates a bit of a maintenance nightmare. One other way forward could be to use hooks a bit more, where you take full control over any customizations that you need to do for for me. So you can also check things to make sure that some things are not being done, for example, by changes in the easy blocks. Have you considered that is that a good way forward. Well, first we're not really heavily customizing easy blocks we're avoiding customizing them by writing more complicated easy config files. As C S C S many of our packages are installed with the C make make or the configure make easy block, the generic ones rather than using the specific easy blocks for those packages. I did configure hooks for easy build but I'm not a big fan of doing too much with hooks. Before you know you get a very complicated implementation and of course you can try to split it in different files with Python and so on but otherwise you really. I mean, as you saw one of the big advantages of easy build for me over spec is that you have those separate files and different users can work on different configurations of a program without merge conflicts. Easy block you already create more merge conflicts if you go to a hooks file you create even more merge conflicts. So that's why I'm not that much fan of hooks. So I do use them for some things like I automatically aligned to which module about where to find the easy build support. But I'm not a great fan of using them to make too many customizations to software installations. Also it becomes really hard to figure out where something is done. Usually you first see the easy config but then you have to think I also have to look in the easy block and then even further and it's not even documented in the easy block you have to look in the hook. Or even other easy blocks because they depend on each other. Do you think there's any any generic easy blocks were missing centrally that would help. So far I've been able to fix everything with easy blocks but a more elegant one to really just have kind of a script running but nicely split up in a configure phase in a build phase and in a install phase rather than you can do it I think with binary or someone you can fix it and you can give an install command where you can really type the whole command but having that more split up in several phases may make it a little bit easier and they make it a little bit closer to the easy build philosophy. And then I'm a favor of having a very powerful bundle easy block so I fully support that there were some new features added to it and maybe it might even need a post install command for instance for each of the bundle components. Yeah, okay. Yeah, I'll open the floor for some other questions. I was crossed about next. You're muted or your microphone is off I don't hear Bart. It was the phone of my microphone probably. So the same thing with zoom puts the volume to 15%. Yeah, just have a question about the real real color. Reload capability of the of the module tree does that just apply to the modules themselves also to the software because I didn't mention for the software would be very hard once you've installed the software you lose that because it puts the path to its install directly everywhere. Yeah. There's also a lot of hard coded paths inside the software so made to create the ball would be in a humongous efforts. Yeah, I don't even want to try that but it's really like you can just you have to recompile everything but we just would actually actually experimenting with easy stack files, but I still need to file a couple of bug reports on them. Okay. So, so the flexibility is the stack files at the moment. So the flexibility is just on the module trees like where the Lua files lives, etc. It's really just in the start I mean as soon as you install modules with easy build easy build also puts full paths in there. But it's like okay if you want to start you want to experiment you copy your repositories there, and you can of course you have to start you have to run a script to regenerate the directory structure then you run another script to initialize a specific version of the tool chain, and you can just start using easy build. Okay, thanks. In that repository. So I have I have like three bash functions for the three different repositories that I manage that way. Next question question from Chris Chang. Yeah hi. Can you hear me. Yes. You mentioned one of your responsibilities is writing up user documentation. Do you have a process for deciding what to write up because, you know, theoretically you can spend 24 hours a day writing documentation for users what do you emphasize. I know there are some people in our team who are way too ambitious in that respect. We don't I mean we're pretty unexperienced team actually people coming from line different nations, just being assembled a year ago so it's like, okay we don't have third procedures yet. So people want the documentation on every software package that we had, but I said, look if you want this, I'm not going to write that because that's going to take too much work. But it's like it may converge towards if you have to answer three or four times the same ticket, then maybe it becomes time to think about making it clear in your documentation. Okay, yeah we have the same problem and that's about our approach to so good to know. Thanks. Okay. When they're still open, I think. Yeah. The modules that you have. Does module spider work as intended for you. I know that PDC is currently having problems with their stack and they are based on the same crepey. And the mod versions. It works largely but I have made a few mistakes during the course of development of the modules that made module spider fail in some cases. Like last week I fixed another bug that I had and that in some cases actually module spider fail to show which other modules you had to load. In general, it seems to work quite well, except with that's not just a module spider problem problem that's also what you'll avail problem. Every now and then it looks like a lot of screws up its internal data structures. Yeah, but you're not using a spider cash right because the cray delivered a mod doesn't have that enabled. I have not tried to install a central spider cash yet, but if you look carefully in your direction and L L L M L D no dot L L D slash dot cash. It actually built something which looks a hell of a lot like a spider cash. Yeah, perhaps. I haven't thought about it yet. But it's on user level, and so if you have problems with... So it's like erase that file and let it build it up again. Yeah, I mean for the central... But I haven't tried the central one yet. It's on my to-do list, but... Yeah, because Henrik at PDC was asking me questions about how to... With module... Showing modules that shouldn't be available because they haven't loaded the PDC corresponding... I haven't had that with modules, but I do have that problem with extensions. One of the reasons why I do not want that extension list that you... No build tools was available, so it should have shown the extensions, but quite often I have that extension list, while the modules of which they are an extension are not yet available. So module avail doesn't seem to work for extensions. And that's not just a problem of Cray-L not that seems to be a problem of many newer versions also. I haven't tried with 8.6 yet, but I had it with various 8.5 releases also. Yeah, but this was not extensions, this was just pure standard modules. Yeah, that I haven't had yet. Yeah, I'm going to take a look at their system later. Of course they may have copied the bug from me that I have corrected since then. Yeah, since they based it on what you did. Yeah, they based it from my code I know. Yeah, I'll take a look later. Thanks. Does anybody else have a question at this point? Any questions and slack work raising here? There was a bit of chatter, maybe in Zoom. There's a hook remark. Oh, there is that remark about it's never too early to let the kids start working with EasyBuild. Well, in Leuven, this is the case. Maybe I should post some photos later on in the workshop, but at several schools in Leuven, they painted three blocks on the road that look a little bit like the EasyBuild blocks, except that the colors are in a different order. It's so fun to see the first time I saw that was my reaction was able to make advertisements for EasyBuild here at the school. But it seems to be something typical for Leuven and they have two versions of it. They have the full version, but nowadays they stay from paint and they just draw the circumference of the cube. There's no additional questions. We should wrap it up here, Simon, and switch the stream.