 Welcome to the seventh EasyBuild Tech Talk, in this we will be having two of the teams from the Student Cluster Competition at ISC 22 will present about their experiences and to give us some information and some time for questions and answers. The first of these teams is not only Flops, who will be talking about the unconventional cluster of risk following developer boards and their experiences with them. So over to the next team. Okay, thank you Simon, thank you for the introduction and also thank you to all the organizers for inviting us. So we are, Martin and myself are two of the advisors of this team. We are here in the back in case of questions, but the real important message would be given by the students that are here. So it's their moment. So I just wanted to thank you for inviting us and I leave the world to them, we stay here in case of nasty questions and be nice please. Thank you. Okay, so thank you everyone for being here. I'm Chabi and I'm going to explain a little bit about about what is the Student Cluster Competition and what we did there. So what is the Student Cluster Competition? Firstly, there are two big conferences around the world of high performance computing every year. And the main two are not the main two, but there is one in Europe, which is the ISC, which takes parts between May and June. And the other one which takes part in the United States and it's in November. So each conference hosts the Student Cluster Competition and you can think about the conference when people go there to give talks and all that stuff, but there is also an exhibition where companies have their own booth and they show their technologies and they do also talk. So now us, the students in this competition, we have our own booth and as you can see in these images, they're from the previous competitions. So how does the Student Cluster Competition work? Basically, you have to take in mind that these were the rules before COVID-19 and now have changed a little bit. So the rules were that there were 12 teams of 600 graduate students plus the advisors and each team has to bring its own cluster. Also, each cluster has a power budget of 3 kilowatts, so you cannot exceed this power. So as you can see in these images, there is these red lines here and if some teams with each cluster pass this red line, you get disqualified by some points. Also, there are three applications where you have to compile them, run them and do also some benchmarks and these are some real scientific applications. Plus, on the last day, there is a circuit application which we will talk about it later. There are also three awards and one is for the Highless HPL, which takes LIMPAC, the first place winner and some favorite one. Now, about our Student Cluster Competition, we were between COVID times, so we had two parts, the online and the on-site part. The online one took part between March to May and the Council gave us access to three supercomputers around the world, which we will talk about in each of our categories later. And also, we were asked to also run and compile and do some tasks that were provided by the jury to three real scientific applications, which I will talk later. And on the on-site part, we brought our cluster there and there were the same rules as mentioned on the last slide. Also, the main difference could be that instead of being 12 teams, this year we were five teams of 600 graduate students plus the advisors. As I just stated, there were three applications. The first one was NWU Chem, which is a computational chemistry package. You can think about it as an excel of computational chemistry. It is written in FORTRAN and it uses OpenMP and MPI with global arrays. It also has support for multi-core CPUs and GPUs and its open source. It was created in the 90s and accuracy could be that this application is older than the MPI and even older than us. And this application is still maintained by some laboratories and some developers. The other application was ICON, which means Icosahedral Non-Isostatic Earth System model and it's a weather prediction and climate modeling system. It's used in the German Weather Service and is written in FORTRAN. It also uses MPI and OpenMP and there is a working progress that it could be ported to GPUs. The last application was X-Compact 3D and it's a solver dedicated to the study of turbulent flows. It's also written in FORTRAN and it was initially designed in France in the mid 90s. It was supposed to be used in serial processors but later it was converted for HPC systems and it also uses MPI. And now I'm going to leave the world to my colleague Enric, who is going to explain a little bit about the team. Thank you, Tavi. So the Not Only Flops Team has been active since 2015 when it was created and started participating in student class head competition. Since then there have been eight different teams and every year there's a new team that is formed and it's formed through a selection process that the advisors that are from the Barcelona Supercomputing Center prepare. We are not afraid to use near or emerging or unexpected technologies because we think that is more about learning and having fun than the flops we get. So that's exactly why the team is called Not Only Flops. We don't like making up excuses and leaving a challenge unaccepted so being online or on site we work really hard on each part. Also we think that this is not an everything member on his own type of competition so we collaborate with each other and every time the skill of another member can be useful we use that to be more efficient. And finally we think that the SCC is the door to the HPC world. Now it's time for you to meet the team. We are Tavi, Michal, Roque, Eric, Marc, and me, Enric. We have all just finished our third year at Universitat Politécnica de Catalunya. Two of us are studying computer science, another two computer engineering, and the other two software engineering. The place in the Facultat Informatica de Barcelona is actually very close to the Barcelona Supercomputing Center. That's the place where our advisors spend most of their time. Because all of this could not have been possible without the help of our advisors who are Marta, Guillem, Filippo, Kilian, and Fabio. They are all researchers at the Barcelona Supercomputing Center and have been and still are a huge help through all the learning process. And now my friend Michal will present you the adventure with our RISC-5 cluster. So firstly I'm going to introduce what is RISC-5 itself. For those of you who don't know, RISC-5 is an emerging new technology. It's an open source ISA and we think it's the perfect playground for research. Because not only it means that this community to develop and also that companies can take part in its development actively, like Sci-Fi for StarFive. But it is also a modular ISA. So this enables to have RISC-5 machines from embedded systems to high-performance CPUs and also specific CPUs like GPUs or other accelerators can be made using this architecture. Also like one thing we're proud is that our Barcelona Supercomputing Center is working on different research projects to further develop this architecture and we think they'll see light in a few in a few time. About the cluster itself we cannot talk about it without having in mind the adventure we had trying to get a nice cluster. Because in March 21st we contacted the RISC-5 International which is the the organizational who standardizes the RISC-5 development and we asked them for help in finding a cluster. So they put us in contact with Shanghai StarFive team and we tried to agree to have a cluster from them but a week later on March 28th, Shanghai went on lockdown due to COVID-19 resurgence. So logistics from and to Shanghai became challenging and it was completely unfeasible to have the cluster for the competition. So two months prior to the competition we were left without a cluster to go in. Fortunately a month later during the RISC-5 week at Paris we got the chance to meet other RISC-5 enthusiasts and during the event we talked to E4 and to the Universitat di Bologna and we agreed to collaborate in order to have a cluster for our not only club team. So finally we we get to the competition site on May 30th and basically we have a cluster which is a compound of two parts one from Bologna and one from the Barcelona Supercomputing Center which we will have to set up in order to work. So our cluster had some limitations from the start because even before starting it, powering on our cluster we know we knew that it would be slower comparing to other Intel machines because in other words a floating point peak performance is 16 times lower because of the design of the architecture and also it had limitations in network because only a gigabit other network type was supported. Infiniband is still not supported yet or at least in our models so we we couldn't get it worked any faster than that and also the maximum power that it used were 300 watts that's a 10% of the power because we couldn't get more boards available for our cluster. If we had had the chance to to get more boards we could have had more power computational power and that was a big handicap to begin with but nevertheless we got a cluster which was fully functional and working. The cluster was based on seven boards from the Barcelona Supercomputing Center and six boards from the Universitat de Bologna. The six ones inside the racks like you can see the picture on the left where the ones from Italy and the top of the rack ones where the ones landed from Barcelona Supercomputing Center and well once we got there we had to set it up for the first time ever make all the boards work together and we successfully achieved that and also we we had to state that our cluster it really uses ARM because our Raspberry Pi was acting as a DHCP server to isolate an internal network for all the boards so that's that's basically it and now we are going to talk about the preparation at home in order to reach the beyond sci-fi. Now I'm going to talk about the six months before going to traveling to Hamburg and the first stage of these months we got an introduction to high-performance computing because we all come from university and we only have one or two subjects about parallel programming and architectures so we didn't know how to submit jobs on a big cluster like the ones we are going to use in this competition. The first what the first we did was to submit the typical benchmarks like HPCG or LIMPAC in machines from Barcelona Supercomputing Center then when the judges let us the applications we started to compile and study them in order to get the best configuration to have the best performance on also a computer from Barcelona Supercomputing Center. Finally when the online competition started the judges let us access to the sub-supercomputer from Pitchwork Supercomputing Center and University of Toronto and we also started to compile and run and get the outputs in those machines. Then when we got all the outputs we submitted the results and finally we you started using RISC-5 clusters such as the words we took to Hamburg which was another cluster from BSC called the RISC-5 with the same words that we use in Hamburg. Now my colleague Rick will talk about the adventure on site. Thanks Marc. Now I'm going to talk about our experience in Hamburg. The first day the first day we had to take the flight to Hamburg and our advisors made us to put in our brackets part of the cluster that we were going to take to the competition. Once there we went to the exhibition and got the rest of the notes that lend us the University of Bologna to start mounting the cluster. This day was totally dedicated to the hardware part of the cluster. On the second day we installed all the system software configuring a slur installing on the benchmarks downloading the application source code and the dependencies. This was the first day that we worked with RISC-5 cluster so it was a challenging day. The next day was the benchmarks day so we had to execute each PCG, HPL and HPCC. We executed them with a configuration that allowed us to get the results on time. Also, this day we started compiling the applications for the next day, its compact icon and WKM and at that moment we realized that the compilation took a long time. That was going to be one more problem since it limited us even more the time and the number of different configurations that we could carry out. The first day was the applications day so once we had the competition input we launched an execution to estimate the execution time of an iteration. Then we adjusted the number of additions of the input to be able to finish the execution and had valid results within the time limit. All the problems and how we solved them will be explained later. The last day we had to compile and execute the secret application which was 4.3D which we didn't know anything about until that day. The work plan was similar to the other applications. It's one of the groups compiled with different configurations and once compiling we adjusted the iterations to the available time. After delivering the results we amounted the cluster and we went to the award ceremony. So as you can see in the image during the ceremony we recite the Framefabric award. For all the work we do and for doing something unusual and innovative. Now my colleagues will explain how the process was and the problems we had with each application. Okay so first we're going to talk about X-compact 3D and this was the easiest one application because it doesn't have any dependencies so we only had to load the corresponding modules that you can see that are GCC, OpenMPI and Compiling. We also managed both to run it in SQL node and multiple nodes. The only thing that we had to modify was the size of the input given by the judges because our cluster was slower than the rest so we decreased the from 10,000 iterations to 100 iterations. Okay and for the icon application we could successfully compile it but we realized that when using the GCC version the same as the X-compact 3D that's the 10.3.0 and the OpenMPI 4.1.1 it would give us a crash but we didn't know like we didn't have enough time to figure that out so what we did was to switch to mBeach for the MPI wrapper and then it compiled without much problem. This application had dependencies more precisely HDF5 and NetCDF both the C and the 4 conversions and another dependency it had was UCX for Infiniband but there's no support for UCX in RISC 5 here and when we tried to compile it from ourselves from the source code we got an error saying that RISC 5 architecture was not supported yet and we are not aware like if someone has successfully managed to do it yet or not so what we did was to move forward without this pregnancy and then the execution in a single node worked finally both in a serial and using multiple core executions but then when trying to execute in a multi-node execution it would press using for a segmentation fall and because of time limitations we couldn't figure out like what would be the cause of it because as my colleague said previously like the compilation and the execution took much longer than when comparing to other machines so we also had to to reduce the simulation input like the original one was for one year and our run was for one day to give a number of how much difference in computational power there are. How well then though you can we could compile the application but not of the first the first time so in the first time we tried to compile it with the GCC the open MPI and open BLAS but we couldn't get to run this application we don't know why but we were lucky because we had the main developer there which is what we is Jeff Hamon and with the main developer of this application he told us to change some flags and also to change the open MPI to empty it so once we we changed this stuff we could manage the application to compile and to run in one mode but when we tried to run this application in multi-node it crashed and it was because of an issue with a model so we talked another time with Jeff and he told us that maybe this issue was not related to our configuration but we don't know yet as this application was never supposed was never tried to run in a RISC-based cluster so maybe in a few times and this application will run smoothly there and the last application was the secret application which is for 3D which is a uh sorry a Fortran based application that simulates the passive transport of volcanic particles aerosols and radiation we managed to compile it with GCC and open MPI and we also had to set up some dependencies which were HDF5 and next DF but we got it to execute it in single-node and multi-node in our first try we had to reduce the input from 55 hours to one hour but a fun fact about this application is as you said in one of your previous presentation in one given by Kenneth the validation test were not maintained so the all the tests failed for our cluster setup but the application actually was working successfully. Okay so now I'm going to talk about what we have to learn through the online competition and the on-site competition because we in the on-site competition we brought a RISC-5 cluster so like the rest of the teams we had to set up the cluster and make it run the day before the competition started this is why we had a limited time to do it but we also had to pack the cluster in our mileage because we had a part of it in the BSC bringing motor boards and parts supplies in the suitcases was a little bit suspicious but we didn't get arrested in the airport security control we also learned how to compile applications in Intel and AMD-based clusters because in the online competition the clusters that were given to us were of this kind but we also learned how to compile these applications in a RISC-5 cluster that is an uncommon architecture so now I'm going to talk a little bit about what we learned compiling and running these applications in the online competition we could follow the guides provided by the judges and make and compile the applications and run it and run them without much problems we had only we only had several problems with ICON but in the on-site competition we had to find alternative configurations since the guides provided to us were thought of for Intel and AMD-based clusters for example in mbcham the default communication between nodes was set to infinity band and we had a 1 gigabit ethernet interconnection another thing that we had to do in the competition was to measure the power drain but well since our cluster only consumed 300 watts we we didn't bother about that so we had to focus on measuring the compilation and execution time since the time in the on-site competition was very input and then we also well since we were bringing a cluster based on a common architecture we had we liked to to build all the dependencies and the software ourselves for avoiding any problems in the future that in a common base cluster you can directly install the the software and you don't expect any any problems but since we had a common architecture we wanted to to to know what flags and just in the in the compilation of that software uh so that has been a little bit a summary of our experience so now if you have any questions we will be glad to answer you okay thank you very much this was really interesting I can definitely come up with several questions but let's see if anyone else has a has a question here in the in the zoom session if you do please raise your hands I will let you raise the question or if you're watching through the youtube live stream maybe post it in the easy build slack and let us know if you have any questions well people prepare their questions or think about a question I have one or two so you mentioned you have quite a bit of issues compiling because the compiling directly on the risk five boards is quite slow they don't have a lot of core don't have a lot of memory so maybe maybe even ran into memory issues as well at some point did you consider cross compiling so having a more powerful intel server for example and then cross compiling for risk five or compiling in an emulated environment on an intel system because that's probably a lot faster than building directly on the risk five port so basically with our resources we thought adding a cross compiling option would be like a layer of complexity and we didn't have like the resources or the time necessary to implement that in that moment that's why we did it all in in the risk five cluster itself yeah and we also only tried the risk five cluster in the outside competition so before we didn't have any access to the cluster so we couldn't try that yeah okay I guess you only learned how painful it was to compile software from source during the actual competition yeah it was already too late it was already too late to then yeah okay and another question I had that's maybe a more a more general thing but you you described the application quite well and the dependencies so it's mostly compiler, MPI and then some pretty standard libraries like hdf5 and netcdf is that typical for this type of competition because in in my experience a lot of applications in hdf5 are way more complex than that have more dependencies or dependencies that you only see for some applications and nowhere else so something something common as net as netcdf netcdf and hdf5 is maybe easy now it's probably not easy on the risk five but in general maybe there's a reason they do this as well for the competition otherwise it would be too extreme or too difficult but is this typical? Well maybe I can take this question because I had previous experiences I think that's the the idea that guides the judges or the organizers or the competition is that usually they take something that has a community driven HPC application right the where open form want to express so this kind of say big scientific applications and with whatever they came from right so you're right can happen and it happened in the past to have applications that were like a little bit more hard to have up and running so so but the basic idea is that you have real science there and yeah sometimes when it is too hard the judges they know and they try to at least give you an input that do not use all the models that are safe that we have this one and so usually they are they tend to be nice but but yeah in other years we had problems in finding for example I remember one here that we were going with an arm system and they gave us an application with Intel in 3.6 so whenever you find that I mean it's not a fact that you can change MPIA labor and you go on right you really have to record it or understand what to do so yeah it can happen you're right and so the applications except for the surprise one how much time did you have to prepare to experiment with the applications building them maybe preparing scripts to build them more easily was that something that was allowed stuff that could you could reuse during the competition yeah yeah this is allowed yeah usually you get the application a couple of months before and and then yeah we documented in a wiki in case of somebody forget about it one when when when the time puzzle goes by right but then you can also script it and yeah this is all allowed and during the competition you're you're then supposed to start from an empty system you have to image the nodes and then pull in those scripts download the source code and basically start from scratch yeah okay yeah that's a big challenge okay um there's another question coming in from someone else so both application two and three were built using mpich but only application three run fine on multi-node well application two failed because of you you see exist is that correct and maybe you can say something more about it why did it work for the third application and not for the second one you have any idea so if we had had more time I think we we would have got to the real issue on why that happened but still like I think like there are so many variables in it that we cannot assure it was for one single thing maybe it was like a accumulation of fights that that made the the thing that didn't work but we we cannot assure single things and so the the second application failed with the sec fault right do you think that could be due to the amount of memory that's available on the boards and then maybe big stuff being created in memory and it just assumes that enough memory will be available and then goes down hard if that's not the case you wouldn't always yeah you would expect a nicer error message but we see sec faults a lot when just yeah memory like it cannot be well it could be perfectly possible like we we do not know and we we should trace a bit more the or and the error to to get a proper understanding of it also I mean as a map but if it would have been in say more time in preparation for the respite with the respite class that we could have like take the application compile it and run it with the smaller inputs or with tests and try to try to move on from there but in this case was everything really quick quick to do I mean we didn't have time to to start with the basic test or the small inputs and then moving to complex runs right so that that basically limited a little bit our yeah level of understanding how much sleep did you have during the competition how much sleep did you have the students during the competition did you lose sleep all trying to fix or did you say yeah you know I mean you are in a conference and and it's I think at me that they close it so you have to get out and they kick you out right so and you you continue reasoning about it but at some point you sleep or cry or cry or cry and sleep something like this okay there's there's also a well it's maybe not a question more of a comment by Alan that Fortran seems to be a little bit over represented at least in this set of applications and actually for risk five that's maybe a downside because we know there's a lot of ongoing development in LVM for example for risk five but there's also a lot of development for Fortran it's not really mature yet so I guess that that could have been an interesting direction or maybe it is for next year if there's a more mature LVM based tool chain and you decide to continue this adventure with risk five that could be interesting oh yeah and the question by Alan as well what was the experience like as students did you learn a lot and do you love or hate HPC now well it was hard at first because we started here in December or January more or less and we barely know anything so it was a big challenge for us for example personally we don't know what the module was so now I know what it is so that's the level so now we know how to maybe build some applications from scratch how to run it and maybe to extract some information to to increase the performance and that's basically what I'm and I know about my colleagues yeah for me I didn't know much about HPC so yeah it was the first experience with it and and I liked so I think that that summarizes it yeah okay sounds good I'll I'll pass the word to Simon sorry sorry I'll pass the word to Simon here to see if we can make the switch over to the other team unless there's any other questions or any final words from from you guys thanks a lot that I think this was a very good presentation very interesting to get an insight on how what it's like to join a competition like this yeah and I hope this was fun it was recorded as well so and maybe this was a bit better than the the let's say interesting interview you had during yourself yeah all right thank you very much thank you okay back to Simon we'll we'll switch over to the other team to hear their experience as well I'll be right back but I think Simon has things under control I think we are still missing a member who was supposed to give the general introduction uh there he is all right yeah sorry I will so yeah the first thing thanks for having us here today and just give me one moment I'll introduce you as a team quickly first and then I'll pass across to you okay yes so our second talk today or second discussion is with the team Ratlet and they used easy build to install software for their more traditional CPU GPU cluster so I'll pass over to the team and they can introduce themselves so hi we are Tim Ratlet so yeah we are a student HPC let's say cluster competition team I think our the guys from not only clubs explain pretty extensively what what we do basically so we take part in these competitions like the ISC SC and also ASC in Asia online and on site depending on the situation so yeah today we have here most of the members who took part at this IC competition we are missing two which could not be present the first who we are missing is Rahul our team captain and Piotr then today we have here me I'm Nejo Marceau uh there's also Faber and Niklas for this competition we had a more traditional uh cluster configuration we had four nodes and we choose to have two GPU nodes and not to use like GPU some of the nodes it was more because of the power consumption since we were limited to three kilowatts we had to make some choice and cut some computing power there uh the CPUs we had were the 7742s from AMD and as for GPUs we had the V100s we had some issues getting the A100s in time so that's maybe the first story of our experience at IC we didn't get our GPUs in time which maybe penalized us a little bit but I think the judges actually took that into consideration and I would say that's very typical from the true HPC experience nowadays as well yeah with the all of the chip shortages it's pretty hard to get some new hardware right now so yeah we got our cluster from our sponsors which is kept at the CSCS facility they are probably our biggest supporters besides also our advisors Torsten and Hussein and as you mentioned we were the only team at the competition who actually used EasyBuild to let's say build our software it was a choice led by let's say the philosophy of the team of learning new stuff and it was suggested to us to try this new software by some people at the CSCS and let's just say that we took the chance to try it and had a great experience with it so but that's something the other guys are going to talk about for each application how was the experience with EasyBuild and I'll pass my word to no father sorry yeah right so I also mentioned because I think that's kind of interesting for the online part we had access to several clusters I think it was three it was um bridges two from psc uh niagara from signet in canada and then also the thor cluster from the hbc advisory council itself and for some of the applications on these clusters we actually also ended up using EasyBuild instead of the built-in packages that was pretty fun um the the nice thing there was that essentially once we had set up uh the easy configs for our software we could just copy it over and then if EasyBuild was set up correctly and it was running properly on that cluster then we could just compile the packages and essentially have have the same setup on every cluster to start out with with experimenting with different options yeah and this usually because the the home space is restricted or at least the number of files in the home directory is restricted in in clusters like this it was kind of fun because we did some very non-standard stuff like uh um running EasyBuild mainly on scratch so changing all the directories to have EasyBuild on scratch and stuff like that um but especially given the short time frame um we we ended up being a bit late with with compilation and uh as long as scratch stays persistent for a week you can do that uh mostly without any issues um we kind of approached it in a way that we were kind of learning by doing um so essentially learning about EasyBuild as we were trying to actually uh implement these things um it's the first time that we used EasyBuild so essentially our advisor mentioned to us last summer um that EasyBuild would be something good to look into uh we were using SPAC before that um and some of the team members were still we're still sticking with that um but over time within over the year we slowly kind of rolled over to uh EasyBuild so personally I actually have no experience at all with SPAC I started out using EasyBuild right away um but yeah like I said it's it's a fairly steep learning curve and just learning things as you start meeting them so that was kind of fun um yeah I didn't mention it I was the one who mostly compiled so I mostly compiled ICANN alone um with somehow from the team captain Rahul who is not here unfortunately um but um yeah the other the other interesting part was that uh essentially just a few days before the competition we also decided to reset the cluster and during that process uh EasyBuild was of course very helpful to rebuild the entire stack um get immediately the the compilation or the the builds that we wanted to have with the correct build options um and also on the other hand at that point since we had already used EasyBuild on the on the external clusters we already had at least some experience with it and could fix most of the uh errors and gotchas that we would encounter um for I'll switch over to ICANN um what I did there uh mostly I did some exploratory compiling with EasyBuild um sorry without EasyBuild so I would get the source um start compiling packages figure out exactly what configure option configuration options I want and things like that we've already mentioned uh CDF and net CDF and HDF 5 um and getting those to work uh with all of the parallelized options um that you need for ICANN uh can be a bit finicky so that's why I was doing this kind of compiling by hand and then writing the the easy conflicts um as I figure out what exactly I need and want for each package um so maybe I'll throw in there that that's kind of a point that for the future maybe we can find a better way to do that or maybe that's that's an idea for for you there's a better way to do this with EasyBuild but it seems like the the source in EasyBuild itself is always kind of hidden it's very far into this kind of tree uh folders directory structure um and it's very difficult to at least with this approach make changes for debugging to any files in the source without just cloning the source for yourself uh and working with that before kind of diving into EasyBuild um but yeah in the end this no matter how that worked in the end this gave us the custom easy conflicts and I could just copy that to all the clusters including our own and then just once it was set up it really worked like a charm so that was amazing yeah yeah maybe I can already answer answer that specific part um um I'm not going to claim we have a perfect solution for that but there are some things that that could help um like one thing you could do is let EasyBuild essentially set up the build directory for you and then let it stop right before it starts configuring the build or or doing the actual build step and continuing so you can do there's a dash dash stop option right and and if you stop dash dash stop prepare um essentially EasyBuild will set up the build directory unpack maybe apply patch files everything set up the environment and then stop and then if you then manually go to that build directory EasyBuild has everything prepared for you there's one other thing you can do next to that and if you combine both I think it's close to what you need um you can let EasyBuild dump a script that sets up the build environment and if you then source that script you get all the environment variables like EasyBuild will set them as well so if you combine the uh the stop option that sets up the build directory for you and then also sourcing of that script you're essentially in the same environment as EasyBuild will do the build as well and in there you can play it out manually configure reconfigure start from scratch again um and then gradually figure it out what you what you should put into the easy config file now this is yeah this is this maybe something you didn't know maybe it's something you don't have good documentation on either so that that could definitely be helpful I think there are better ways we've discussed the idea of dropping you into an interactive shell environment so just let EasyBuild prepare everything and then then drop you right into that environment in the terminal and I think SPAC has something like this so that's definitely an idea we could work on as well to make that uh interactive experience a bit a bit better yeah that's very good feedback I'm not sure if that's yeah I'm not sure if I have a good solution to this um the first option sounds interesting with the stopping um the environment but I guess in that case every time you rebuild the the whole thing gets recreated right unless unless you restart from that point yeah you could just let EasyBuild prepare the thing for you and then you copy that as you can easily restore it later and reconfigure by hand that's that's a good point to go forward with yeah okay cool um yeah but yeah again as soon as that stuff works it's great because you just have a pre-built thing as long as EasyBuild works and your easy configs are correct and the source code loads properly which is also not always the case sometimes there's problems with retrieving the right source code and things like that but I think it's not really on the side of EasyBuild that's often with uh with the software um packages or providers um yeah there were a lot of interesting moments where I found myself uh because of time uh essentially editing files that I usually shouldn't be editing and just having kind of very dirty fixes uh but yeah again not really not really a thing with EasyBuild but just in general with the the whole um competition so that was kind of fun and and like I mentioned during the the the previous theme as well I think you got the easy experience in terms of dependencies and all the kind of trouble you could run into of course yeah it's this is just the starting point of what what support teams and people who are building applications on the daily basis uh run into every day yeah as soon as you don't want to use the the newest stable uh release of something you want to use something older or something more obscure it gets very interesting um yeah let's see uh yeah I mean that that already more or less finishes what I had to say about econ um yeah in the end so having having everything then on different clusters having seen how that worked just kind of merging it all onto the own cluster and at that point I really understood how to essentially modify easy conflicts how they how they uh play into each other um and especially because it was a correct install of EasyBuild and not something running as a user uh in scratch uh everything then really kind of fit together and was very helpful for the final builds you you mentioned you also played around with spec for a while during the preparation and and um I didn't other team members did some some people did yeah so what was it like a team decision in the sense that let's pick something and stick to that everyone so we can help each other out as well and was there how was that decided was a democratic vote or did you just have a discussion on these features are interesting in EasyBuild and let's go with that or I think it was a very strong recommendation from advisor maybe Nicholas can can also talk about that but uh yes yes sure I can EasyBuild I can talk a bit about that yes so um as we already told you we are working very closely with CSCS together and CSCS also uses EasyBuild and um then we got some some strong recommendation to also use EasyBuild since most people at CSCS are already using EasyBuild so we can can ask some questions and they can help us and also CSCS is working with you a few guys very closely um since they already did some some praise stuff that is um like some of the the basic stuff where everybody builds on and so we we also have a good access to the developers and from EasyBuild and that's why they strongly recommended that we use EasyBuild um but then we all said I think especially Rahul our team captain um he already was using spec for like one and a half years and so um he he he still used um spec for for the benchmarks but for everything else we use EasyBuild basically and going forward uh we will only use EasyBuild um since all the new people that that we are getting into the team we will start them with EasyBuild I'll just quickly interject since you also asked if we had a discussion um um basically all of our team members this was pretty much our first competition uh except for Rahul so we didn't really have the I don't think we would have had the experience to decide based on the features what is better for us um so it really made sense to just go with what was suggested and and go ahead with that all right and and in the end because as far as I know you were the only team using EasyBuild I don't know how if you have a good view with other teams who are using it at all or using something else but do you think this gave you an edge in the competition and you're losing less time in properly compiling stuff or checking out all the all the details if everything went okay the way it should have been did you win a lot of time thanks to that and then you could focus on other stuff or again I don't have the the anything to compare to I don't really I can't really compare to to Spark but I have the impression that we had a lot of opportunities to make very custom builds that were still um it's it's very easy to to customize everything without breaking anything if that makes any sense you can you can customize it um and still have this whole package manager experience where everything is done for you but it's done for you in exactly the way that you want and I think for the competition that's very helpful I'm not sure if other packages like Spark are flexible enough to do that so so in a sense am I getting it right that EasyBuild in some sense forced you to do things in a proper way rather than hacking around and then later regretting that and losing lots of time to figure things out again I guess that's one way of interpreting it as well yeah because you're kind of forced to work with configurations and you can't just change some hidden files somewhere and implicitly make the the the system do something else that's that's a good point yeah okay good and also as fast I know we were the only team using EasyBuild at the competition that's yeah that that's still interesting to me why other people aren't aren't picking up on it maybe the learning curve that you mentioned is too steep if students only essentially have a couple of weeks or months to to learn everything in HPC not only getting stuff to to install maybe it's too big of a jump to adopt a new tool like this unless you have people backing you where you can help get some I think it could it could also be that spec is wider known I'd say since since most people in the HPC community they they know spec and what spec can do and not everybody I know I think knows EasyBuild and also at the competition I think to our booth that was like five times or so came somebody from from spec and said yeah we should use spec gave us stickers and so on so I think maybe they have like the better PR team in that regard again other thing for all it is I'm going here like that it's also it's one specific characteristic of our team is that we don't do HPC like you know lectures and stuff so maybe also other students can actually be influenced also by the stuff they do during the lectures and their projects so that might also influence the use of spec compared to EasyBuild or stuff like that meanwhile for us it's just a pure hobby and three times so we can actually take the chance to learn and try different paths and yeah all right yeah also like a short anecdote from I was speaking to the the people from AWS a bit and they're they're doing a lot of work with spec and I don't remember exactly what the situation was but they I think asked what we're working with and we said EasyBuild and they they said like oh you're so you're kind of on the European side of things I wonder if there's also a bit of like not clash but kind of different mentalities here yeah for spec the idea is you want things to be to be sure built custom for your system but you want it to happen as fast as possible and EasyBuild is more of the mentality of you really do everything exactly as you want it and you can really touch every part of the build process so yeah and in some sense there are different use cases and also the communities are indeed quite different that's it's not like we're excluding people outside of Europe or something or that spec is excluding people outside of the US but that's just how things have grown over time and there's there's definitely some some cross usage there as well yeah I just heard that comment from someone at AWS and it was quite amusing yeah and in terms of stickers and stuff like this if one of you gives me an address to send some stuff to I'll send you not only stickers but some coffee mugs of EasyBuild as well so as a thank you for using EasyBuild during the competition cool I think we should unless there's more questions for the icon build we can go ahead with Niklas who's going to talk a bit about nwcam yeah um yes so so I was mainly doing the nwcam build I had a bit of help from Fabio at the beginning but after he went over to icon I basically worked on it alone and nwcam was interesting to work with since in a way it was pretty easy in the setup but it was also pretty hard since the documentation was not always very good and you had to find your way through four different files to to get to something and what's what we ended up with was that we used the Intel Compilers for everything I think Marcel will later also talk a bit more about why we use the Intel Compilers even though we had the AMD CPUs and there we also used iFord and ICC and then we all used OpenMPI for the communication and there were some dependencies but they they were not really a problem to install and once all the flags were set and all the exports were set which was a bit difficult to find everything then it worked fine the the only big problem was with the MPI network since nwcam they they want they want you to use armci MPI which they also have we will show also can download from them the the only problem was that the tool the toolchain to download didn't work directly so we had to be a bit hacky and find some some other documentation to get that to work but after that armci MPI worked fine while other MPI versions couldn't talk to other nodes in our system and that is a general problem in nwcam that can sometimes happen and basically we were told don't don't worry about this just use armci MPI and don't don't look back basically another thing we had with the MPI libraries was that we could run armci MPI on multiple multiple nodes but we couldn't use all the course on every node probably because there was too much talk going on and then we basically would crash the system and with easy config everything worked fine this was my my first competition my my first experience with modules and so on and apart from some difficulties at the beginning where the father helped me and setting everything up easy build was was good to use and it was also very easy to pick up to change some some easy conflicts to just get different versions and that was a really good experience I can't compare to specs since I have never used that but at least once you have an easy conflict that's working and the install is working then getting different easy conflicts with different version numbers is really easy from from my point of view and that that's really well all right yeah thanks a lot I think I had another question that maybe fits into into this yeah let's let's ask it here so it's not specifically to nwcam it's a broader question but if you would let's say do the same competition again with the same applications knowing what you know now are there any significant changes you would make and how you approached the competition I mean for me it's not really some significant changes I think for me the biggest problem was that with with all all the university stuff and different projects there we had to we had to do we had very little time for the preparation phase and that was especially for for icon and nwcam since we couldn't get them to run in the short order but had to work on them some some hours and with that I mean if if I would have known everything I know now it would have been way easier to get it to run in the first place and then we could have done way more of different configurations to to get even more performance do some more optimization but I mean with all the easy conflicts that that's that we already did change for different versions I don't think that would have been a big problem to do since the install of the of the compilers different compilers and different libraries would have been fairly easy so so I think it was in the end just a knowledge about hvc and in general probably that that we needed to get and then only the time constraints and other than that I think we were on a at the end on a good path and we could have probably done way better with more time but but but we still had had a had a good go at it I think yeah I was just gonna ask you you guys ended up second in the competition right how big was the gap with the first one do you know I'm not exactly sure it's also difficult to compare since at the benchmarks they had the a 100s and we had the v 100s so there we can't compete for us it's I think it's more for the online part where Marcel will also talk a bit about that later but for the online part we we couldn't get another current icon to run the way we wanted or at all because of time constraints and at some point just said okay we want to focus on the on-site competition to get it to run there and so with more time and with more knowledge we could have also gotten gotten to run it at the online competition way better and could have also had a high finish there I think I think that's that's that's the main the main the main thing we missed in the way because because of those constraints I think either yeah either we could have spent more time or I think we were close enough that with the newer hardware we would have been in place to beat them I think possibly from what I've heard yeah difficult to know up front of course yeah yeah all right so I guess we still have Marcel who wants to share yes the experience as well yes absolutely okay so my background is a bit different in a way that I was actually not supposed to be at ISC but I replaced someone who couldn't be there for personal reasons so I was kind of dropped in this whole thing after it had already started after the online competition had been going for a few weeks at least right so I had a lot of catching up to do and it had to be done quickly and we had to get good results I was tasked with the X Compact 3D application right so it's we had to run some computational fluid dynamic simulations and I was working with another teammate whose name is Piotr who helped me out a lot in this in this experience and I actually already had some background with EasyBuild because it was introduced to me by my father who is also into HPC so we kind of passed down this knowledge a few years back because I wanted to back in high school I had this idea I really wanted to run some chemistry simulations because I was just interested in that and the the biggest fault the first wall I hit was okay I want to use Gromax at the time I want to use Gromax I cannot get Gromax to run right I couldn't compile it and so he was like yeah just looking through EasyBuild and this is kind of how I got it got introduced to it and then I had help from someone else at CSCS who kind of gave me like the basic fundamentals of EasyBuild so for me it was not exactly new once I logged on to the cluster for the first time I was very quick to install EasyBuild without even knowing that other team members were already using the tool so I just installed it in my local directory before even having a meeting with them and start testing out the the application luckily Xcompact was really easy to compile it only took a few minutes and it just went all with no problems it just compiled and it ran right away so the issue was at that point was not really okay let's get the software to run it was beating the other teams at it because I knew that if I was able to compile it that quickly and that easily then other teams would have been able to do that as well right so the task was really okay how can we squeeze performance out of this as much as we can right as much performance as we can to get the best numbers and EasyBuild was really really helpful in this process because it basically gave us the the the power to install software in I don't want to say minutes but let's say hours since you got to compile a lot of dependencies and it takes a while but it sure is much much quicker to install say the Intel compiler to toolchain get EasyBuild then to sit there and manually compile everything and this is actually something that really really helped us and in the end we actually got first place on the PSC cluster when it comes to the Xcompact 3D task and we were also among the first for the other cluster and I think a big part of that was that we decided to compile with the Intel compilers on AMD hardware and the the I believe that when when when I first logged on to the PSC cluster which is an AMD based cluster the sorry bridges to cluster by PSC they didn't actually offer the Intel compilers as a pre-installed package I don't believe so so the solution was simply to install EasyBuild and go through that install it in either our local directory or on scratch I don't exactly remember where I think it was on scratch and this gave us the ability to testing things out without having to put too much too much work into it right and surprisingly it was just yeah let's try it right we don't since we were using EasyBuild it's not gonna be a lot of work so let's try it out right instead of using GCC and it turns out that just by compiling with the Intel compilers on the specific hardware we were using we got a like 1.4 times performance boost which is pretty good right and the more we tested stuff out and we installed optional dependencies and what not the more performance we got out of the out of the application I believe the not only flop slide said that Xcompact2D does not have any dependencies and in a way it doesn't but you can install some optional software and compile it with some optional software it supports for example Intel MKL and FFTW3 and some other input output libraries and what not so there is certainly some like quite a bit of margin to test stuff out and using EasyBuild was really really helpful in getting those dependencies installed and just brute forcing the best possible combination right because at some point you have to test things out you can't just you know experience can give you a little bit of a hand in guessing or like eyeing out what works and what doesn't but at the end of the day testing stuff out is what got us to get such good results and coincidentally the hardware the CPUs that were running on the Bridges 2 cluster were actually the same CPUs we had on our cluster so I just had to copy paste my easy configs and my scripts to install everything back on our cluster and we actually got first place for the Xcompact3D task at the on-site competition as well and it was quite surprising because we got I think it was twice the performance of the second team or like the team behind us and they had better CPUs, newer CPUs so from that point of view we got really really good results and of course it was also thanks to EasyBuild because we had time before the competition to brute force the best combination basically and then get there we only compiled the Xcompact3D once, did one run and we got the best time so like from that point of view we were really well prepared and it was also thanks to EasyBuild. We did some other stuff as well we ran some like a profiler tool to check out say communication between processing we tried to optimize other stuff as well but a big part of it I feel was also thanks to being able to try stuff out without having to spend too much time installing dependencies and compiling stuff by hand and you know you just eb say intel.eb and it will just install it all alone and you don't have to really do much aside from wait and enjoy. Keeping an eye on it to make sure it actually completes and do some other stuff. Exactly. Intel by hand is super frustrating trying to go through all of this one API stuff it's it's very nice that EasyBuild somewhere has like the hard coded links where you can just download the software that you want and be done with it. Okay I had another question as well which is something more more general I was wondering since you were all pretty new to HPC did you learn or did you run into something during the competition that today you still say this doesn't make sense and this would change this in the way they approach it so any any tool any aspect of managing an HPC cluster or installing software on it where you say this is just so weird and why is it like that it's a bit of a surprise question I know but yeah um like well maybe what was the biggest surprise what was the weirdest thing you ran into while working on this like personally I I don't really know how to answer this question because it's it's my first time doing a competition but it's not the first time I've done work on a cluster and when I first started you know working with these kinds of systems I had someone who was following me very closely and was just kind of paving the way so I kind of learned to do stuff in a certain way and that's just how it is right so I got used to it in a way I mean I remember at first I was very confused by modules for example a few years back but now it's totally the best way to do it or at least a very good way of doing that right so I think perspective changes the more you gain experience something that might be strange or unintuitive at first will probably become more intuitive or you might be able to appreciate its value later on once you have a bit more of experience right that's maybe something I was I was hinting towards so there's I think there's a lot of things we do in HPC in HPC community that's mostly due to tradition and and I think environment modules is maybe a part of that they've been using the system since the 90s so long before you were born or long before your your parents were even thinking of having children probably this was the the standard way of of giving people access to to software on systems and and maybe this is due to a revolution or a rethink rather than people teaching other people this is how it's done and we've always been doing this so you should do this as well so I was just wondering if there's anything for you that jumped out like stop doing this and please fix it make it easier make it better yes maybe I can I can say something to that since it was my first like real experience in that in that domain at first yes the different modules were like okay what is this but but I actually think that it's it's it's quite a good system because in that way you can fairly easily change different things try different things and there's this modular approach I think it's actually quite good what what is still a bit strange to me I know why it's done that way and I can I can see that that is needed to set everything up but all the export variables is sometimes quite difficult to to get behind what what do I need to export and what don't I need to export especially if you have no no real idea or how how the program is working or what it's doing in detail at every step and then you then you just see different documentations and they all tell you to export different things and in the end you you just try to put everything in there and I hope it works and also what was interesting to me with it is that documentation is very difficult to find a good one since even very extensive documentations you you come to you come to a problem where you say okay I can't be the first one that got here and there's nothing and then you have to to to look into further links and then further links and further links just to get somewhere and then you try everything and at some point it works as you never touch again so so that was some really interesting experience for me maybe to bounce that back to you you run into stuff where you say we can't be the first ones that hit this and then you've somehow fixed it then you then share the solution and wherever the question was raised or or I will write I will write back to to at the end we can developers with the things I found once once our system is back up again and I have access to it to to just go to all the directories again but yeah I mean it's it's also stuff where if you talk to somebody else they say ah yeah sure I know this I got this before this is the fix and and then you're just okay why isn't it in the documentation basically and that that was a bit strange since in a way if you look from outside it's it's such a sophisticated field it's a huge field there's so much money involved and then the documentation it's just if you come from it with no prior knowledge it's just okay what is going on here and then that was really interesting to see all right yeah I think that's a very good answer and maybe if I can hop in I feel like this is also a little bit the case with easy build we had a discussion before we had a little private meeting before the talk and we were kind of discussing you know what were the pros and the cons of easy build what we found helpful and what we didn't like very much and something that came up was that the documentation is written or seems to be written for someone that already has experience with easy build in a way right so if you but if we go past the installation page and how to set up everything else it seems to be written for someone who has already experienced with easy build and to be fair most of the problems I encountered I looked up the I tried to look them up I I checked the documentation I was not able to solve them and then I went to the Slack channel just wrote a little message in there and in no time someone came up with the solution and they were and Nicholas described the the situation perfectly someone just someone just replied with oh yeah I know that problem just do this and that's the fix right but nowhere does it say there's a problem with that in the documentation or you know exactly what he was saying so this is also a little bit there's also concerns in a way easy build yeah I think it's very true yeah so we need we need either an AI bot or underpaid intern to convert all of the easy build that the Slack discussions into documentation on the website that's something I'm actually very actively pushed for and people who often hang out in easy build Slack notice when when somebody raises a question and we somehow figure it out that we should get it documented somewhere and very often that means make sure there's an issue for it that has the errors or whatever you were hitting and if you figure that out yourself post the question close the issue but then we have it somewhere because in Slack also over time it eventually disappears which is a which is even a bigger problem because you know three months ago somebody asked this and we figured it out but we don't have the answer anymore that's even worse than having it proper not having it properly documented at least it's floating around somewhere so yeah it's it's it's difficult to be consistent there one thing I think we did good there quite recently is the easy build tutorial I don't know if you've used that at all so that's separate from the documentation and there it really takes the point of view from somebody who doesn't know easy build at all and it builds up from scratch explains it what easy build is how to install and configure it how to build your own your first easy config file and and why it's done like this so that's a bit more of a beginner view I think but that's maybe a bit too disconnected from the documentation currently I think I'm not sure if I've seen this tutorial but if it's what I think it is I think there were still some some sections missing where it's kind of there's kind of a disconnect between the like you said between documentation and tutorial where there could be more kind of maybe interoperability where you're saying okay if you're looking at this part of the documentation this this part of the tutorial will connect with that and if you're doing this in the tutorial like this is what's covering this these parts of the documentation and kind of that's why the documentation is structured in this way so that kind of helps absorb the information a bit better yeah certainly from documentation to tutorial that's very true I don't think we have a lot of pointers in that direction the other way I think we have a bit more but yeah we can definitely improve on that because at some point you're not really working through a tutorial you're kind of looking for specific solutions to things that you're trying to do and that will usually point you to the to the documentation because you're looking for keywords right yeah then if if that kind of had pointers to okay if if you want to learn about this section look at these chapters in the tutorial that would be super helpful yeah yeah all right that makes a lot of sense okay do we have any other questions for the raclette team as well are you guys planning to join again in the in the future competition I think you're the team is also joining at super computing right yes so so we were at we are at sc-22 in November and so we will be in Dallas in person all right excellent and is do you have any details on that competition yet like applications or I think the applications are in at least at some parts online or that we check real quick if you give us a stack of stickers we can put them to our booth and compete with the with the SPAC people running around trying to get their stickers away all right a batch of stickers and some coffee mugs are coming up yeah and you have as long as you use the coffee mugs in and at sc then yeah I'm sure they reach to reach you in time some merge for us and some merge to give away at the competition yeah yeah that makes sense okay yeah I was looking at those applications I'm just yeah usually when people tell me applications I know if they're supported in easy build or not by heart and if they're not it's it's maybe something we can we can work on together as well in preparation to the to the competition since you do have some time to prepare there um yeah that's maybe something the community can also help out with that could be interesting okay do we have any questions popping up Simon and I see Sam has raised the question on nwchem but but they already answered it in in Slack uh so yeah let me raise the question maybe you can echo the the answer as well so the question was did you use the nwchem easy config file from the usable repository and if yes did you make any changes to improve that performance yeah so Nicholas can also correct me on this if I'm wrong but um as far as I know we we so we definitely had source provided and I think there may have been some alterations to the default nwchem source um yes I think I think so we we had we had the source provided and they basically told us to work with that and um and and so we we we did use that and didn't download the the nwchem from from easy build honestly I actually saw that it was there but um we we just uh tried to follow their uh the solutions because they also had some some stuff that they told us to set for the exports but uh it was uh in no way um I think we we had to double the amount of exports extra and so yeah that's that's the thing I I didn't realize before so so maybe for the next for the next competition I look at it both at the same time so you actually checked the the easy build repository for nwchem I I think I searched for it saw that it was there but then didn't uh didn't use it didn't ever even cross my mind to do that because like for all of these uh applications we already get a specific repository to use and I think for econ I checked for fun to see if uh easy build has something for that I don't believe they do at least at the time no I don't think so so I didn't even think about that for nwchem but um could be interesting to compare it as soon as we get access to the cluster again and and see um I don't know maybe we can we can contribute maybe Nicholas can contribute some configuration options to the to the easy conflicts but maybe something we can do it maybe something we can do in preparation for for ec is have a separate call where we show some more advanced features of easy build and in particular how you can easily customize installations with easy build so we have this this hooks approach where you can write some small python function which basically hooks in a specific part install step for example where you can tweak some things that easy build does by default and and that way you can get some yeah so pretty good control over what easy build does if there's something you don't like or something you want to play around with without using your own stack of easy config files and and having to copy everything basically now we do have some good ways of dealing with that so that's maybe something interesting to to show you and then also look into together how you can leverage that and in the competition that would be really cool yes so the applications I mean it's uh lin pack for the benchmark then hvcg for the benchmark then io 500 benchmark and then for the applications it's faster and lambs okay but both of those are in easy build I think and I think lambs that's going to be an interesting one because if if you're looking for something that has a lot of knobs to turn lambs definitely fits that so that's going to be a that's going to be a fun one to play with I better finish my pr yeah allen has spent quite a bit of time on that in easy build already and there's some open pool requests we have there as well okay one other cool thing that would be nice to have for future competitions I don't think we need it for for sc but online competitions would be some way to some easier or more standard way to to put to run easy build as a user on a production cluster where you don't have where you're not the the system administrator that's like one of the things that was that kind of played into your question earlier that that surprised me with easy build is just the the the absolute waterfall of files that that easy build produces and puts in your file system because if you talk to system administrators with this who maybe haven't heard about easy build before they're just like well your your HPC software is not not supposed to make a lot of small files and easy build does exactly that so it would be very cool to have some way of using easy build in a very confined environment of course I don't know if the if people need that enough that that anyone would want to work on that but it would definitely be very nice for for online competitions there's definitely some some tricks we can share there to make that easier a lot less painful yeah first you had some comments as well yeah I mean spec is about as bad in that it's just the property of being vital software yeah it's really vital that is the disaster it's not easy build I'm wondering if there's some way to on the fly untar the files that are needed and then kind of have everything else in some kind of tar archive well one thing you could definitely do is install easy build in the container image then essentially if you run that through singularity so run easy build itself through singularity everything is one big file and then you don't have all those small files on the files you're essentially pulling stuff from from one big container image but then that could that could already be a big help if you have access to singularity I don't know about other clusters but for example the eth cluster singularity is limited to uh to shareholders but yeah but but there they have shifted which is pretty much the same thing so you could prepare the container image that and not not shifter but okay what's the other tool they have their saris they have their own container tool and you could prepare a container image that works with all all these container tools container runtime so that's definitely possible and there's other there's other tricks you can pull as well so yeah but there are some kind of workshop on that or or update our documentation for for tricks like that yeah yeah if you don't have access to singularity on the cscs cluster then that's probably because not csc they have a commercial license and because you cannot have two instances of singularity on the cluster because the craze software just comes with singularity so they must have replaced the craze software with a different version of singularity which i cannot give to all users i'm not sure about cscs i'm talking about the eth internal oiler cluster which is hosted at csc but i think the the administration that might already be a difference yeah because i mean craze system singularity is just the standard container environment for craze there's probably still some way to containerize it so yeah yeah there's if you're a little bit creative you can work around any any issues yeah okay thanks a lot i think we're close to yeah to wrapping up very interesting to hear your experience as well and yeah it's clear it's a very maybe different approach maybe more traditional approach compared to the not only flops team but yeah very interesting i'm very happy to hear that easy build was a very good help and hopefully that will remain the same if you join more competitions like this and and as you already know yeah if there's anything we can help with as an easy build community please reach out and the easy build slack is the best way of doing more maybe if i can take one minute before we close to do a little bit of advertising we actually have some if you guys are interested we have some visualizations of the work we did up on youtube so i'm gonna just post the link in the chat so just to see a bit what the output of our work was i'm gonna post a few links in the zoom chat yeah and yeah you should do the same in the slack tech talks channel as well all right all right there at least the link the links are going to stay there for a while in the zoom chat they won't disappear okay i'll post a bigger thing excellent any any last words from the not only flop teams as well before we wrap it up oh i don't think we have sound anyway just the wave is fine as well all right thanks a lot this was fun and maybe we should having us yeah maybe we should make this a yearly thing and have some teams report back on how they did during the competition this was very interesting you have at least one team using it about now so yeah maybe more in the future who knows one is better than zero already yes next year we'll have a team using easy to set our projects we're interested in yes that's also something we should show you because that could see if you're quite a bit of trouble as well and if we're talking about installing compilers and dependencies from source what if they are just sitting there and you could start using them straight away without having to do anything so that's something the the easy project is is aiming to do so that's definitely also something we should talk about all right thanks a lot everyone and see you around thank you for having us bye bye thank you for having us see you bye bye