 So, the next talk is about using GNU Geeks on HPC systems, so by Ludovic and Kirtel. Right. So, good morning, everyone. So, this is, again, going to be one of these two people talks, and so I'm Ludovic Cortez and Piotr Prince is going to talk afterwards about how we can use Geeks on HPC systems where we don't have root access. So, before I go into the details about what Geeks is and all that, let me first recap what the situation is on HPC systems. So, I'm guessing that many people here are familiar with what the situation is on HPC systems is not that great. We already had a talk this morning about how to improve things. Let me just describe a few key ingredients of what you can find on most systems nowadays. So, this is a recipe. So, the first ingredient is you have your, you know, super computer. It's super fast. It's wonderful. And so, first thing you do is you install an old GNU Linux distribution, right? It's outdated, so people are going to complain, you know, that, right from the start. And it's also inflexible, right? If you're not root on that machine, you're not going to be able to do sudo apt-get install blah blah, right? So, as an HPC user, you're not going to be satisfied, right? So, C-Sat means have come up with solutions to try and improve on that situation. So, the main solution that we're seeing so far is to add a layer of homemade modules. So, if you're not familiar with modules, it's this thing. So, it's, you know, it's a set of commands that you can use from the shell and you can say, okay, I want to load this specific package like GCC at that version. Yeah, you can, you know, select what you want to have in your environment. So, that's already a great improvement because it brings a lot of flexibilities that HPC users actually need, right? You know, you have a machine with tens of users, they have different needs in terms of libraries, tools and versions, and so you want them to be able to choose that. Now, the problem is that to actually achieve that, C-Sat means are essentially making a distribution of their own and doing that by hand, right? So, they're compiling lots of packages by themselves, installing them in those special directories and so on and so forth. So I don't know if it's the same for you, but my experience with modules on a cluster at work has been pretty much like this. So, first thing, you know, C-Sat means are kind and they produce new modules, you know, with packages that you actually need, but it doesn't work right from the start, right? So, they first need to tweak the modules until it kind of works and then it's like, oh, wait, there is a runtime linker error, like I cannot, you know, run this command, I get an undefined reference error or something. And so, they tweak things again and, you know, some user complains that this particular piece of software needs to be compiled with different flags. So, you know, they tweak that module and then new versions are out so they need to rebuild everything and then some user isn't happy, you know, like, you have a deadline for a paper submission and just before the deadline they decide to upgrade everything and to remove the old versions and you're like, oh, I'm screwed. And so, that happens in reality, right? So, we have flexibility, but still users can be in a hard situation and so on and so forth. So that's not so great, right? And of course you can have user build software on top of that, especially with all the language-specific package managers. So, Chris Weber yesterday was saying, this is one package manager per child. It looks like it. You know, there are so many tools available. So it's becoming a real mess. So fortunately, people have started to consider, you know, changing the status quo and we had to talk about singularity this morning and we'll have a talk about easy build right after. And so, let's see how we can fix HPC cluster environments. So essentially, there are two approaches in use. So the first approach is to write specific package managers for HPC that address those problems. So the main ones I know of are EasyBuild and Spark, right? And so these are package managers which are designed to be used on top of your, you know, old inflexible distro that you have in the first place. And that's really good because it solves quite a number of the problems I mentioned. So first of all, people can actually share their package recipes as opposed to having each sysadmin of each machine building things by themselves just for their specific machine, right? So people can actually share their recipe, their packages. And it provides easy ways to deploy software because anyone can install EasyBuild and Spark on the machine without being root. So if you install EasyBuild, then you can, you know, install your favorite packages on that particular cluster and it works, right? So that's a great improvement. Now there is some limitation with this approach. I think I'm going to maybe not make friends with the next speaker, but OK, we'll discuss that. So one of the main issues I can see that, well, so first of all, as I said, these are tools that are meant to be used on top of the distro that's available in the system, right? So every user is going to recompile software in their home directory, and that can be kind of annoying. You know, that's kind of a waste of resources and CPU and storage. The second issue is this one. I don't know if you can read it from the back of the room, but this is a sample bug report from EasyBuild, I think, where people are saying, well, GCC libraries sometimes end up being, you know, built with this flag and sometimes not. You know, that's not a great situation to be in, right? We'd rather have something reproducible. This is kind of unavoidable because, you know, as a non-word user on GNULINX, there is simply no way you can get a fully isolated build environment. And so that's what you end up with, right? OK, and for good measure, this is the same kind of bug report but forced pack this time. And again, you see, on some computers, we have this behavior, and on some computers, we have this different behavior. That's not a great situation to be in. So that was the first approach to fixing the HPC cluster environment issue. Second approach is to, I would say, give up on packaging. So what does that mean? Have you heard of that thing called Docker? Well, that's a story. You basically make an app bundle with Docker or with Singularity like we saw this morning. But what you're doing when you're using an app bundle is essentially you're telling people, OK, this is my application, right? This is all shrink wrapped, like Ricardo said yesterday. Essentially, you're providing a complete image that contains all your dependencies, plus your application, plus everything. And that's not so great because it prevents experimentation. Like from a reproducible science viewpoint, you not only want to be able to run the software as is, you also want to fiddle with that particular box here. And how do you do that? If all you have are the bits of this app bundle, well, that's not very helpful. And also, what if somebody comes and say, hey, look, I have my application also available as a bundle, right? And now, how do you comport them, right? You end up with maybe this box and this box is the same, but there's nothing you can do, right? They're not shared at all. So I think this solution is the wrong one to put it simply. So at this point, the question is, can we have our cake and eat it too? And that's where Geeks comes in. So how many people in the room have heard about Geeks? OK, that's half of the room. So essentially, Geeks is a package manager. And it's a transactional package manager that can be used as non-root. It allows you to manage software environments. It provides lots of tools and APIs to fiddle with, you know, packages, environments, and so on. It has almost 5,000 packages available on four architectures, and we are providing binaries to continuous integration. So when you're using Geeks, and if you want to trust the binaries we provide from our server, you can simply use them. It has a pretty active community with roughly 40 contributors every month. So I would say, yeah, it's pretty active. So how did that work? How much time do I have? Two minutes. OK, so let me just a very quick demo. Essentially, you have a command line interface that allows you to search for packages to install packages. And this is happening as non-root, right? It tells you what environment variables you may need to set in your environment if you want to be able to use the software. And this is all transactional. So in a single transaction, you can say, I want to install something, and remove something else. Right? And that's a single transaction. And if you're unhappy, then you can always wall back. And this is all per user. So if you're on an HPC cluster, that means that everyone can run these commands for themselves. Which is pretty nice. OK, you can list the generations and finally wall back if you're unhappy with the new setup. OK. So how did it work? Well, under the hood, all the packages are built in isolated environments. So this is really the crucial part here. So we are building packages in CH routes with separate name spaces, and so on and so forth. So containers. And what we get as a result is a unique file name in that store directory with that long hash, right? And essentially, everything is going to be, well, almost everything is going to be bit identical for every person. Like, if I build this package and pure build this package on a different machine, we're going to get the very same bits most of the time. So in a paper, Ricardo Vermes and myself wrote, we explain how this kind of tool can be useful in an HPC setup, and what kind of good it provides. And so from an experimentation viewpoint, it means it has tools to create package variants. You can say, I want to use this package between different source. I want to use this Mums package, but I want to use the different dependencies, and so on and so forth. And you can have your personal packages as well. OK. So Pierre is going to talk about non-word usage. Can you explain just? Oh, OK. So just one last thing. So I was talking about isolated build environments, right? And essentially, we have a buildDemon here, and that's what provides isolated build environments. But that Demon needs to run as root, which is a source of problem on HPC machines where you cannot have this Demon running as root. Right. Now I have a solution, and this is work. All right, so taking off where Ludo left off, Geeks basically installs in a path called GNU slash store. And this is done for reasons of reproducibility. We want everything to be exactly the same in every environment. But on a typical HPC setup, and I'm using a super computer at Oak Ridge, we're not allowed to run a Demon with root privileges. So the build system running on Oak Ridge is out. One solution we have, which is also described by Ricardo, who is here, you could mount a GNU store through NFS and have a buildDemon sitting on a separate note somewhere, which is a possibility of creating these packages, downloading these packages, installing these packages, and making them available in the cluster. So if you have a slightly enlightened system administrator, this is a possibility. But if you don't, what do you do, right? So some years back with Nix, I tried to, there's a link, actually these slides are online. You can download them. If you click on this link, it will take you to the Nix distribution, which is a sister distribution of Geeks. Geeks was forked of Nix five years ago. And I tried to use P-Root, and P-Root worked great. So it actually slams in a local directory for a global directory. So the GNU store actually got remounted somewhere in my home directory, and that way I could run the software. But of course, every time this directory needs to be resolved, it slows down the process. So a couple of years back, with Elko Dostra, I'm also at FOSM here, I had a discussion, and he said, OK, you have actually a pretty unique path here, right? So for Glip-C 2.2.3, the Glip-C library, the shared library, we have a path that looks like this. Includes a hash value. And if you would scan the software, you could actually find this hash value. So if you do look for the LinkedIn shared libraries of the LDC compiler, the decompiler, you will see something like this. So this also explains the way what Geeks actually does. It hard links all these libraries, so they are found separately on your system. So one thing you could do is try to rewrite these targets inside the files. And Konda actually does this too, in some cases. And the only dependency you actually is left is the Linux code itself, because actually when we ship this, Glip-C is included, and the shared library loader is also included there, which is this one, Elk load Linux. So what happened here? The path is a fingerprint. We scan all the files, replace the fingerprints with something else, which has a relative path. So this is slash GNU slash store. And here I have a home directory opt LDC test. And it worked. There's a tool written by Elko called Pathshelf, which actually can do this for you on Elk files. And the link is here. And in textual files, like in Ruby and Pearl, there's also these long file names, these fingerprints, which can also be replaced easily, of course. But unfortunately, when it comes to Python and JVM, it doesn't work that well, mostly because they're not zero-terminated. And you cannot just, without knowing the internal file format description, you cannot just expand these files. So one night, I had an idea. I said, well, why don't we keep the file path at exactly the same size and patch everything? So basically, the second insight, you end up with something that has the exact same file length, sorry, string length. So home user opt gets replaced. And the hash value basically gets cannibalized to make it the same size. And it works. The only downside is that the prefix can be up to 40 letters long, because then your hash value is gone in the prefix. So I've put in an example. It's online. You can do it. So you can download the latest decompiler, which was released only a few days ago. It's a 24-megabyte download. It unpacks into 140-megabyte and installs in three seconds. So in the installation, of course, is the rewriting of the path. And it comes with the batteries included, right? Because all the shared libraries are with it. The clip C included. So you don't have this typical problem that you get these clip C incompatibilities, or LLVM incompatibilities, because the kitchen sink is just included. Yeah, so so far I've tried LEC. I've tried Ruby with SSL and Noko Geary, which are also notorious. And then a Sambamba tool, which is a tool that is written in D, in the D language. And it's actually deployed among many HPCs around the world. And this tool, until today, we've actually done by statically linking in the libraries. So it's a binary without any references to outside. But the current version is actually, we're using now to debug problems, comes with the shared libraries. And more is coming. So Oak Ridge actually takes a great interest in this project. So maybe there we get some support from them. So Geeks also has cross-copulation. It's built in like Ludo said. So you could say something like Geeks built, and then set a target for the MIPS in this case, and then compile something. And then you move it to the new architecture, and it will just run. So I think in the foreseeable future, we will have fire support, and also NVIDIA. So this is actually give us. The thing to understand really clearly is that Geeks is an existing distribution. So people are using these packages every day. There's thousands of people who deploy these packages. Maybe 100,000 people use these packages. So unlike many other systems like EasyBuild, which is kind of ad hoc, you have actually a completely reproducible and tested environment. The only thing, the only trigger that I'm doing is rewriting the prefix. So if there's any issue with the software, it could have to do with the prefix itself. But so far, we've not hit any problems. And I want to conclude that two simple ideas, which happened at FOSM, can come a long way. Then we can build HPC. You can easily see this happen. You can create a repository of binary packages. And these can be one click installs. And essentially, they are already, but the install script is inside the table at this point. You can have it outside. You can ship software easily. Yeah, maybe we get it in Linux. Thank you. Thank you, Piotr and Ludovic. Any questions on the use of Geeks for HPC? I actually have a question. So I'm one of the people that opened the bug report that you showed. I'm the EasyBuild police manager. Actually, you're hitting good points. Isolation is a big issue. And there are actually people now who combine EasyBuild with Nix, which is very close to Geeks. So a couple of guys in Canada are doing exactly the job on top to build all the scientific software with. Is that a use case that makes sense to you? Or do you say just throw out EasyBuild and go the Geeks on Nix way? Yeah, so about using Geeks as a back end for EasyBuild. I don't know exactly how that could work. I guess I don't know EasyBuild sufficiently well. Yeah, I don't know. Maybe it would make sense. Do you want to comment? Yeah, with Konda, we're seeing the same thing. So and I think with both these systems, you have real problem with bootstrapping. Because you have to bootstrap from the underlying distribution. So one thing Geeks can do immediately is create the bootstrap environment and build up from there. And then slowly, when you have Geeks available, you can start replacing the stuff that's already there. You don't have to do again an EasyBuild. How much scientific software is now supported in Geeks, like readily packaged? Ricardo? A lot. Yes? Whatever we need, mostly bio-compatible stuff. But there's other scientific software. There are a few HPC deployments already. People, mostly bio-info people, like these two people, have been packaging a lot of bio-info plus HPC plus algebra software. I think about 400 R packages in there. It's just somebody has to do the work, right? Well, Geeks actually also has a generator for packages. So if an existing R package and you want to pull it in, most of it is generated. That's the easy stuff. R packages, Python packages, that's trivial. You can write a generator for it. And EasyBuild is very easy as well. But there's other stuff in EasyBuild that I'm reasonably sure it's not in Geeks yet, which is really, really nasty, which has interactive configuration scripts. I mean, the army is going to talk about something. Right. Have you packaged a note to the package manager? NPM? I'm not sure. I would have to check. It's possible. We've already had two people die on the way, right? Yes? A real question. I have to install our customers cluster in the UK. That's real. I have to do it this week. Would you say I should use EasyBuild? I'm going to use EasyBuild to show you the new Geeks. If they don't want to use R Studio at this point, I would suggest Geeks. OK. Why strong? Feel like. Sorry? You just give a Y. Y? Yeah. I know why he's giving a talk. Maybe Lidl should answer this. Right. I mean, obviously, you should be using Geeks. That's pretty obvious. I mean, the reason is that you would get. So the advantage you would get with Geeks of our EasyBuild is that reproducibility is thing, right? So we have R plus lots of R packages. And if you install them with Geeks, it's going to be reproducible. You know that it's going to work just like on every other machine of Geeks users, right? There's not going to be bad surprises just on your specific machine. Now, you will also, I mean, as it is now, you'll have probably to run that build demon as root, which may or may not be a problem in your case. But the solution that Pierre showed is precisely one of the options we want to offer for people who cannot run the demon as root. But I think you cluster white. Yeah, right, yeah. You can make it cluster white. Yeah, so we have a couple of more Geeks talks tomorrow in the GARL dev room. So if you're interested in the technology, you'll get a lot more out of that. I guess even Geeks would also pull in the crowd. If you build on a slightly different logic, I'm talking here about Geeks 86, 64 Geeks, different micro architectures like, let's say, Sunday bridge everything, that kind of stuff. OK, easily, that would be the cloud point. Yeah. Must not miss that part of the discussion. Right. That's all we have time for in terms of questions. If you have further questions, you can meet up with Ludovic and Kjelton outside and keep the discussion going. Thank you very much.