 So, welcome everybody. My name is Pard Obman and I'm an analyst at McGill University. It's part of the Collaborations of the Digital Research Alliance of Canada. I'm going to talk about the way we are using EasyBuild. We've been using EasyBuild for about seven years on a national level and our present challenges and developments with a future outlook on the design decisions we have to make around this point in time for a new iteration of our national press. So, first I'm going to explain a bit about our organization and then get a bit more on topic about EasyBuild, etc. So, what is the Digital Research Alliance of Canada or what is taken over from Compute Canada and you see that name popping up pretty much everywhere still because changing names is hard. It's basically a collaboration of different organizations across Canada where we provide supercomputing and other digital research services to researchers across Canada. So, in Canada and every university if you're a professor, a principal investigator of PI, you can apply for an account, you can sponsor your students, your postdocs, etc. and they will get a free account on the national supercomputers and you get a bit of basic compute time and then every year we have a contest where they have to send a proposal and depending on merit and how much they need, we give them allocations. So, there's no payment there and then some researchers have special credit money, they can have contributed systems as well. So, part of these clusters are basically owned by the researchers in sort of a condo style setup. So, we are organized in five regional consortia. So, we have ASTEP in the east, Uncle Quebec, Brian, Compute Ontario and then out west we have West DRI. It's formally split into two organizations, BC, British Columbia and Paris. This is the research infrastructure. And then we have member institutions which are the universities also like research hospitals. We have about 250 technical staff distributed on one of these universities. Almost 20,000 user accounts of all the students and postdocs and TIs across Canada and six bigger like nationally managed clusters for clouds, about 300,000 cores, thousands of TPUs and hundreds of petabytes of storage. So, you see quite a diverse range of research disciplines. Of course, we have all the traditional disciplines, but in recent years it's gone all over the place with a lot of genomic, very informatic users now and digital humanities also growing. So, how it used to be that we had pretty much every university that did some research had its own cluster, but that wasn't very efficient. That was a lot of fragmentation. So, we decided to get a smaller number of national systems like five proper HPC systems which are bigger and then can be managed sometimes even by distributed system administrator teams. But every researcher in Canada has access to these national systems. So, these were mostly put into production in 2017 to 2019 and with the newest one of ours, we had McGill University was put into place in late 2021. So, we now have some sites that have no cluster anymore, but they still need some support like they have like an analyst that is interested in Saskatchewan and he just worked with the researchers there. So, they have like local support. So, what we wanted at the time is to say no matter what cluster you log into, you can get a similar interface. So, you can have your submission script and it works on one cluster and then you can pretty much trust that the same submission script maybe a few modifications in number of course things like that will work on a different cluster. You don't have to recompile your software, it can be easily transferred. So, if one cluster is down or you get your allocation somewhere else, you can without with very minimal effort move with the cross, which didn't used to be the case. It used to be the case of gas to pretty much rewrite the whole script. And for that, we decided to use a distribution mechanism, CPFS. There are many other people who use that now and I'll explain a bit later what that means. And one innovation that we did on top of that is to introduce what is called the compatibility layer, something that sits in between the operating system and the software modules. Basically, we abstract away the OS. So, you can run this independent of whether you have sent to a 7 or 8 or Ubuntu or some other distribution installed. We do the automated installation because we're using Easy Build. Otherwise, I wouldn't be here. And we have Elmot as the module tool like most other people here. So, the background is really like where does this come from? This came up in the poll just yesterday as well. It's like, yeah, most APC clusters use enterprise linux distribution called Reasons. It's mostly vendor support, like the vendor support IB drivers for infinite networking and often only support on a couple of linux distributions. And so, system administrators will not put the latest, greatest fedora, for instance, on the cluster. That's like playing with fire. So, they tend to have the older enterprise distribution. We saw yesterday that's still a large proportion of the server users are using sent to a 7, which is getting quite old at this point. And you can see the versions linux kernel 3.10, TC485, etc. And so, sent to us 8 and 9 are quite a bit newer. But then when some people are used to desktop linux distribution, they get, oh, we have Fedora 38 and we have linux kernel 629 and TC13. And I'm running a program on that. I'm just copying it over to the cluster. And I expect that to work. And I complain to support that. That doesn't work and say, sorry, you can't expect that to work. So, and some people just install a software and what you do when you try to get software to work when you go to the documentation of that software and the software will just say, please install the software, please install the dependencies to install dependency to do sudo and to get install whatever you need. So, they type that in on the cluster and they'll get the sudo lecture. If you trust that you will receive the usual lecture, please type your password, type the password and so on. They can even carry a message you're saying, you have been reported. And these messages are all sent to Santa, which then will not give them presence for Christmas. Anyway, so they try that, but it doesn't work. Yeah, but it doesn't work that way on HTC systems. We have modules. So, traditionally, people write modules. What does that mean a module? Most people don't know what this is, but there's some that might not be so familiar. Like, okay, you just write a bunch of environment variables and file. You put that file somewhere and you load the module file using say module load Python 279 and you get these environment variables in your environment and you can load Python 279. Very old now, but this is for illustration because this is how we used to do that back in the day. Now, how were they created? Of course, by hand. You take another module file, you copy it, you adjust what you need and then you have a new module file. If you did a little bit better, you have like a logging system. We said, oh, I did this today. I created this file. I modified it like this, so you have like a record of what you did. So at least you have some history, but it's still very messy. One advantage is you make yourself invaluable. So people put a high insurance value on you if you're the only person who does that on the site. So this is why easy building exists basically is to automate these things in a more consistent fashion. So what we are having is a design overview and this is the layered view that you will also probably see in Casper's talk. So I'm spoiling that a little bit, but I cannot present my talk without a spitter. So our software setup is layered with a top layer. It is called the easy build layer. It has all the modules. So the compilers of the software is an easy build. Anything that's compiled with the Intel compiler or NVHPC or UCTPGI, anything that uses MPI, put in a module, people can load that software using module load, whatever. Under this, it's a compatibility layer which has more of the boring dependencies. When you use easy build without any modifications, you'll get a whole bunch of dependencies in the beginning which are fairly boring that people usually don't load directly like the other tools. You have a module for it, but scientific users are not particularly interested in that software. They want to load Chrome X or NAMD or some other software that they need for their research. These are more like the boring dependencies that we have put in the compatibility layer. And it also replaces parts of the operating system. So we can have very a minimal compute nodes on clusters. We have a very minimal amount of software which then can live inside a RAM disk, just an image in memory, and all the rest of the software comes from CVMFS by our software. Then under this, it's another gray area where we have something that could be in the compatibility layer, but it's very system E, like IB client libraries, CUDA client libraries are in the system, but IB libraries, we put them in the compatibility layer because it works fine. It can be overridden if it doesn't work well. And then at the very bottom, we have stuff that needs to be on the system, needs to be installed by system administrators, like the Linux kernel of course, the demons, the drivers, CUDA runtime libraries, anything privileged, like we're not touching anything that needs root, that is always local. And sometimes there's software that has licensing restrictions that we cannot put in CVMFS, not even a restricted CVMFS that is internal, like fast past these restrictions. It is, no, we have to abide by the copyright and then that's it. So what is CVMFS? A CVMFS like a distributed file system. It means that you have, the few of the files you have is the same no matter where you log in, but it's distributed using various layers. So basically when you publish some software that we do, we first put it on what is called the stratum zero, then the software is distributed from the stratum zero to multiple stratum ones, and then the individual sites, they will pull that data from the stratum ones, so they can have a main stratum one server and if one of the stratum ones is down, you can use another one, again cache into what's called a squid, and then we have client nodes which take that data from the cache. So the first time you use a file on CVMFS, they may be a bit overweight because it can download over the network. The second time you use it is all cached and it's all super quick. So what we're using for the compatibility layer is a system called Gen2 prefix used to be, we used to use something called NICS, but we found Gen2 to be a better match, a more minimal solution to what we want. And so Gen2 prefix is just like Gen2 Linux, so Linux distribution where everything is compiled from source, but it's put in a prefix. So instead of putting everything at a slash, the top directory level is put at some lower directory. And it's just the management system to use e-build to compile software. And we use that Gen2 to compile anything that's of little scientific interest and where the performance is not crucial. And so we have that layer of lots of convenient tools, including editors, libraries, build tools, just common LS for users, et cetera. And then we have newer versions of what is in, for instance, CentOS 7. And we use that as an abstraction layer between the OS and scientific software stack. So you load the module and then as soon as you load the module, Gen2 2020 you get more or less whatever Gen2 look like in 2020. Yeah, the question is whether I'm using Gen2 2020 to compile stuff on easy build? Yes, it is the bottom layer for easy builds. So we're using that as the system layer more or less. So for instance, when you install easy build as a bootstrap or your bootstrap PC core, really, and without any modifications, it will try to bootstrap from the system. And for us, the system from the easy build perspective is Gen2 presets. So for easy build, I put the same scheme for the logo, but it's not quite, we get rid of Nix, we replace by Gen2, but here we get rid of the old logo and replace by the new logo. We use that to just automate installation of scientific oriented software and then it models files them with Lmods and things covered already. So we use that for the hierarchy, sort of a hierarchy model system so that we lose it below the compiler and an MPI and a cooldown module to get a particular view of the module. So conceptually, what we have a builds, we call them recipes, like an easy config is a recipe, an eBuilds recipe. And we store them in Git repositories. We have worked the easy build repositories, but over the years that got a little bit messy and every time there's a new easy build release, I do. I sync it with the upstream easy build and particularly for easy block. And I got into quite horrendous merge conflicts at some point. So I decided to go the way of some other sites like you, which has like a custom repository, which is not for the easy box system, custom bunch of easy blocks. So we have a more clean separation between the between our easy conflicts and the upstream one. So that's sort of work in progress to do that. And Gen2 overlay already works like that. And Gen2 overlay is just an overlay over the upstream Gen2 where you just have the eBuilds that Gen2 themselves don't provide. So how do we decide what goes in easy build? What goes in Gen2? Let's say, is it performance critical? Or does it depend on the MPI? Or does it use Fortran? Yes. In that case, we'll use easy build. Are there multiple versions needed via modules? Like for instance, for something like CMake, people often need a newer version then provided by the base layer would mainly be easy to use easy build. And for anything else, we will use Gen2. So this is a little bit different from the decision list for easy which we covered in the next talk. So for easy, they put a lot more in the software layer. It's easy build and we put quite a bit more in the compatibility layer. It's just a different design choice. There's no right or wrong. It's a different way of doing things. Then the way our workflow works is that we have a specific build note. We have a custom user, an ED user that compiles software. We can do a test install on our home directory on the build note. Then we install it as the central user. Then we can test it on the build note. We can deploy it on the CV Invest test repository, the DEF repository. We can test it on the cluster with P-ROOT to change the path. You can use F10 or Singularity to change the path on the cluster to do some testing. And then we can deploy on the production repository. We know it works. Sometimes we shortcut this a little bit when it's a new module but nobody else uses it. Then there's no risk of breaking things when it's just a new module. But when we update a module that's already there, we have to be more careful and test it more carefully. You may have noticed that we used to use NICS and we tried a Tentool. Why did we do that? In NICS, there's a somewhat complicated scheme where they can use multiple versions in parallel using hashed directory names. This mostly worked but we had sometimes that hashed directories would leak into users' environments that would sometimes confuse users. Sometimes we would update software in place and the hashed directories would be garbage collected and just disappear and user stuff would break. And the main problem that we had with that is that NICS is designed to work at the top level and we had it at a cent which is in between two levels. It wasn't really designed to work that way and we ran into that because of basically using something in the way it's not designed to work is asking for problems that way. So with Tentool prefix, we don't have the complicated symbolic links with hashed directory names and it's just like more of a minimal solution of what we need. Another important part of our software stack is the Python wheels. So basically instead of having a very fat Python module plus custom modules for Python extensions, we have a more basic Python module, a bit more like Python there, where we just have enough infrastructure to build Python wheels and to install things with Python and pip, etc. And we supply wheels that users can install in in virtual environment. So we have a wheelhouse, which is all set automatically, people install Python packages. So what we tell them to do is to set up a virtual environment to do virtual and some environment. They enter that environment, they pip install whatever they need, and then they can load their stuff. And so this way we have a very flexible setup for the users without having fixed versions. Users can even compile their own wheels if they want to. And how that works is that we tell users if they want, say, TensorFlow, they can type a fairwheel, TensorFlow, and it will find TensorFlow that we're having the wheelhouse. We have now a lot of wheels in the wheelhouse, more than 10,000. Actually, right now we have more Python wheels than we have modules, amazingly. I was quite astounded by that myself, actually. So we have a witty page about that and people will just type a fairwheel, TensorFlow, and they say, oh, we can install TensorFlow 2.11 via pip, and it will get the TensorFlow that we know it's optimized to our system. And it's available for free Python versions at this point, 3.8, 3.9, 3.10. Those are the ones we Python versions that we supported at this point. We dropped Python 3.7 about a year ago, and now we've just introduced 3.11 to the mix, but most default is now 3.10. Do you see how it goes over the years that the number of modules keeps on growing, also because we keep the old environment? So right now we have just over 10,000 modules and even more Python wheels. So what does it look like? Now you may know that in EasyBuild we have the common tool change, Boston Intel, and there's a new release twice a year, so there's always 2028, 2020D, etc. We have a more slower cadence for the default modules, like we have something every two to three years, but we have something where we compile everything, and we don't do that every six months. So at the moment our production environment, what we call standard M2020, is mostly based on Intel and OpenMPI 403 and GCDyn 30. So now there's a lot of software that sits at the GCD core level. It means that we doesn't need MPI, it doesn't need BLAST or anything, it just needs GCC. And so when people do module fill to see, for instance, the vast majority of bioinformatics software is just GCC core compiled, and they see that and we already have the arch optimization flags to make it work fast enough. And then there's a much smaller subset of software that we compile with Intel or MPI, or both. So we compile still, we compile most software with GCD930, it's getting a little bit older, but it works fine for the vast majority of things. We did a slight modification where we had a new production cluster, Naval, which used AMD by the other ones, used Intel and after some benchmarking, the link is here. I found out that for AMD it was, BLIST was more performant than using MKL, so we used FlexiBLAST to decide which back-end used for linear algebra. On the AMD process we use BLIST and on the Intel process we use MKL, but the users just see all we're using FlexiBLAST. We do have newer tool chains available, for instance, there's, we have 2022A tool chains. Users don't see the tool chains themselves, that's more an easy built internal thing to us, but they just can do Motorola OC11.3. If they do that, they get a newer open MPI, they get a newer software stack, if they have to, but it's not what they see in the default module view, it's just that people need something newer, they can just load the modules. And we made that the default in April 2021, but the older software stacks available for people who want it. Now we're working on a new environment, of course, we need to replace the standard end 2020, and that's like Intel, maybe we use standard end 2023, that would then have mostly software from this year, but it would be in production in 2024, because it takes some time to, we compile everything and then make it a default. So from beginning to end, we give ourselves about a year. So the plan is now to use something more based on force 2023A. We found out that there's not so many benefits to compiling things with Intel, there's some things that work better with TC, some things that work better with Intel, but when we want to distribute to the public, it's always safer to depend on open source software and not have licensing issues like you'd have with Intel. So this is all based on software that hasn't been released yet, so we're probably sticking with a TC 12.3 and open a PI415, I suspect those will go in the 2023A common tool change, they haven't been set in stone yet, but probably that way. So there are some things in the pipeline, we have to weigh pros and cons of using that, that the CC13 is better support for newer AMD GPUs. There's an open MPI5 that's been a long time coming and it may be more performant to use. So if we are very risky, we could include those, but time experience has told us that it's based off to be a little bit more careful and then leave the more risky stuff to users who actually benefit from it. So every time we had a new iteration of the standard environment, we have to say, oh, what goes in the software layer, at the top layer, what goes in the compatibility layer, do we have to move things around? And so some of the things we've learned is that build tools are also a bit of a headache, like what do we do with CMake, Ninja, Mason, etc. They are sort of automatically installed if you install Gen2, but easy build often ones like a specific version, sometimes the version from 2020 is too old, so you want to have a module for that as well. And so we made a say, okay, instead of having a base version in the Gen2 prefix and a module for it, we just kick it out of the Gen2 prefix and have only a module for it. Now there's a lot of overlap in the compatibility layer with what EZ is using, you could almost say that EZ is developing a subset of what we're doing, so I'm trying to learn from EZ much as they're sort of learning from us. And so I'm picking up a lot of ideas about how to do things more properly, more automated in particular. Another thing that caught my eye is the solution for our path from a guy named Harman Stoppels who works in Switzerland, seems to be Dutchman. So what we're doing instead of using LD library path in our stack is that OR software has an R path and links to a particular library using the R path embedded in the binary. The problem with that is that in order for that to work, we have a linker wrapper. So instead of using the regular LD, users execute a linker which then inserts the the libraries in there. Now EZ build also has functionality for our path of minus minus rp, but instead of the linker level, it uses a compiler level and it doesn't directly translate into the users. If the user compiles their own software and loads the module, they still need the LD library path otherwise at runtime the software will not pick it up. So there's an idea from Harman Stoppels saying let's change the SO name like a particular field inside the library and then when you link that library it will just pick up the runtime path automatically without any need for a wrapper or LD library path. So that's a very promising solution because it will work. It will basically work without additional hacks apart from changing the libraries themselves. So we're looking into that. Yeah, that's another thing. Yeah, but what Kenneth says is speed things up. It eliminates some load storms. We're not as affected by load storms as an upstream EZ build. That's because so much of our stuff is in the compatibility layer that they already find a majority of the libraries usually in the compatibility layer and then the R path is fairly small only has like two, three, four entries and so it only looks in those two, three, four entries. If you have the libraries embedded in the binaries, it will not search a list of directories for that particular file. It can directly open that file and doesn't have to spend a lot of time searching for the library. So that's why this is not a good solution. So I know that that Eleanor Kais was was looking at this as well and getting a full request for EZ build to do this. He is more of a spec developer so it's been optional and now in spec for for some time. Another thing that we've we've seen is a bit of a war story. Like we ran into a bug in CVMFS where we updated some some file in place. We did a Tlipsy update because somebody said, hey, this is a cool feature of Tlipsy. Okay, let's let's put it in. And then I I deployed it. I tested it and then boom, we got a lot of complaints. So it was quite embarrassing. And so basically I pushed a new Lipsy library into CVMFS and processes started crashing randomly. And I was just checking what's going on here. I did a shot 256 some on the on the library and it would give different results every time I did it. What is going on here? This was very bad. So we scrambled and referred it to the older version, which you can do because CVMFS keeps a bit of a history. And so this became a bug in CVMFS and eventually the main developer CVMFS concluded this was a particular nasty case of cache poisoning, which they have fixed in a newer update of CVMFS 2.10. So people can be a bit it can stop worrying about in place updates. And because it's fixed at this point, there's still synchronization issues if you do in place updates that have MPI processes. These updates should be atomic because otherwise you can have a process that has a newer version of a library on one compute note than on an other compute note if you see the best updates doesn't arrive at exactly the same time. So it's still something you have to be really threat carefully if you do in place updates on CVMFS. Another thing you see when you use a compatibility layer instead of just using the basic RPMs on a cluster is that there's software that is not used to it and particularly you see that with proprietary software from vendors. They've done their QA of course on particular Linux distributions. There's different workarounds we're using. We have a script called setRpaths.sh which basically patches all the binaries to set the RPath correctly so they link to our libraries. And then we have to work around some users. Some use for whatever reason who user lips 64 as LD library path in the BESH RC, of course that doesn't work, we tell them please don't do that. But there are also commercial packages who do these kind of tricks in wrapper scripts that launch their binaries. So then we have to patch the wrapper script of the commercial vendor. And of course there are also users who just download binaries from random places and it happens a lot with Anaconda to a lesser extent with Julia. So they just do condi install much they do on their local computer and they fill their home directory with a ton of binaries. Sometimes that works sometimes it doesn't like in yesterday afternoon. Some people end up downloading a custom open MPI which they use in ethernet network. It's outside of control and it becomes very hard to support. So we basically strongly discourage people to use Anaconda. Our Python page is very much like please don't use condi. It's just so hard to support. And so there are cases where it's really too hard to make things work with our software layers. Okay well we throw our hands in the air we give a used use container. Okay it works it may not be optimal but at least you get your work done you get your paper published everybody's happy. The other thing we're working we have some compatibility layer OS boundary challenges like sometimes gentle prefix is designed in a way slightly differently than what we're doing. They even are set up in a way that they can run demons in the compatibility layer because it's more aimed at local computers where people have have root and privilege like that. So we have a slightly different use case where the demons are all working from the OS level and for that reason the directory says farge should really not be in the prefixes should be at the base level. So there's some things that we work around with symbolic links just simulinking from the prefix to as far you can also patch it in files so it's still working on trying to smoothen out the compatibility layer a bit like that. But other other hand it works remarkably well we have unlike easy again we even have a whole main desktop compiled in the compatibility layers so people can can run can run VNC so they can run VNC server via tiger or turbo VNC on a compute node and then connect to it via VNC even using Jupyter Hub so you can have VNC in the browser window and use virtual TL accelerated graphics even in a window. So there's a bunch of things working with with TN2 that works remarkably well and we just have one main desktop where we don't have to rely on the OS. We even have some SE Linux support like the file system labels it's like on time system in society LS that you have in the compatibility layer doesn't show some information that comes from SE Linux so we have that as well. Then another peculiarity is that we have a funny system where we basically collapse tc core into a system by using a common lib standard c++ library we're using hooks and easy build basically change the name of pool 1.0 tc core 113 we just pool 1.0 so basically change from tc core to system on the fly and it works but it's a little bit messy and sometimes it conflicts with the dependency of the way easy build results dependencies. So I'm still sort of trying to figure out how to best resolve this and and I noticed in the easy build repository there's something called tc core minus system.ev that may be a better solution for us it's been put there a few years ago it severed a bit from bitrock but it may give us a better way maybe using as gentle compile tc instead of an easy build compile tcc may be best because I mean why would you why do we have two different versions of tc 930 at the moment we have one compiled with gen 2 another one with easy build is really necessary to have the two of them so it needs a bit of sorting out an experiment. So that's basically concluding my talk one thing I should say is that the public part of our stack is available everywhere we have documentation linked here you can if you set up cbms on on on the linux laptop it has to be x86 unlike easy by the way but most people still use x86 you can get the whole stack you can load the modules which is nice a chance to research as well because they can get the modules on their own laptop or workstation and thereby get the same environment as on the clusters you just use that testing at their own leisure and so the the what we can do with for instance sharing a bit of work with with easy on the compatibility layer for the future and that's about it for my talk at the moment let's try this you can use this as a mic for the audience here are are there any questions for bars yeah I was wondering how many systems make this cfmfs stack available in Canada you're asking how many deploys is available how do you mean how many freedom wants or well because I was looking at the website I was curious and I saw something like six or seven systems being listed there as being part of the alliance but I thought you had a deployed on more systems now he says a six systems well that part of the that mounts the cfmfs but the more systems that mount it so if we go to the earlier slides we have we have five large national hbc clusters they all mount this software stack this is the primary use case but what we've seen is that the regional clusters or the fundamental clusters they also mount the cfmfs and we actually have a uh a voluntary spreadsheet where we ask people if you use our stack please please tell us that we use it and we found out that we had about 30 entries and and there's some from like really small use there's some individual laptop but there's some some research institutions some universities most about half across Canada and then a few in France a few in other countries that use it for various reasons great thanks any other questions yep sorry maybe I didn't just didn't understand it enough but what do you do with Python packages that are not in your real world post-story then I'm not in our real okay so what happens when somebody just install some random Python packages like if they do clip install numpy they will get the numpy from a wheelhouse because we have it there or tensorflow if it's some random Python package that we haven't as a wheel yet it will attempt to download this source Python package from PyPy if it's pure Python that is never an issue just download the source code it will create the wheel and clip install it so it will it will not install binary wheels from PyPy but we have a configuration file for PIP that basically blocks the installation of binary wheels from PyPy because they're usually not compatible with our stack but it will download source wheels the one exception we have is MPI for Py because that needs to link to a particular MPI version and for that we basically block it from PIP installing it so PIP install will not find it because it refuses to install both from source and the binary package and for that we say please use module load MPI for Py with ORF like a custom module for MPI for Py and there's also some cases like QT we have the Python the corresponding Python package installed with the module so when people do module load QT they will get Py QT on the site as like an extension so it's not it's not completely purely real I mean the fast majority is wheels but there are some things implemented by modules so when you say that if you don't have wheels you try to download the source and then build it yourself does this usually work or are there any tweaks that you have to manually make and how do you handle dependencies in that case if they are defined you know whether in requirements or stuff like PIP file and so on so is that taken care of automatically how do we deal with dependencies so usually it just works like people will just install the virtual environment they start a virtual environment they do PIP install my random PyPy package so as long as they do that the PIP install will just download the random Python package from PyPy and try to create the wheel now if it's pure Python code it will just work it will create the wheel install it boom as long as it's compatible with the particular Python version fine if it has a list of dependencies then of course the dependencies will be resolved mostly dependencies will usually be in our real house we'll use the wheels from our real house the particular version is not on our real house it will of course then download that version from from PyPy and try to compile it and if that uses particular c code for which the module is not loaded then of course it will fail in that case they would tell users okay please write to us with your issue or help you out how often does it happen that users have it's one of the more common support tickets that people have issues with Python packages so yeah I mean it's it's like I mean we can we can handle the load and and like I said given that we have more than 10,000 wheels it's one of the one of the big workloads of our team is just installing those wheels in the wheelhouse so so so Python is an extremely popular ticket topic it's yeah there are multiple tickets per maybe even per day it's it's that yeah and and but most of them are straightforward so maybe small add-on to Dennis this question on the dependencies it usually works but you're you're not controlling which versions get installed right now PyPy figures it out by itself it's going to take the latest version and if you're lucky enough it'll all work but if you do the same installation tomorrow you'll get different versions of dependencies and that yeah you're losing that control yeah we're losing that control maybe users don't care right if it works yeah and I had one more more challenging tickets where the user just for whatever reason tried to combine a new PyTorch with an older NumPy and he just had the requirement of NumPy 193 in his requirements and I said why do you do that and he never answered that question I say but you can't combine old with new that way and expect things to work but yeah so I don't know it's it it it it it then spit out a message of the ABI being incompatible and he said yeah but my my machine learning still produced outputs so it seems to work even though I cut this ABI yeah but I say yeah but you're playing with fire it's it seems to work but it may crash at random times or give incorrect numbers and you just don't know we have a we have a question by atlack as well so on collaboration between the alliance and easy how about running the same reframe tests on both software stacks would that make sense so the easy test suite we're working on that gospel is probably gonna talk about but that's an excellent idea of looking at your your reframe test yeah I'm not sure if they work out of the box but we should certainly do more every reframe testing there's another question they have a basic suite already of reframe test is the osu micro branch marks and the module loads just to make sure that all the combos work yeah another question in zoom how many employees are involved in the software installations um I'd say about 15 are on a on a regular basis installing software or wheels or easy 15 or 20 out of the top of my head so we have like the four rs and p people Maxime Poisson, Charles Coulomb and me and Doc Roberts and then uh and then there's a bunch of people more local you may have seen them in easy build on github as me atlantic paul is oliver stoker oligy if he said alekirachi at our woodcar so so just out of the top of my head I have like five names there so it's and there's probably about 10 others so