 Can you guys hear me? It's good? Okay. All right. Today, I'm going to be talking about building an easy build container library in SyLabs cloud. So this was kind of a talk done in just about a week. It's pretty short notice, but I think it will be pretty interesting. I want to get you guys thinking at the end of the talk what we can do to build containers and what would be next steps. So just a little bit of background. Easy build singularity support was first introduced in version 360. And then following that there was support for Docker container. The easy build container is still an experimental feature. And this requires a lot of community testing and feedback and hoping that after this talk, I want you guys to think about what we can do going forward and I'm going to cover a couple of things in this talk. One of the things is my experience using SyLabs cloud and actually building containers. And then I'm going to cover some of the bugs that I've uncovered in the framework that we need to fix. Some of them need to be fixed in the subsequent release. Then I'll talk about the easy build container library with different bootstrap agents. And then talk about the implementation of the singularity template in the framework and ways on how we can improve it. And then actually talk about the container library that I have in SyLabs. Hopefully by the end I can show you a little bit of a demo of how you can build and then we'll wrap up. So first thing, I want to point out these issues. The reason why I'm doing it is because if you go ahead and try to actually do it, you're going to run into some issues. First one is this issue 3135, which was to upgrade PIP and install wheels package in the singularity recipe file. And this issue has been resolved in version 411, which is the latest version. So if you have easy build just upgrade it and that should fix it. Now there are some other issues that were uncovered during the testing, which was one, you can't switch the easy build user inside the container. It's 3171. So I've been talking with the SyLabs folks. What the issue is, I think Kenneth was in there too. We found a solution, so there is a way to fix it, but it's not done in the framework. There are some couple of dependencies issues that I've found in using different bootstrap agents, particularly Docker and library. Library is the bootstrap agent when you push containers to SyLabs. Initially there was a 30 minute timeout. They pushed it up to 60 minutes because I couldn't build GCC on SyLabs, but there's still limitations so you can't build really large containers. They take a lot of time. In some odd cases I ran into issues like SyLabs not streaming logs. You can see this, I failed build, but I don't see anything. So it's really hard to troubleshoot. On the left it's probably a little bit hard to see, but that arrow right there shows a build timeout when I was actually trying to build GCC. These are just a couple of issues I just wanted to point out. This is what I thought about how you can actually generate a workflow for easy build singularity containers. The first thing is you need to generate the recipes. This is already in the framework. You take EB, some easy config like Minutels. That C is for enabling container mode, experimental, because it's an experimental feature. Container config tells you, okay, I want to specify particular bootstrap. In this case, you have an OS version. Next, if you want to build on SyLabs, you need to do singularity remote login, which will ask you for an access token, which you'll have to get from SyLabs. After that you can actually do the remote build. Then the SIF file and then the actual recipe. This recipe is auto-generated by EasyBuild in this format, singularity.easyconfig name with the .eb taken out. Then step four and five. All of this is kind of optional. This is basically if you want to push signed containers. That's the default in SyLabs. What you have to do is singularity key in U-Pair, generate a key, and then you can do a key list to list your key. During step four A, if you select no for, would you like to push the key to key store, which is the SyLabs? You could do a singularity key push. If you say yes, then you don't have to do 4C. After that you do sign on the container and verify. Then you can then push. In this case I'm pushing binutils to this library. That's basically the format of how to get an EasyBuild container built and then push to SyLabs. Now talking about different bootstrap agents. Singularity has different agents. That's basically what a base image used to build a container. The ones I'm going to talk about are specific to EasyBuild. I've only talked about YUM library and Docker because those are the important ones. I think EasyBuild also supports local image and SHUB. But the one that works, or supposedly had worked before I even did testing a couple months ago, was this YUM. The one that you do bootstrap, YUM, and OS version 7. CentOS 7 in this case. I'm going to talk about... If you do the library, you can do it. Bootstrap equals library and then the actual image. There is a CentOS image in SyLabs that you can get. Similarly, there's one in Docker Hub in this case. I'm going to talk about some of the issues that I ran into using different bootstrap agents. The idea is that if you're familiar with Singularity, what's happening is actually changing this bootstrap on the recipe file. When you actually pass this into EasyBuild. Here is the command, building M4 with, in this case, library CentOS 7. This is testing with the SyLabs image. What I ran into may be a little bit hard to see, but the issue is that it can't find PIP in that CentOS image. The issue is trivial, but it's actually not practical from the framework to actually fix because if you let users pick any base image, you don't know what is in the base image. EasyBuild does not have a way of satisfying all the requirements, requirements meaning installing YUM packages, whatever, in the recipe file. Just doing a check right here. PIP is not installed in that image. I'm just doing a PIP version on that library. Now I'll just try Docker because it didn't work with library. I'm trying to build Anaconda in this case. Bootstrap equals Docker and then CentOS 7. After that, when you get the recipe which is right there, you can then do a remote build. I ran into the same issue where I can install PIP. I wanted to try this a little bit more and see how I can fix this. I tried to explore this post command option, which I was going to try to install PIP inside the recipe just to see how things work. It turns out that it writes this command after the PIP. It doesn't work. I can move it over after I generate it. But there's another issue. I'll cover this later, but when you set these commands, it doesn't actually create the EasyBuild user inside the recipe and that's covered in this ticket. I'll cover that later. Then what I did is I tried to copy this stuff from the EasyBuild framework, the template, for creating the user and then creating this directory path where EasyBuild needs to write manually and then try to build. Then what happens is over here, I'm trying to build, I run into a witch problem. There wasn't installed because that image didn't have it. You get the point of these issues that are happening because I'm using different images. The other issue is that EasyBuild is running as root, which is related to it. It can't switch to EasyBuild user inside the container. There is a way to fix it, but that's what I ran into. Some of the things that I've learned is using different bootstrap actually breaks container builds because different base images have different package dependencies, so you can't even get EasyBuild working inside. What works? It still doesn't work to the full extent, but it's the CentOS yum bootstrap. A proposal I have, these are just a couple that I mentioned, is want to be standardized on one bootstrap. That means remove all bootstrap support from the EasyBuild framework at the moment. I think it supports five or six different bootstrap, so container config has bootstrap equals other stuff. Build a base image that installs all EasyBuild dependency with the exception of EasyBuild. PIP install EasyBuild. Right now the recipe generates all of this stuff for you, but I'm saying build a base image. There's another feature that says that users can specify their own template file, so that's just container template recipe. It seems like it may be easier to let EasyBuild generate and then afterwards, users can just edit their recipe file, just because command line doesn't really work with this. Now I'm going to cover the recipe file. That's part of the framework. It starts from right there. It's in the framework. There's a variable called singularity underscore template, and there's this bootstrap, which is the red. What it does is, whatever you said in bootstrap equals, well, that's what this is. Then you have bootstrap config. If you said OS version 7, it prepopulates the mirror URL. In this case, it's mirror centos. And then this includes equals young. In the post section, this is the section where it installs the OS dependency. These are the system packages, which is right here. A lot of it are just generic packages. There's one line here that's about open MPI dependencies that are also installed. We'll cover that also, what I did with this. And then the install EB is where you install the EasyBuild through PIP. Now this screenshot was taken before 4.1.1, so there's still an issue, right? But you'll face this where it doesn't install PIP in here. You need to do that. And then this is where you generate the EasyBuild user account and then the directory. And that's through post command. This snippet right here is a code. It starts on line 326. Well, I'm sure you can't read it, but what it's saying is that if the post command is not defined in the template, it will actually write this content into the template. So it's a weird thing, but I'll show you what I've done to fix all of this. She has a question. Can you go back? Yeah. I don't understand the issue why PIP is missing if there's a Yom install Python PIP there. So in the post part, how come PIP is not there if the package is installed? Oh, no, I totally agree. That was in the other... Sorry, that was in the Docker and Centos. And the issue is because they could be using different Yom repos, so Python PIP is not... That could be one of it. Yeah. Just moving on to the template. So on the right there's just more stuff. This is where you switch to the EasyBuild user and this is where you actually do the EasyBuild commands through the EasyBuild user in the container. Most of it is just setting up the EasyBuild configuration and then running EB robot on the EasyConfig and then update the Unlock Cache. There is an issue with running this command, SU to EasyBuild in that ticket, 3171, and it's going to install EasyBuild slide slash app and then all the other stuff in slash scratch, which is right here. And an issue that... Just to highlight, yeah, we run into this issue where you have this root privilege, which there is a fix to this also. So... And then just moving on, we'll clean up the scratch and then this is where the environment section is set up. It's kind of this little magic that kind of allows us to... What it's doing is loading the modules inside the container and purging everything outside the container, right? Module, force, purge, remove everything, including sticky modules, unused, and then use the module inside the tree, inside the container. So what I thought is we can do something better with this approach. What I did is I extrapolate the base image from the EasyBuild and I actually took all the content and put it into another file. So I have this EasyBuild container here. This might be a little hard to see, but I'll show you what it looks like. It's very similar to... Can you guys see this? So the same thing. So I'm bootstrapping from young. Same exact thing. Post, everything is the same. The only exception is the OpenMPI dependencies. I don't want to have that installed as part of the base. I only want that installed when you're building the OpenMPI container. That makes sense. Then I create the EasyBuild user and then I actually do the same thing as what I do with EasyBuild configuration, but I put it under the the startup files, BatchRC and BatchPro files so that when you switch to this user, you should have this. I've just been extra paranoid, but I want to make sure that both interactive and login works. So it's the same exact stuff. I prefix a scratch, install path as apps, and etc. And then just pre-creating the directories and that's basically the same and then the Lmod stuff. And then we don't do anything else with the other stuff. So what that looks like in terms of the sections is this is the EasyBuild stuff, package dependency, obviously, and then the Lmod. And I've named this container as 1.0. This is what I consider a base image. Once we have this image, the framework doesn't really need to have this implemented as part of this code. It could just bootstrap from this. And then we can bootstrap from this image. Let me just switch over again to the other. So I'm going to show you how you can actually build this. Obviously I wrote some of this. What I did is I just generated the template from EasyBuild and then I had to modify some stuff manually. One thing I had to do manually was bootstrap from the image that I created, then do the pip install of pip and EasyBuild and all that stuff. Notice that it's a lot cleaner. There's not any other stuff that's in there. And then this is the part that's exactly the same. Switching to the EasyBuild user, the fixes, the double angle brackets, EOF, and then write the content and then close it. And then everything else is exactly the same as before. A question here. Can you go back up? The SU EasyBuild and then the commands running as the EasyBuild user, that's a breaking change that Psylapse made, right? Before we could just switch to the EasyBuild user, run the commands, exit, and then we're back as root. And this was discussed on the Singularity Slack. And they don't see it as a bug they introduced, but a fix they did, right? So we were just assuming this was okay while it was actually not. It was working. It was working since we released this in April of 2018, which was when 3.6.0 was released. And this was working. I've built up many containers. It doesn't work anymore. And Psylapse doesn't think this is an issue. They never thought about having to actually switch to user accounts in the post section because everything is supposed to be run as root. And are we just the weird guys using a separate account rather than root? Well, it's also because EasyBuild refuses. You can't configure it to allow you running as root but even in the container. It's not a good idea. So that was kind of, I mean, it's a fix. I honestly am puzzled, so I don't really know what to do, but just do the fix. Yeah, so one of the things there may be a little confusing about the module stuff. What we're trying to do is just showcasing, so I have modules listed outside the container, but then if you list them inside the container, well, we see M4 here, but you don't see the stuff outside the host. So we're trying to fix this by this environment section, which it works to some extent, but we're still going to do more testing. Another issue that you'll find if you're trying to build, like for instance Java, which tries to download tar balls, you're going to run into, like, you know, this JDK is not available. Well, to fix this, just place the tar ball in the host, temp EasyBuild source, and then it will work because we have it bimounted. It's this thing right here. It's binding temp EasyBuild source to scratch source. That way it passes it to the container and then EasyBuild will find it, and then it will work. So if you do that, then it works. Then, you know, just a couple other points. You know, we're getting the Java easy config from the upstream repo, cleaning up at the end, and then generating the CIF file. Other couple things. If you do singularity shell, you can see that you have a module environment inside, and binutils is loaded. This may be a little hard to see, but what I'm doing is singularity run on binutils, and you know, if you echo on module path, you will see the modules on the host, not only inside the container, even though module path is set, because singularity does pass in all the environment variables. So this may be a little confusing, to work through if you've got this thing done correctly, but it's still just something to point out. Singularity passes the whole environment when you shell into the container. Didn't that used to be different? It used to be. You used to have to tell it like these pass them into the container and everything else? Well, there's one that's clean environment that's not supposed to pass in all this stuff, but that still doesn't pass in all this stuff. It still passes in. Yeah, it still passes in. Just if you're curious, I've done ENV in that, and then grabbed for easy build, or not easy build, but module path, it does show you the actual one inside the container. It's weird, but it's something that I'm still wrapping around why it works that way. So the next thing I want to talk about is how you can do easy build toolchain stacking. So, we just talked about easy build 1.0, which is the base image I created. It's on silabs, and then you can actually use it to build basically to build like a FOSS toolchain. That's kind of the goal I want to have multiple FOSS toolchain. In order to build it, we need to have Bumpy, then GCC, then GCC Core, then Binutils, and then we have to go in the right direction. So we have to build Binutils, push it to silabs, then GCC Core, push it to silabs, push GCC. You get the point. And then you can do other toolchains. Once you have all of these toolchains in silabs, users can just bootstrap from any of the images, and build other containers. Makes sense? Doesn't make a lot of sense to have the first Binutils container. Why would you have that in silabs? Yeah. I mean, having one for GCC Core makes sense. I thought about that too, yeah. The Binutils, and probably also not for the GCC layer, because nothing gets installed with GCC. If you already have GCC Core and Binutils, it's just an empty bundle. Yeah, I thought about that too. And I think we could also move. And I kind of agree. I'm just not sure which approach is. The idea is to shorten the time, because in the easy build, you're actually doing EB robot. If you bootstrap, you will save the time. If you're doing FOS, and you don't, and you bootstrap from this, you actually have to build all of this again. So that's the whole point. So this is kind of what I have. I don't know what the font's there, but you can see these are the containers M4. I built them, Binutils, Anaconda, Java, and then the base image. I have this pushed to a repository which I have to figure out. All the recipe files are in here. This is the base image, and then I have everything under apps. And there is an upstream repo that's kind of it's been there, but nothing has been. There's no activity there, but we have to figure out how we're going to push all this and manage it properly. SyLabs, I found out a couple of these details because I have to ask them just some facts about their build infrastructure. So they have a free edition, if you log in and create it uses the AWS T3 machine with a 60 minute build timeout. The containers hosted in SyLabs Cloud are hosted in a library associated to your let's say GitHub user account, whatever. There's no concept of having like hosting containers in an organization. In this case like GitHub organization, something like Easy Builders or like a group library. That would be so that we can share all of our containers in one place. This SyLabs web builder, which you see right here can build containers so you can have the recipe and build. But one of the issues is I don't know how you can pass it in source tar bowls or other stuff in here if you need to build it over here. There is this enterprise support which you have to pay if you want to have on-prem instance of SyLabs Cloud if you want to have full control. So there's a lot of options right now that I can't do in the free edition. But these are some things that you can play around with. I'll show you a quick demo of SyLabs. So if you're in SyLabs and you log in. So the first thing you want to log in at cloud.sylab.io and the first thing you need to do is you need to go into user account create an access token. Once you have that, then you can actually on the command line remote login and then push containers. If you go to the library this is where you'll see the kind of your project kind of your home directory like menu of your containers. So you can actually build like click on any of the containers. This is my easy build container. It shows you how you can pull and sign whatever download. So this is the first feature where you can actually as I can see build a recipe and over here you can actually see a history of all the builds. You can click on one of these builds and it will show you all the content of what's actually doing. Right? So that is pretty much this. There is one other thing for instance keystore where you need to push a key. So I have my key in here trying to sign containers. So now let me just kind of switch over and I'm going to do a live build. So that's what you kind of have to do to get this build working and it will take some time. Now I'm going to switch over to Sylab just doing a remote stream. So if you go here and click on the build builder you should be able to see this build kicking off. Right? It's now trying to install EasyBuild etc. And you can actually see the same exact stuff on the command line. There is this works pretty well. So while this is being built it actually takes pretty it's pretty short. It switches over to the EasyBuild user and now it's actually just generating the SYF file because it builds pretty quick. So just going back this is my last slide but just some things I want you guys to think about hopefully you have seen kind of what I've done. One of the things EasyBuild does is that it automates the recipe generation and container build. Container build that's a container build image. You have to be root for this. You know it should be support both of the features. It's not clear to me right now. Right now I'm just building them manually. But recipe generation is certainly nice. Should we store them in a Git repo somewhere? If so we need to discuss how we're going to do that. We're going to go ahead and publish all the containers. I don't know should we push them in Sylabs or should we have our own remote builder instance? Because you can have a different endpoint. Can we build the EasyBuild containers locally? That's going to be a little tricky because at the moment right now for Java I had to build it locally and push it but imagine if you have to build something locally and then some other image has to be installed then you're going to have some issues. If so which architectures do we support? According to Sylabs you're using T3 machines so whatever maybe Skylake if it's that machine then maybe we should just stick to that. This is not something you can control though right? Not in the free edition but they say you can do that in the enterprise support. Because you're building it on-prem control. With the free version you don't have any control but does that mean you may end up on a different architecture? Well they only have one architecture they only have one. That's one thing. One of the things I'm actually doing in this build you see this line right here? Well it's not done yet but I'm actually trying to get the CPU model and I actually got it on the build so this is the Intel Xeon. Now I made this customization it's not part of the easy build template so I'm actually doing this but the thing is I want to see how can we actually use these? It doesn't work in labels that's the one thing. You can't put in shellcode into the label section. There's something Massimiliano and I should show you for this. We're actually me, Todd and Massimiliano are collaborating on a small library that allows you to track what kind of processor you're building stuff for and you can query it and it will tell you it's Intel Haswell and you can use it as a label for the container. So this is probably what you want what kind of architecture am I on and you can use it as a tag or even in the container name maybe what to expect. And certainly like we have all that detail we need to know what kind of stuff we actually need to query. But I mean I know the starting point we probably agree on certain aspects of the CPU stuff that we need to get. So that actually container built successfully so some other things to think about okay well if you sign containers in Psylabs then who's going to sign it? It's easier for us to sign it because we are a user but if you have an easy build user then I don't know we might have to create an account or we can just leave it unsigned it may be simple that way but but then again I don't know I want to create an official repository something else to think about can we integrate like ebnewpr if we are going to push containers but for recipe files we do something like that easy build base image right now is CentOS 7 but maybe we want to think about CentOS 8 and other versions and should we create some domain specific base images so we could actually fork off easy build with a new repo and then start building recipes and just have a base image and then you can just bootstrap from there that's kind of a naming scheme in the right now is Singularity app version I'm not sure if toolchain is in there but you know it may be a little complicated when you have multiple recipes bootstrapping from different base images what I mean is you have a FOS 2017 easy build 1.0 but then you have an easy build 2.0 the same exact one so that's pretty much it thank you why do you think we should have this bunch of containers in the Sylabs library what would people be using it for actual production yeah to some extent for me I know some other folks have multiple sites so instead of having to build the same exact software for multiple sites build the container once save the time because easy build works inside the container to some extent only the very small short running jobs within 60 minutes work but then the challenges when you're trying to build something like GCC even 30 minute didn't work in the 60 minute I tried with bootstrapping from Benito's and it still didn't work because it was still configured it was still in the range or big stage I believe so another option is we get it locally and then build it it actually builds GCC three times because it does a bootstrapp already more than three times so you can do the GCC installs slightly different to speed it up which may be enough to push it down in the 60 minute range but it'll be tight and also you only have four floors I think yeah I don't know yet but I have to find out something like that so it's very limited for GCC it's going to be an issue but you can do the build locally and just push the base container you don't have to be careful which architecture you use locally I mean it's interesting to see what kind of issues you run into and the improvements that have been made this year are I'm well aware they're only for singularity not for Docker so I'm not surprised that the Docker support is pretty broken right now because I haven't used it since a year and a half I think yeah I'll show you some other stuff so yeah this container that you have if you shell into this container you see that BZ2 is loaded in there so you can directly use it and it's a module environment setup for you some other things to think about well the containers automatically have like sections in there like for instance the recipe section so if we need to for instance if somebody builds a container we need to find a way maybe they forgot well I think there is a inspect-d it will get you the recipe from the container so it could be some potential way ideally we should have something like EV and UPR for this or something like that but if somebody builds containers in their own place they should have it so that they don't do this it should just be in a repository the reason is like when I'm building this container sometimes I'm actually deleting the container and rebuilding it because I find a bug yeah so you could actually get a lot of this like you could do inspect that will get you the run section and then you know the environment section you know dash sheet so like all that stuff is there it's just a matter of kind of get this detail somewhere in git how are you going to solve the how are the dependencies different because you know how about can you set the city model to your style with AVX no I don't have that but I think you can fix right I mean I think it's also you know does Singularity support multi-architecture container does I know Docker does sorry does Singularity support multi-architecture containers so Docker does so you can have like one single label with multi-architecture inside I don't know if Sylabs or Singularity supports it yeah I don't think it would I mean at the moment I don't think there is a feature to do that and honestly this label section which is this section thing out I thought about putting this stuff in here but the problem is this thing is actually it's a JSON file so you can just like I was thinking about okay maybe I could just update the JSON file to put it in but I need to do it through shell functions so I was thinking maybe I could just put it in the post section and just update there's a file in the dot Singularity that has this stuff in there but it's not clear to me what the right way I don't even think it's going to be labels are not used for anything they need to be used for something like part of run, Singularity exact run where it needs to be part of there and it catches the labels to say if you're not in this architecture it should just stop Yeah or it just stops you know that could be part of like an if condition in the run script I don't I do have another question regarding your base image is it build 1.0 why don't you have is it installed in that image oh yeah the reason is because then you want to keep up to date with the easy configs easy build framework and everything because that's always going to be updating if you put it in the base image then you're stuck to that version at point in time so if I do it now I'm not one and then if I bootstrap from future I mean yeah certainly we could do an upgrade but there's no point of putting easy build in that base image because it's not going to install anything it's just to set up the environment the other stuff is actually going to install you get the point right like if I'm going to build bzip I need to still install the latest version if I use the old version then you don't even get it just another question so you're loading the module in the post it's the environment section so is there any reason to have modules in the container so from a user point of view I just want to run the container and have all the software that I installed in the container loaded and available so do you load everything that is installed in the container yeah well not exactly so I'll show you an example of one I think bzip installs multiple containers I mean easy config so if you do so yeah this one you only load binutils so would it make more sense to have a profile where you keep adding all the variables that lmod adds when you do an ml load when you load the module just add this to the profile so every time that you execute the container you have that environment already loaded what's the advantage that then you can do a singularity run of the container and just the application without having to do a module load before the modules are loaded by default in the environment if you build a container in this case only binutils are installed so I'll show you I don't know if I can we're not feeding a build let's say I install in singularity a couple of softwares that are not dependent on each other I want to have both of them loaded when you so this is the actual recipe file if you're under that easy build environment what's happening is in the file system tab this is the file in singularity container that's actually writing this section so you can see we're sourcing NC profile and all this stuff what easy build does say you build a container that has openfoam and gromax like the eb openfoam gromax build container it will load both modules because that's what you ask the easy build to build so it will do a module load openfoam, module load gromax but it will not do it if I have a base image with gromax and then I do easy build install openfoam on top of that and probably you don't want to install both softwares at the same time because you will run out of time in setups so you may have your tool chain and then keep adding softwares on top of that with different installations so from my point of view it would make sense to just run the lmod command not the ml but the lmod command and put the output inside this environment file then you have all the variables and you don't have to run module load at the beginning which will also speed up the spawn of the container so we're still installing lmod just because of easy build we're not using it inside so it seems a little seems a little bit odd but okay this is a bit odd but a normal user will never look at the environment anyway you have to just shell into the container start using the thing and it will work and they'll be happy so they don't care how it happens if it helps with the startup of the container or if it gives you more flexibility in terms of what gets loaded or activated in the environment then fine, yeah, makes sense yeah and the other thing is right now the easy build base image I built it with lmod at that point in time and it's actually lmod is installed in that base image so I know because easy build depends on lmod we may want to think about that as part of a base image where we say easy build 1.0 is tested with lmod 8 in this version and then going forward you build a new base image and then go from there now because of the base image that I've done we can scrap out a lot of the code in the framework and all you then have to do is bootstrap from that image now where that image resides is also a matter of where we push it so there's all a lot of these things are just kind of related to each other to get back to Carlos's question on why you're not adding easy build in the base container so you want to follow the latest easy build releases basically right you do a pip install easy build and the image that you build on top of the base so you get the latest version you have to be a little bit careful there if we ever get to easy build 5 we'll make some breaking changes in there so if you then rebuild the container image what you were doing may not work anymore so maybe we should change the pip install easy build to pip install easy build smaller than 5 just so we stick to the latest forward version and that way it should not break even if you rebuild it 10 years from now that's probably wishful thinking but you don't want to auto update to easy build 5 when easy build 5 is there because that would or that may backfire any more questions for Shazab if not then I guess that's a wrap