 There it goes, okay. Hi Tim Randall's was almost national lab So I'm gonna talk a little bit about our homegrown unprivileged container on time called Charlie cloud Ignore the date. This is not a recycled talk. It is a recycled talk At the same time what I wanted to do is a little bit of a live demo. So I'm gonna be bold. I'm on summit at Oak Ridge They don't know I'm doing this but not really their problem And if you think I just changed the host name of my desk of my Mac, it's not it's a power PC machine So I'm gonna get this started real quick. I don't have Charlie cloud on the system I'm just normal me. I don't work at Oak Ridge. I don't have any privileges there And I hope this works. Okay. So while that's doing stuff, I will switch back and forth So Charlie cloud landals lightweight fully unprivileged HPC container on time Let me Just get this going what I'm doing here is building Charlie cloud in my home directory It really is fully unprivileged and one of the things that we're proud of this year was the fact that we got the ability to Do fully unprivileged builds of containers now So it used to be in the past we told folks use dock or use build or use pod man Use something on on your laptop to build your application But we know that that's not really what our scientists want to do So when we wrote Charlie cloud we focused first on the runtime we said there are other tools to build a container Let's get the runtime solid get it working And then in the past year we've come up with this thing called CH image And it is fully unprivileged image building and management. So there are no set UID helpers When I say fully unprivileged we're using unprivileged username spaces here. So there's no set cap stuff like the UID GID mappers that pod man for instance uses in its rootless mode. It's purely unprivileged It's allowing our users to build their application directly on our HPC systems and all of our environments are open science are our Secure machines and even more restricted environments than that We do have sorry We have registry support. So this thing called CH image will push and pull Kind of need the pull to be able to do the build because you have a from line in your docker file Let me see here. We should have so what I've just done is installed Charlie cloud and I made a typo, but it's there now We'll back out of that and switch in what I'm going to do is build clamor, which is a an adaptive Mesh refinement Hydrogen hours code, but it's a little one. So it's just a test app. Here's a docker file. I'm going to use Should look pretty similar to what you've seen before We're going to be pulling a power PC 64 le image. We're going to use Debian. We're going to do some stuff To install a bunch of packages into that image and then we're going to clone The clamor repository and do a quick make So we're going to use the CH image build command here I'm going to get this running and it'll take a little bit to run Something I mentioned back here is that I did not mention but it's here There's a force option that we have now and what that does is it automatically injects fake root or sudo Not sudo, but sudo the commands in there and those are our system call interlopers that will Intercept calls for things like chone and tell the package manager. Oh, yeah, sure. It's fine That just worked the way you thought it did and it allows you to install troublesome packages like open SSH and System D and some other stuff. So we'll kick this build off and then I'll go back to my slides. You'll see it did a poll here from Docker hub and we're going through all the package install. So while that's going on That I thought this might be interesting to you folks Don't write a tool that knows how to push and pull to registries. It's not fun. It's Convoluted mess This is a little test matrix of what we've tried with Charlie cloud see each image pull and push Some stuff works. Some stuff doesn't people will tell you there's a standard. There's no standard There is a standard, but it's open to interpretation and You know a lot of what I'm showing here goes down to our layers Landl's container strategy. So The our main mission users are part of the advanced simulation computing thing that the ask in the slide ask Integrated code teams and they are adopting containers For all of their future code deployments or application deployments. They are no longer going to build binaries And then put them somewhere in a shared file system until their users to update their their job scripts to point to the new version It's going to be all containers all the time. That's all they're going to use so There are reasons for this they're not hard to imagine provenance being the number one one number one reason for that for the code teams Right now we are heavily relying upon environment modules They don't always control all the environment modules. Some of them are provided by the systems folks at Landl's some of them provided by themselves They also don't like environment modules because it's like it's not 2005 anymore and SPAC exists. So they really want to use SPAC to build their applications And we're doing a lot of work right now to make sure that Charlie cloud works well with SPAC to be able to containerize their applications We see here now we're into building clamor In a container over on summit So this will take a little bit of time to run. I think it uses CMake and Clang or something like that So we're addressing a bunch of requirements One thing is that we're thinking of going to Red Hats UBI for all of our supportive base images that would allow us to Take advantage of Red Hats security scanning and publication of what all our PMs are in the base as opposed to pulling something off of God knows Where like I did just now And it finished. I don't know what's in this devian image, but we want to know what's in the devian image Split brain Presentation is fun so The big benefits we think from the container strategy allows our co-teams to have the total provenance of their applications They control the application in the runtime and all the libraries it was built with It helps our users build provenance for the simulations what Charlie cloud does and I'm not actually going to show This workflow what I'm doing here is pulling it out of my build cache and turning it into a tar ball to deploy it on Summit that just works anywhere the workflow. We're going to be using to distribute applications at scale is to Is to turn this into a squash FS and mount it over luster Everywhere in an allocation kind of like shifter image gateway does Real quick so I can run This I think Ryan had a question. I saw it pop up. I didn't get a chance to read it All this is doing is un-tarrying it into a temp file system You know And then we'll try to run the application Not that one not that one either Here we go. Now. We're just gonna and there we are. We're running the clamor out of the container image Think I'm probably out of time. Let's see what Ryan has. So it's a versioning issue. Aha I'm versioning issue that users have in their respective job scripts Let me think about that for a second The versioning issue that users have today with their job scripts So our job scripts are massive They'll have a module purge and then a module load of gazillion things and then the point at whatever application They want to run that needs all these modules today They don't they're not real good at changing the 15 different module things if this is addressing your question I hope it is Ryan. I should pull up my windows. I can see if you're nodding this or no So a common one was the default version of HDF 5 changed The application gets rebuilt is not yet rebuilt against it So it's your five output and the application breaks when a user just reruns the job script They've been using since the last release the application came out another problem another spin of the same problem is The application is using a specific version module for HDF 5 They tell their users if you want to use this application load this module the user forgets to do it and they're still grabbing the default What we want to do is put all of it into the container so that we're not pulling in things dynamically And breaking applications we know by looking at our past tickets It's really going to reduce the number of support tickets that we get from our users And in our testing of this our most problematic users, which we approach as a friendly user We know that you'd like to try this new technology They suddenly stopped putting in tickets about why did my application die because they didn't have any control over their application They just ran the command that the the code team told them to run That makes sense Tim what's the situation with open HPC support for Charlie cloud? Is it still good? I don't think it's ever been good It was in there when Reese Baird was sort of babysitting what version of Charlie cloud was an open HPC