 Welcome to another edition of RCE. Again, this is Brock Palin. You can find us online at rce-cast.com You can also find us on iTunes as well as an RSS feed and get the entire back catalog there. I also have here Jeff Squires from Cisco Systems and one of the authors of OpenMPI. Jeff, thanks again for your time. Hey Brock, it's the middle of the summer. School's out. You're just I imagine slacking off for the entire summer, right? I mean that's that's what all you academic researchers do during the summer. Am I right? I'm actually trying to replace our entire cluster by sometime next year, so I got a big big project coming up. No, no rest for the wicked. How terrible. No vacation for me. Okay. What are the deadlines we got coming up? Isn't there's a super computing boff deadline is coming up end of July? I think yeah, I think there's a couple super computing things, but I don't think there's really anything else that I'm aware of coming up So I think that's the only one. Okay. All right. Well, what do we got today? Okay, so today we are going to be covering shifter. This is a often requested topic and we finally got a hold of some people. So we have with us Shane and Doug. Shane, Doug, why don't you take an opportunity to introduce yourselves? All right. Hey, I'm Shane Cannon. So I'm a member here at NERSC at Lawrence Berkeley National Lab. My background is my educational background is in physics. So I've got a PhD in physics from Duke, but all my professional career has been in high performance computing and technical computing. For everything from system administration to being a group lead at Oak Ridge and a group lead here at NERSC. And right now I'm in the data analytics and services group where I work on trying to help users use our systems more effectively, especially on the data side. And then I'm also a member of a project called Keybase, which is a genomics platform that we're trying to build up. And hi, I'm Doug Jacobson. I also work at NERSC at the Lawrence Berkeley National Lab. My educational background is in bioinformatics at the University of Michigan. But I've spent my entire professional career here at NERSC, ranging from user services to systems administration and systems software engineering, which is where the shifter project really comes in. Okay, so where does what is shifter? Maybe I'll take I'll start off and Doug can chime in. It's maybe useful just to think about why we invented this. What we wanted to do is we wanted to we've got a growing number of users that are trying to use systems like Corey, which is our Cray system to do a broader set of things than just modeling and simulation sort of the traditional HPC workload. So we want to enable more users to use the systems more easily. And we saw containers, Docker containers is an interesting way to kind of open that door, but you couldn't really run Docker on our system. So shift was really a response to that problem and trying to think through the issues with Docker and but still trying to enable as much of that capability on our systems. So at you know what it does is it allows you to run containers on our on our systems and other HPC systems at scale securely without you know some of the side effects of Docker. Anything to add, Doug? Yeah, so this has also been not a recent sort of investigation for NERSC. I mean, NERSC has been in our non capability machines. So so things like our traditional cluster systems we've been doing forms of virtualization for quite a long time starting with Shane's projects. So really we've been dabbling in this kind of thing since 2005. Yeah, for a decade or over a decade. Yeah, but but shifter is just the latest incarnation of that. But it's the first one that allows the user to directly control the every aspect of the environment. I would like to clarify though that shifter is not actually a container system. It's a container compatibility system. Yeah, we don't do anything. So, you know, a container would would imply both sort of the operating environment, which is really what shifters thinking about. But it also thinks about resource limitations. And that's really the job of the resource manager. And so actually one of the design principles of shifter is that it does not at all try to do resource limitations. It leaves that to in our case it's slurm. That's our workload manager. But so we really don't do a whole lot with sea groups with shifter and that's by design. Yeah. All right, so Shane, let me zoom in on something you just said there that was kind of interesting. So this is not a shifter is not a container system in itself and it's in addition to Docker. But you made this interesting comment that you can't run Docker on your systems. Could you give a little insight into why and what some of the problems are? Yeah, sure. And it's kind of multifaceted. So probably the main the biggest reason is sort of security concerns. So if you're trying to run stock Docker, if anybody's played around with Docker and sort of try to do this, you can once you sort of allow a user to use Docker, then you've kind of given them the keys to the kingdom on that that node or that system. Right. And so since we want to allow users to use Docker images, but also be able to access things like shared file systems, it really would be a non starter for us to just put Docker in an altered and just open it up for users to run containers. There's also other problems with in a kind of HPC environment that we're running on the craze. There's no local disk Docker. You can work around this, but Docker really assumes local disk and that would be problematic. We really want to make sure that the solution that we put forward can scale to the full size of our systems. And so there's constraints around that. And then we also wanted to make sure that we could allow we could run it on systems maybe that didn't have the most modern kernels on them. So some of the systems can be a little you guys know the kernels can sometimes lag in the HPC space. And we didn't want to have this dependency like you had to be running the latest version of the kernel to be able to use the system. So, you know, you just told us what shifter is. I mean, why containers in the first place? Why is this a useful technology for science? Yeah, and this is I think this is good because this is something we're really excited about. So first and foremost, I think what's interesting about containers is they really give you a framework for enabling reproducible science. So, you know, one of the challenges in technical computing is you're running your application, but things are changing out from underneath you. A lot of times as the systems are upgraded or modified or changed with a Docker container, especially if you have a Docker file that you're using to build that that image up. You've got this auditable recipe of like this is how I created that, right? So one, you've got some that's auditable to once you've built it and and you've registered and say a Docker hub. Now you've got something that you can always go and pull back down. Others can pull that down and reuse that same image. So you really start to make it easy for someone that you can say here's the analysis I did and then that this other user from the community can go pull that image down and run the exact same thing that you did. And then and also you can do that too. So a month later, a year later, if you want to go back and reproduce those results, you just pull that image down and do that. So I think this is really powerful. And this is why I think it's going to be it's going to grow in adoption within the scientific community is more and more people understand what what this enables. So where does the name shifter come from? I'll let Doug take you because he came up. We were like, what name should we call it? Well, so actually, we had tried a lot of different things. And one of basically, I think that Shane originally posited that we should call it a Proteus, which is a shape shifting. Yeah, or something else thinking of some mythology kind of thing or something like that. And I was having a difficult time moving off from that. And so I decided to sort of latch on to the shape shifting aspect of it. And that's where this shifter comes from. Yeah. And so we talk about like sort of shifting into an image or an environment. And so that's sort of the idea there. So of course, a big question for all these container-ish type systems, whether they're relying on one or implementing their own is do you require root permissions to set this up or use it? Yeah. So the short answer is yes, we do. So what we do with shifter, and I should make this clear that these are very conscious decisions we made along the way. We tried a lot of different things. As Doug mentioned, we've been doing this for... Can we back up for a second and restart the question? Because I think what I'd like... I think... Can we clarify the question? Are you talking about, does the user need root privilege to introduce a container onto the system? Or do you need root privilege? To install the software. To set it up or to install the software? Or let's just clarify what you're... Yeah. Sorry. Actually, I was asking both. Okay. All right. So... Okay. So yeah, shifter does require some level root privileges. So the short answer is yes. We went through a lot of different iterations when we were developing shifter because like Doug mentioned, we've been sort of doing this for almost a decade now. So let me explain where we need privileges in line. So one of the... The main thing is the binary that does the runtime construction for the user, it needs to have root privileges. So it's a set UID binary and that's so that it can do a loopback mount of the file system. So what we do as shifters, we take the image, the Docker image, we unpack it, we repack it and squash it in, like I say, a squash file format. Place it out on the parallel file system. And then at runtime, when the user wants to use that image, they'll... We mount that up to a loopback mount and so that requires privileges. The rest of the operations don't. So we will drop privileges once we've done the things that require it. So at no time in any part where the user has control, do they have privileges? So that's why we can do things like allow them to access their file systems, access the parallel file systems. But there is this caveat that we need a set UID binary just to perform that main operation. Okay, so just to drive that point home. The system gunk on the back end requires some root level permissions to set things up. But from the user's perspective, they just say, use my container and it's running as them and they never any see the root level gunk setup stuff. They're just their user outside the computer and then their... I'm sorry, outside the container and then their user inside the container, no root required for them at all. That's correct. This was really important for us, right? Because we're running a system that's got thousands of users on it. We can't allow a user to potentially access somebody's else's data, corrupt it, whatever. Obviously we want to secure the systems too. So similar to the way, say a resource manager kind of has to do its job. It needs privileges to be able to set up the run and then move into that and switch to that user to start the job up. That's kind of what we're doing here with Shifter as well. But then also the user, but because of some of this and the way that Shifter is designed, this is one of the major design points of Shifter is that at least in the way that we've implemented at NERSC, although their Shifter gives the site a lot of flexibility to configure Shifter however they want, there is no need at the NERSC site at least with the way that we've configured our Shifter installation for a user to do something like file a ticket or something else to get a new container on the system. So long as they put their image out on Docker Hub, they can at any time pull down that image and start running with it under their own control. Yeah, that's right. Now you also mentioned that Shifter gives you some scalability controls. I think we've got the security part down. What kind of scalability advantages do you get? Yeah. Well, so this was actually, it was by design to be sure, but it was not actually, performance was not one of our initial goals with Shifter. It was mostly about productivity. Yeah, scientist productivity. I want to build my environment and then produce it. But the choice of how we pack the image and then put it at least on to our parallel file system and we've done a lot of tuning to get that just right for our installation. But then using this loop device mount, it turns out that that optimizes out one of the most challenging aspects of delivering software in very large HPC environments, which is that, for example, on our system, if you wanted to start up a simple Python MPI application across, say, a thousand nodes, we've seen that take as long as 28 minutes if it's done that naively. But running that same code through Shifter, it only takes seven seconds to launch. And the reason for that is that by distributing the image and locally mounting the image, it doesn't copy the data to the compute node, but it copies the metadata, the file system to the compute node. And so as Python is starting up and walking its hundreds of directories and files and everything in Python paths, so long as it's within the container environment, it gets much faster access and doesn't need to get locks from the central files and all the other stuff that goes along with it. Yeah, you think about normally it'd have to go to something like, in our case, we're running a luster on these systems and we also run GPFS, it'd have to go to those metadata servers to start accessing files. Really, you just have to do one look up to get the image, then you mount it into the kernel namespace of the compute node and now everything's in that sort of local kernel space, so it can be very efficient. We mount the image up read only, so that has some downsides to it, but it does allow it to scale more efficiently. And then there's some other things that we've done that we integrate it with the batch, we can integrate with the batch system and when that's done, that can give you some additional scalability features. In that case, what happens is that all of the processes that are started by the workload manager can basically share the same namespace, so we don't have to repay the overhead of setting up shifter again and again and again. Yeah, which is still low, but it does add something. So how do you handle the file systems and mapping in user names and stuff like that? So you said that you're handling it, you're the same user inside the container as outside, but how is that actually accomplished? So the shifter executable, so the job of the shifter executable is to sort of launch a process inside the container environment. And what it does is it actually collects all of the user information credentials from the external environment. And then it just, when it drops privileges, it assumes all the same identities. So there is no change in privilege outside from inside the environment. One of the things that we do with shifter though is, you know, we said that we bring in things like the shared parallel file systems. Now clearly, the user doesn't have those mount points probably in their image. So a big part of what shifter does is it actually, at runtime, edits the container that the user has presented. So the shifter does not give the user the container that they requested. It gives the user a version of the container that they requested based on site policy. And so the site can set up the shifter configuration file to say, well, this content needs to be copied into the container or these mount points need to exist and these file system mounts need to exist. And then that's really what the shifter executable is sort of legislating is how the container is set up and how it's expressed to the user. So once you're in the container, whatever user was that requested a container, you're that user. And so there's really no additional mapping that needs to be done in that case. So this probably wouldn't happen, but say they're executable in all the environments inside that container and when that container was built, it was built under a UID different than their own and it does not have world permissions on it. Do they basically get a permission denied when they try to invoke it on a machine that's enforcing these policies or does it do something different? Yeah, that's correct. I mean this is and it's probably worth noting there are because of these, some of the things that we do in shifter for security reasons, it can call some incompatibilities with a Docker image if it's not constructed quite the right way. So for example, I've seen cases where people have the software is all installed as root. It's put in the root home directory and then they go to try to run that container and they're like, I can't open access these files. It works when I run it in Docker. So luckily, these are very easy to work around. Generally what we say is, okay, just make sure you make any files you need to be able to access. Just make sure those are read world executable, world readable, and then everything pretty much works. We generally recommend people put the applications in standard paths. You might as well. It's your container. You don't have to put it in some oddball path. So sometimes people need to do a little bit of work in the image to get it to be compatible with shifter, but usually those are pretty straightforward. One of the things that we feel is that a container that's done well for this environment really should not require a complex environment at all. If your path is more exciting than bin colon user bin and your LD library path has to be anything other than empty, it's probably have made it more complicated than needed. So really like a common complaint where people are like, I can just yum install this or set everything up the way I went on my home machine where everything's in user bin, user local bin and everything just works. I should do that inside my container image and then I can have that same experience anywhere. Yeah, that's exactly right. And I think part of the problem is people just aren't used to thinking that way, right? They've always been constrained so they don't think about like, well, I just should make it look the way I wish it would look. Okay, so one of the things that I understand about Docker and please correct me if I'm wrong, is that when you have this Docker specification file, it actually goes and builds your container on the fly to potentially include like what Brock just said, like yum install, whatever, probably open MPI because all right thinking people use open MPI. So does that happen on your system too and does that happen on each compute node? Like is that distributed or is it centralized? How does that happen? Yeah, so maybe it's worth noting here that first off with Docker, there's kind of two stages to the Docker lifecycle, right? One stage is you build an image and that's where the Docker file that you mentioned sort of comes into play. That's a recipe for how this image should look. So it says things like inside this build environment, add these packages, you know, set these environment variables, you know, run these commands. And then if the result of that is an image, right? And then you could push that to a registry like Docker Hub. Now what, and then the second phase of Docker is you run, you know, you run an instance of that image that's a container, right? So what we're doing in shifters, we're trying to leverage as much of that as we can. So we don't want to provide a bunch of tools to build images because there are already good tools to do that. Docker provides those. And we don't want to build our own registry because Docker provides that. So what we do is we have a special component that's part of shifter where it will pull down the contents of the image, which are really just a bunch of tar files. It unpacks those, repacks them, and then squashes them into a single squash file system file. And then that's placed onto the global file system for the HPC system. And so you only incur that penalty once when you first pull the image. Now it's packed and ready to go. Whenever you start up a container using the shifter runtime, it just mounts that image in on the compute node. So there's no additional translation or conversions that have to happen. The user is able to cause an image to be pulled. So they get control when they build their image. And they get control to say, I need this particular image on the system. But it's entirely system software and system administrator configured software that is actually handling the image itself at that point. So the user doesn't ever actually have direct control over the image once it starts to come onto our system. Yeah. So we're doing a decent bit of work with Docker here at Michigan for supporting things like persistent databases, data science applications, and others, not on our big shared resources, but persistent services we provide. And a common thing we ran across is friends don't let friends use loopback. And I've noticed you've mentioned that a couple of times. What is different about this sort of environment that you consider loopback to be okay? Obviously, it works on pretty much any Linux system out there. Is it because you're not doing any IO internal to the container? Everything is mapped out to these external file systems? Yeah, I would say that's the primary reason. If we had local disk, perhaps we could copy it on and then mount it up as a native file system or something like that. We think in general, the loopback trick has been great for us. We've used it in a number of ways. We're seeing it even get picked up outside of Docker kind of use cases now. So it's something you have to be cautious with, but I don't think it should be feared. Well, and really the question on this is what is the rate limiting step that you're suffering from? And what we were, the problem that we were having was metadata operations traversing the parallel file system. So by using the loop device, we actually now make the loop device the rate limiting step, which even then it's still more page transfer than anything else to and from to get the data. So the loop device, sure, it causes a penalty of a kind, but the only place it's really causing the penalty is an application startup. And application startup is not where you should be spending. It's not where ideally an HPC application should be so you should be spending most of its time. Any sort of persistent read and write operations should not be targeted at the container. In fact, we prevent that by making it a read-only file system. Instead, those should be obviously directed to more traditional IO mechanisms, MPI or shared memory or even the parallel file system itself, which is tuned for being able to accept large amounts of reads and writes very efficiently, but not perhaps these locking operations of metadata. So you just gave me the keyword there to move us to a slightly different topic here. How do you handle these OS bypass networks and HPC class networks, whatever you want to use the phrase here. How do you handle MPI and all the extra special things that it does inside of a container? Does that present any problems as opposed to bare metal? So I think when we first started developing Shifter, we were really targeting these high throughput type applications, more data intensive applications, but at the same time we wanted to be able to use this for MPI style and traditional simulation type workloads as well. So we started looking at ways to do this and there are a couple of different ways. One, it's important to note, we just passed through the device file system. So inside the container, you still have access to those devices. You're still that user, so you don't have any extra privileges, but you can access them. So the other key thing that you sort of touched on that you have to deal with is, well, you know, a lot of times these are, the hardware is very specific. It might have a version of firmware on it and you need to match that to some degree in your runtime, right? So one model would be, well, you just have to keep your image up to date with the system you're going to run in. So if you need a version of, you know, RDMA libraries to match your OFED release or something, you've got to make sure those are in the image. And that's one model. We have another model that we support in Shifter that Doug really figured out the way to work this where we place the libraries, we take advantage of ABI compatibility, for example, and MPI. We put the libraries that are sort of part of the system and match to the hardware in a location on the system and then we volume out those into the container and set the LD library path so those get picked up at runtime. So that way they're automatically mapped in and used, you know, by the MPI application, for example. Let me make sure I understand what you just said there. So you're saying that, like, I have a container and I'm running, I don't know, Red Hat 7.4 and you're saying that you have, like, an MPI installed for Red Hat 7.4 or 7X just taking advantage of the ABI for Red Hat 7X and that gets mapped into my container so I don't have to do anything for it. Is that what you mean? Almost. What we would do is we have some example containers that already have it sort of set up the right way but I'll be frank with you here, Jeff. We have this working for MPI on our systems. We haven't yet had it work for open MPI but there's no technical barriers other than just some details of the Cray systems and sort of the state of the fabrics. But anyway, what we do is the user in their image would have a build of MPI that has the, it's a sufficient level to have the ABI compatibility and then the only thing that needs to get volume mounted and mapped into that container space is just the libraries, the correct libraries for, say, the Aries Interconnect or the Infiniband Interconnect of the system. So we accomplished this with Shifter's ability to inject content into the image. Yeah. And then it's really just we rely on the linker to do its job with dynamic linking. And so this is sort of, at least at large scale, HPC, this is sort of a novel concept using dynamic linking. Most of the state of the art has been static linking for a long time or statically produced binaries. And so we are actually taking advantage of dynamic linking to provide this compatibility. And this is actually itself enabled by sort of that earlier observation that we can now traverse large amounts of files quickly in application setup with the way that we do things. The thing to keep in mind with the type of system that we're working with relative to perhaps other systems is that we don't have any local disk at all. The entire OS itself is on a remote disk on these compute nodes. The types of pressures on the system are a little different, perhaps, than a locally installed cluster would have. So are GPUs basically handled the same way? Yeah, that's right. And at Narisk, we mainly have Xeon nodes and nice landings nodes, KML nodes, or files. But we are another partner on this project is CSCS. And I think it's important to call them out. They've been a strong proponent and they've contributed to the project. And they have large, as you know, PISDON has a very large GPU system. So they've been doing a lot of work to get GPU support within Shifter. And they've demonstrated that and they have most of the pieces worked out. We're in the process of getting that. They have their own fork of it that they've been doing a lot of that development on. And we're in the process of trying to integrate that work into the upstream branches now. But it uses the basic similar concepts as what we're using for MPI. So let me, again, make sure I understand this is complicated stuff here. So is the idea that your containers are really just mapping the file system? Because there's a bunch of different namespace types in Linux, right? And the word container is actually somewhat amorphous. Most people usually mean Docker containers, but there's all kinds of different flavors of them out there. Are you using all of those namespaces or just the file system ones so that you don't have to do anything special to get devices? Or how could you explain that a little bit? Yeah, so really, Shifter is not a container solution. Shifter is a bunch of file system mounts and a charu. Yeah. Or a namespace call. Yeah, with a bunch of logic to manipulate the target environment. So in our sort of shared space version of Shifter, where you might have multiple jobs or in the same node, we do use the mount namespace for that version. But for a node-exclusive Shifter where you want the container to be somewhat re-entrant and things like that, we don't even use that. Yeah, there is just the charu. Yeah, Jeff, to your point, we mainly use file system namespaces. We don't use any of the networking-related ones or process-related ones. Or at present, we don't use any of the user namespaces as well. Largely because we don't see a need for most of those in HPC environments. Yeah. So you've mentioned Kray systems a lot and Kray has special runtimes and everything else. And I also know CSE has a lot of Krays. So does this require a Kray runtime environment or can this work on classic X86 clusters? Yeah, I'm glad you asked that because that is a misconception. I think a lot of people, because we partner with Kray in some of this work, they think it only works on Krays. But no, we have examples out there of people. We're running it ourselves on some standard clusters and then there are other sites that are running it on regular clusters as well. Okay. So then my next question is a follow-up kind of getting back to the GPUs and to MPI. What about shared memory applications? When I've seen a lot of these sorts of systems, you still use your job launch system, but they tend to still do, you know, process per core. You know, how do you handle launching one of these things where you want to do a threaded or an open MP application? Yeah. I think, I mean, the brief answer is we're not doing shifters, mostly manipulating the file system namespace and some of the environments set up, you know, like environment variables. It's not doing anything to monkey around with some of these other things. So shared memory space, as long as the user has access to it, they should behave the same way. Yeah. We port through all of Slash Dev. We don't do anything to obscure Slash Dev's Shemem. Unless there is a very exotic implementation and even then the site could configure shifter to bring in, so actually that is the case for a craze system, is that there are a number of paths that need to be brought through to make craze XP Mem stuff work properly and craze sort of argument credentialing system work correctly. And so long as those are exposed within the container file system environment, everything just works. Okay. So still just use your resource manager to launch shifter the same way you would have had at launch a process that you ported directly natively to the platform. That's correct. Yeah, that's right. In fact, when you run a shifter container image, the way it looks is, you know, in your batch job, you would say instead of saying, for example, Arizona's Slarm, you might say, S run, you know, binary name. In this case, you just say S run shifter binary name and that's pretty much the end of it. So a couple of episodes ago, we talked to Greg Kertzer about Singularity, which is another HPC based container project. I wonder, could you compare and contrast shifter and Singularity to explain a little bit of the differences between them? Yeah, sure. Yeah. And it's kind of ironic because both of these, Greg, he's no longer at Berkeley Lab, but when he was developing Singularity was and we're at Berkeley Lab too. So it's a little bit odd that two different solutions came out at the same place. But we actually had started shifter work quite a while back and then they were different. They were more different at one time. They've slowly kind of converged. So I'd say if I had to sort of say, what's the chief difference between them? I think the biggest one is shifter has this concept of an image gateway that we sort of maintain a central repository of any of the images that have been pulled in to be used, right? So any user can pull the images in but they go into a common location and the system maintains control over those images over their lifetime. So we think this is important because we've even heard of cases in some of these others where somebody will accidentally corrupt the image after they pulled it and then they go to run it and it'll break or it might break once they're already running. Since we sort of maintain control of that, we make sure that we never overwrite an image that's already been successfully pulled and registered. So I would say, you know, the fact that we have this central repository and the fact that we maintain control over the images is probably one of the biggest differences. Some of the other things is we've done a lot of work to integrate this with batch systems. So, you know, here we're running Slurm. We've got really good integration with Slurm but people have done it with other batch and resource managers as well. I think singularities got some examples of that now with Slurm as well. But, you know, we've been doing this a little bit longer and there's a few other kind of minor sort of feature differences that I think are intriguing. One of which is we can allow the user to create a file, a per node file that can get mounted into the image on the compute node that's writable. And so this is not visible across all the nodes, you know, that might be using that image. It's just local to that node. But this is a good way to kind of emulate local disk type performance in a system where you might not have any local disk like we do on our craze. So that's a feature that's actually pretty powerful that I think is still pretty unique to Shifter. One of the things you're talking about was the ability of Shifter to be able to take, you know, IOP intensive startup applications like, you know, an MPI for PI applications and things like that and really scale up some of these systems. So what's the largest job that you know of that's been started up under a Shifter environment? So actually just last week, and we got permission from the people that did this to talk a little bit about it, they were able to run on 9600 Intel K&L nodes with Shifter and they were running a Python wrapped application to simulate their next generation. I think it was the cosmic microwave background. And so that was a single job run through Shifter. It was an MPI job that ran with over, I think, 158,000 ranks across the system. That is, I think, the current largest and that did real science to be able to simulate enough data to really sort of plan out their next generation data collection capability, which is, you know, I think they said the Planck satellite had 70 data collectors and their next version is going to have 50,000 data collectors. Yeah, it's like 35x the data output or something on Planck. So we've done, you know, dynamic benchmark style runs ourselves to kind of make sure things are working correctly. But this was probably the biggest real science example that we've come across and so it's pretty cool. That's just in a single example. What Shifter's also done on our system is it's enabled entire new workflows that now just run in production all the time. And so if you were to sum up the compute time dedicated, I would say that the high-energy physics community has probably benefited the most at NERSC functions. Yeah. Because they're able to now package up their applications, have it work, excuse me, other applications. They're a higher framework of software and those images tend to uncompressed when they exceed over a terabyte in size. And they're able to package those up and now make you full use of Cori to do their work. So those are the kinds of things that we really wanted to do with Shifter's. Enable that. Now, by the same token, and this is a question we ask a lot of our guests on here too, is what is the strangest use of, or the most unexpected thing that you've seen Shifter used for? And perhaps this is something outside of NERSC because you're using it a lot in production, but have you gotten a strange user request from someone saying, hey, I'm using Shifter this way and you're like, wow, never thought that would happen. Yeah. I don't know if we've got any things that are just really absurdly strange. I think we've seen ones like Doug mentioned where the LHC folks are using it. And this is kind of what they're doing is if you've ever heard of CVMFS, this is their distributed kind of file system that the LHC folks use. What they've done is they take the whole distribution of CVMFS and create one big mega-squash image from it and they don't push it through Docker because it's just too big. We have mechanisms in Shifter to allow you to directly import that and sort of register it in the system. So that's probably just by sheer counts is one of the stranger ones. But I think as more people use it, we'll probably come across some more oddball uses of it. To be sure, we don't know all the use cases. We can certainly see a list of all the images that people are using and there are many users accessing Shifter on the system today. Mostly they seem to be not filing tickets. Yeah. We're not always aware of how they're using it. Until after the fact sometimes. So what license does Shifter develop under? Yeah. So it is an open-source project. It's up on GitHub at github.com slash nurse slash shifter. And it's under a BSD, a modified BSD license, which is the labs kind of preferred choice. And we think this is a good license choice too because it's easy for if vendors want to kind of pick it up and maybe support it themselves, there's more latitude there. So what are some interesting things that are happening in the Shifter community? What do you have coming up in? What new features do you have coming up in new releases and things? Yeah. We've been trying to get a new release out for a few months now. Doug has been distracted with a small 10,000 node plus system that he has to help support and make operate. But we're hoping to get a release out very soon. Some of the things that that release, we already have these things working, so we're just really trying to get it buttoned up. But that adds private image support. So there are cases where people have images they don't want to make fully public. And so this will add support for that. There's some metrics capabilities and built into it now. Number of runtime sort of enhancements. So that's right on the horizon. Hopefully within a month or so or less. Also, it adds the GPU. It'll be adding the GPU support that CSCS has been really instrumental in. The other thing that further out, what we're looking at is we want to start leveraging layer FS support to do some more clever things. We mentioned that the nodes are, the images are mounted up read only on the nodes. And with the layer FS, we think we can provide some read write capabilities as well, which will make it a little more flexible than it currently is. So those are probably some of the bigger things. Oh, we're also working on more integrated support. I mentioned these cases where we want to import images that are maybe already prepped. So we've got some work underway to try to make that a little more streamlined and user friendly. Okay guys, thanks very much for your time. Where can people find shifter, download it and get involved? Yeah, so as I mentioned, probably the best source is to go to the GitHub site. If you Google, there's some pages on the NERSC website as well. I forget the exact pass, but if you Google it'll show up. And we do have a mailing list that people can subscribe to, as well as if you're really going to get involved in development or something like that, we have a Slack channel that we can invite people to help contribute. But the model is similar to how other open source projects work. So we can fork it, make a PR against that, and we'll review it and hopefully incorporate it. Okay, thank you very much guys. Thanks for your time guys. Oh yeah, thank you for having us.