 Well, thanks for coming to our talk. This is Images Everything, about dynamic HPC repositories using Merino. I'm Robert Budden. I'm from the Pittsburgh Supercomputing Center. And before we start, kind of give a brief background of what our machines are and kind of how we're using them, just to give you some context. So Bridges is an HPC resource that is about 900 nodes now. We are booting everything using OpenStack Ironic. It's a way that we use it to provision the resource, to blast out things to disk, and set up Slurm on top, and we do traditional batch jobs on top of that, and other things like Hadoop, et cetera. And then we also use OpenStack for virtual machines as well, for doing some gateways, which we'll talk a little bit about later. But that's a little bit of background about what Bridges is and how we're using OpenStack currently. I'm Mike. I work for Indiana University on the Jet Screen Project. It is a $6.5 million, $640 node, two petflops of Ceph, pretty traditional OpenStack cluster. Both these systems were funded from the same solicitation from the U.S. National Science Foundation. So they're both targeting the same audience, which is what we call the long tail of science. There are, when you look at all the people who are eligible to use National Science Foundation funded computational resources, 97% of them don't use them at all. And so both of these systems, while coming from the same solicitation, even target different segments of this long tail. So if you were to take the histogram of user count and cycles used, it's a big swoop. And we're after this bottom swoopy part. And being funded by the National Science Foundation, we follow under the Advanced Cyber Infrastructure Directorate, which is the directorate that doesn't do science. It's the directorate that provides resources, computational resources to all the other directorates that do science, which is why our systems are a little bit different than the cloud systems that have come before. Ours are the first cloud systems for doing science. And the ones that came before are funded by other directorates. And they're more about doing research into how to build clouds. So your chameleons, your cloud labs, those guys. Similarly to Chameleon, we have an award called the Data Exacell, which has been designed to solicitate ways to build the hardware and software building blocks for doing research. And for scientific growth, that's what we've been using to kind of investigate some of these things like Moreno and to do our initial studies into OpenStack, which allowed us to deploy it on bridges. So that's also a very interesting award that we get to have some leeway to do research and to investigate interesting technologies that'll be useful in HPC. So without further ado, let's get into it. Why dynamic VM repositories? What are our goals? What are the problems we're trying to solve? Initially, Inexceed, which is a conglomerate of a lot of the supercomputing sites in the US, we were tasked with coming up with a VM repository or an idea on how we could have a common task of VM images that would work across the different supercomputing sites. So if a researcher want to do some computing at bridges for a certain type of project, but then move to another one based on where they got their allocations, that they could find common environments. They would have a way to be familiar to spin up a similar instance and repeat the science, that sort of thing. So the traditional images are, they're large, they're bulky. If you had a repository of hundreds of images, it would be a nightmare to kind of sync those between all the sites. That becomes a big time constraint just on the data, on the upkeep. How do you keep those images with security patches? How do you do those types of things without having an entire staff dedicated to just managing VM binaries? And then how do you do the versioning between them? So if somebody is using an old VM binary from years back, how can you rebuild that environment reliably if you needed to? So the idea we had was instead to use something like Moreno to be able to keep the piece of the repository in a small section of code, whether it be Ansible, Puppet, something like that under the hood, combining those with Moreno and Cloud in it. And that can be easily version controlled, it can be downloaded, it's very small, it can be synced across all the sites. So I got to bring up the security boogeyman. I have personally seen people throw up a VM, create a username, test with a password, password, and be rooted within half an hour. And if you're just grabbing an image from some guy that you know down the hall, you don't really know how good his practices are. You don't know how well these things have been cleaned, how they've been sanitized for data, whether there are credentials left over in there, and you really don't know exactly what they did. You don't know what the secret sauce is without just kind of got the trust that they configured everything correctly, didn't miss anything. To steal a Feynmanism, if you just grab the VM that you know, and you go through the motions, then you get cargo called science as a service. And that's not exactly science, is it? If you don't really actually know how all your stuff is built and configured, then you probably are just going through the motions and not actually doing science. And we don't really want to encourage that kind of behavior. We want to encourage good, solid science. That's why we do what we do. So if you build these on the fly, then they're always up to date. You don't forget anything, and you can always bail out. There's no reason you can't just fall back to your regular practices of good old fashioned, big blobs of binaries. Yeah, you touched on most of that, actually, okay, so. So what were some other motivations for this project? I talked a little bit about Exceed. One of our ideas in the future, too, is to do some Keystone Federation. Since we're already kind of tasked with developing the shared image repository, the next natural progression would be to have a central Keystone service that we can all federate with that would handle the authentication pieces and be able to pull these repositories, either from Glance or from a standard Merino app catalog. So these are kind of things we're working on in the future. We're hoping to get some projects going. The other motivations are just contributing back to the community. We've gotten a lot out of OpenStack. So if we can kind of contribute back some of the HPC characteristics that we need that we've helped develop, that'll go a long way to helping other people with the same problems that we were faced. And also, we're both part of the scientific working group that was just formed in Austin. So we've been fairly active in there. We're an IRC. If you ever come to our meetings and find out what we're doing, this is a big spin up point for us well to help with that and to further the science. So putting it to use, where do you start with these dynamic images, what are we trying to do? So obviously you have your base images, your minimum installs. Most of the people doing research are doing some type of development. So we need to build these development environments, build the libraries out, find a way to offer them the different versions, say of Python, with the different libraries that they're built with and bedded with. A lot of these turn out to be web portals. So we see a huge influx of the need for somebody that has an Apache web setup that needs to talk to a large HPC resource, whether it be the distributed file systems underneath, or whether it would be to submit jobs to the batch scheduler in an automated fashion. And that's a big point for us. We have tons of different science gateways that are doing that. We have people that want virtual clusters that want to spin up a little mini cluster of torque and kind of do their own thing inside of there, which we can do with OpenStack. And hopefully with Moreno as the automation piece for this. I talked about the offloading for bridges. That's what we mostly see is people wanting to offload jobs through Slurm using some kind of a science gateway web portal. We also, at both sites, we've run lots of Galaxy instances, which is all kinds of genomics data in that. Okay, so we'll crawl first before we walk and run. The most basic easy thing to do is throw out a bunch of cloud and net scripts that people can use to get going, right? So a few lines here, a little assist control and upgrade your packages. This will get you going, this will get you started. Install some packages, pretty straightforward. This is, everybody here should be fairly comfortable with this. You can drop out scripts that do stuff and then turn around and run those. So why do we need this? What's happening that we need orchestration? So when we have these mini clusters that we need to spin up or some of these more complicated instances that are just more than one VM, we need something to facilitate starting the instances. Some kind of a template to say, hey, this node is gonna be a metadata service. These nodes are gonna be IO services. These are gonna be clients, this needs torque, this needs that. So we wind up needing a lot of things like heat templates. And Merino kind of gives us all of that automatically. When you build the Merino package, it's gonna have the heat template inside. It'll set up all the different types of instances and then provision the software on it as you need it. These are all reusable. Again, it allows you to do collaboration with repeatable work, which is very neat. Someone can run the same Merino package and get the same software environment that they had on bridges as they would on JetStream. So why Merino? Again, the great thing is these complex environments, it's easy to package all this up, give this environment to the users. The users don't need to know how it works, essentially. They know what they're getting is this version of Python, this version of that code. The standard environment they're used to, but they don't have to do all the installs, they don't have to do all the upkeep. So it's kind of on-demand computing without having them needing to do any security updates or be patching the VMs. They can just spin the new instance up and get all of these features. Also, it's easy to translate and transport this to and from different clusters. So we're hoping to build this catalog that can be shared across the resources. That users can contribute to if they, we often see users coming in and saying, hey, we need this software package, we need this version, this is what we're trying to do. And we end up working with them with user services. But if we could have that, all that dialogue and all that conversation go into a Merino package, the next person that comes and needs the same request, it's already been fulfilled. On multiple occasions, I've worked with very bright, capable people. I daresay they can run circles around me and I've had this exact interaction. Because they aren't used to building out a virtual infrastructure. So it involves four or five calls back to me to help them walk them through all the steps that they need to do. And if you have a packaged environment, then we can skip all of this. And it looks a lot more like this. So they get their credentials, they log in, click, click, click, go. Wait three minutes for it to deploy, and that's it. We're off and running. So we've seen some of the cloud and scripts that Mike showed. They're relatively easy. You can do things like install packages, run scripts. The harder part is really getting that good heat template. And kind of having this all together is an order of magnitude. If you've looked at some of the Merino things, it's very challenging at first. You look at the PL language and having to kind of write out the UI. You do that yourself manually. So there was kind of a learning curve to Merino. And just from our initial interest in it, we were like, wow, there's a lot of things you can do. It's very powerful, but it will take a little bit of work. The great thing is that because of what you can wrap under Merino, we're hoping we can take a lot of our infrastructure, like say, PapaDanceable scripts that we already use on HPC to deploy things. And then put the Merino wrapper around those and turn those into apps and add that to our catalog. Right, so setting up Merino itself, it's relatively new. It's fast moving, you're going to need some extra pieces. You need a RabbitMQ that is user facing. At last, I heard it cannot be an HA cluster. So don't waste your time on that. And as with most things in computer science, you solve everything by putting it in a black box with an abstraction. And when you do it with this, you start losing the ability to find where everything went wrong. So it's kind of challenging to debug it. So we found that your best bet is to start back at the beginning and build these things up into packages rather than try and start from the package and work your way down. I guess I got ahead of myself. I touched a little bit on this, but when it's hard to kind of sell everything with moving all of the infrastructure into CloudInnit and into Merino, you can really just use CloudInnit as the wrapper to execute your Ansible or your Puppet or your other configuration management tools. We have all kinds of old things like CFN that have just been legacy codes or legacy configuration scripts that are laying around. We can easily still use those under Merino just by having Merino install the packages or have Merino use CloudInnit to install these packages and execute the things like CF Engine or Salt or Ansible or Puppet, which is really cool to leverage all of this infrastructure we already have. One of the use cases we have for this as well is at PSU, we have a lot of these managed VMs, what we call them. We're a little worried about giving users root. In some of the VMs, and some of them they can't because they want access to our parallel file systems, which if we give them root on our luster, they can mess with anybody's data. So we have this notion that we end up working with developers to set these VMs up. They configure their needs to us. We help work with them to get the packages installed, to get the configurations up and running. And if a lot of these packages are the same, instead of having user services sit down and do this over and over again, we can get these things into Puppet and Ansible, wrap them into a Moreno package, and then have user services be able to go through the horizon, spin up the VM, click on the Moreno apps catalog, and select all the different pieces that these researchers need. And that can take some of the burden off of the back-end people from having to do these manual installs. So we have 40 named partners on the JetStream grant. So here's our obligatory logo slide with a handful of these partners. You know, also links about bridges, if anyone is interested in kind of what we do, how we do that. There's some links here. There's links to the Data Exacell for doing these different researchy type projects. If you're interested in experimenting with things, we can help facilitate with some of that. Same thing for JetStream, links, documentation. We put some of our, we put our configuration management up in GitHub so you can look over my shoulder and see what I'm up to. And that, questions? There's one thing I forgot to mention, interesting, I should have put a slide in, but interesting use case. So the first Moreno proof of concept I did was to wrap an automated installed Dev Stack. So I wanted to try to do this. I was going to hopefully have it working for you today. I was going to do Dev Stack instead of Dev Stack using Moreno. Unfortunately, don't try to do it because nesting Dev Stacks is a huge mess and it doesn't work. But the neat thing is seeing that Moreno did work was able to execute the Ansible and spin up the Dev Stack and we were able to build about half of the services before things kind of grinded to a halt. But if anybody wants to see it, we have some Ansible on how to do that. And it works from just a standard VM for deploying Dev Stack and Moreno integrated all at once with just kind of a click of a button. Yeah, questions? So what version of Moreno do you use? Do you use just master trunk or you use some stable branch? We were using the Mataka at the IU cloud. Yeah, for me, I was doing mostly with Dev Stack originally and then just a GitHub check out of the latest for the non-Dev Stack instance that I was testing with. Another question. You also mentioned that you were packaging heat templates into Moreno applications. Does it mean that you used the so-called hot-based Moreno packages or you were writing Moreno PL code on its own? Well, so there's a Moreno CLI, Moreno Crate package and you give it the heat template and it spits out, yeah, it looks like a Moreno template. Yeah, I'm a little fuzzy on the details of, where exactly does it convert? When do you call it a Moreno package versus the hot type? But I was just trying to understand if you wrote the Moreno PL code on your own or you just used this tool to convert heat to Moreno? Yeah, we were targeting heat because it was easier to sort of debug like you could, Moreno on the back end uses heat. So if you start with the heat and you get that working well then you have a pretty good idea that whether or not it's actually gonna work when you wrap it up. Any other questions? How do you let users use very old versions of packages or lock to a particular version? It sounds like a great way to keep these VMs current but if someone's tied to a particular version of G-Lib C or something else, how do you support that? Right, so that would go back into the package manager for the particular distro. So you do pinning for whatever after and you'd script that up. The other neat thing is with all the pieces of Moreno being text files that are fairly small, we use version controlling, we can roll back the Moreno app to a prior version that maybe has those dependencies built into exact version numbers. So you're kind of getting the older versions without having to store the binaries around. Yeah, so Blair was saying, how portable are the Moreno templates between different clouds? They seem to be highly portable. From, we haven't found a place where I couldn't take his packages and run it on my cloud and vice versa. So far so good, but there'll always be little inconsistencies between clouds. On Jetstream we actually have two completely separate clouds and despite our best efforts, there are still little variations between the two. I think time will tell. I'm hoping to put this in production. All of our stuff at Bridges is running on Liberty still. So it may be interesting to see, well, can we spin up Moreno on the Liberty stuff that's existing before we upgrade to Newton and how will the packages translate from Mataka to DevStack to an older version like Liberty? That's a good question. Maybe we'll have some good answers. And how do you, what components do you need to change to federate Moreno between the two sites, do you think? That is not something we've looked at yet, where we have some other things in the hopper. We're trying to set up Federated Keystone as a first step for all of the exceed resources. So once we get that up and running, we're planning on doing some more federation stuff between the two, but still in the hopper. Maybe, we'll have to see. So the community app catalog, of course, went through it on our first look at Moreno and found a lot of stuff that was great but not real useful for us. And I would hesitate a bit to sort of overwhelm them, pollute them with a bunch of stuff that nobody outside of the research community would care about. Lots of people would love a real easy Kubernetes install but not everybody wants NAMD or that kind of thing. You know, that kind of thing. Yeah, yeah, you guys, yes. Maybe everyone in the room does. Yes, but there's another four, let's 5,200, 900 or 250 people probably who here at this conference who wouldn't really care about it. Yeah, yeah. So we could easily, we both have public GitHub repositories for our respective sites and I'm sure we'll be putting them out there. So if they are useful and the community finds them useful enough, then they could easily be pulled into the OpenStack app catalog. Like Mike said, we don't wanna blast the app catalog with 800 different versions of Python compiled with this, running this for that. Yes. Absolutely. One of our reasons for pitching this talk today was so that we could get a bit of practice, a bit of feedback from our learned colleagues here when we go back to exceed management and pitch the same idea that they should not have a massive repository binaries because it's never occurred to them that there's any other way to do things. So yeah, your feedback and comments would be greatly appreciated. So we can sharpen this a bit and make a, at least within the NSF funded resources, start there and see how much more we can pull in beyond that. Yeah, this is very much ongoing research. And if there's other ways that come, other technologies, something that come down the line that make this better than we're all for it, this seems like the simplest way to have something that is dynamic and is portable. Any other questions? All right, thank you guys, appreciate it.