 edition of RCE. Again, this is Brock Palin. You can find us online at RCE-cast.com. You can find the entire bat catalog linked to all of our Twitter's and our blogs and all that sort of thing. Once again, I have Jeff Squires of Cisco Systems and one of the authors of OpenMPI. Jeff, thanks again for helping out. Hey Brock, and you know, it's that time of year again. It's coming up on supercomputing. Yeah, so once again, both Jeff and I will be at supercomputing. We both have booths, Cisco and the University of Michigan. Jeff, I understand your booth is way off into boonies though. Oh, yeah, we have a fairly terrible location this year due to an accounting mistake on our part. So admittedly our mistake, but please come find us. We're kind of in the back. I'll be hanging around my booth. I'll also be doing the normal OpenMPI State of the Union boff and also involved with a libfabric tutorial on Sunday. So be sure to sign up for that too. So we actually share our booth. We split it every year with Michigan State University, and we're recording this one day before the U of M Michigan State football game, which is kind of like one of the many rivalries around here. So this is always kind of fun. Yes, I think I will stay completely neutral and not make any comments whatsoever. I'm heading to Lansing this weekend, so we'll see how that goes. All right, well, what do we got today? So today we are going to be talking about a software project called SPAC. So we have one of the developers with us. So Todd, why don't you go ahead and introduce yourself? I'm Todd Gamblin, and I'm a computer scientist at Lawrence Livermore National Lab in the Center for Applied Scientific Computing. And I guess I'm also the author of SPAC. And my day job is performance analysis research. And I've recently been getting into build systems or package management systems mostly out of anger. I can appreciate that. So who did application support? So quickly, what is SPAC? So I guess we built SPAC as a package management system, but I guess you could also call it a port system. It's basically a build from source package manager. And it was designed to deal with all the combinatorial issues that you face with software in HPC. So essentially, on a regular machine, in your Linux box, maybe you have a package manager, you install something and installs the latest version of that and you're done. On an HPC machine, you might want to install different versions of that thing compiled with different compilers and different compiler versions. And so the space quickly blows up. So just before we go too much deeper here, I'm just kind of curious, where did the name SPAC come from? It's kind of a fun name, but a lot of things come up for it when you Google for it. Where did you get SPAC from? It's the super computing package manager. That's where we got it. We have since learned that there are other meanings for the word. Oh, okay. Well, we will leave those off of this podcast here. For us, the only definition is the super computing package manager. That's cool. So, all right, now with that white elephant out of here, let's talk about what SPAC is itself. So you say it's a build from source package manager. Why another build system? Why not port something like ports or why isn't YUM enough and things like that? So, I mean, we did look at VSD ports. VSD ports is essentially like a directory treat full of make files as far as I can tell. And I guess we didn't really think that was the interface that people were used to using. I think the real inspiration for SPAC is my experience installing packages on my Mac because that's a system that comes from the vendor without a package manager. And people want to install other software on it and so they use things like Homebrew or Mac ports or Fink to install things on it. So, I mean, I think if you look at SPAC packages, they look an awful lot like Homebrew packages. They're just in a different language. Now, you mentioned also at the beginning too that part of the rationale for doing this was because it was out of anger, I believe was the phrase. And both Brock and I were nodding like, oh, yes, I know exactly what you mean. Can you tell us exactly what you mean? Can you cite some examples? So have you ever tried to build the Rose compiler? I have not. You have not? The last time you installed something on an HPC machine, did you find all the dependencies that you needed? Because the software that we use is becoming a lot more complex. And so when we try to go build things on the supercomputer, I end up having to go down the rabbit hole of installing the software and its, you know, 10 or more dependencies to get it working. And that might be solvers or that might be different mass libraries or maybe like a GUI like QT. So that's really the problem that that's the basic problem that we're trying to solve is get this thing installed with all of its dependencies so that people can actually use libraries without completely annoying their users. So doesn't this end up causing other type of bloat? Like won't every install of a high level package pull in yet another identical install of NetCDF or HDF5 or some common library? So not necessarily. So SPAC lets you essentially reuse the version of that other library that you've already installed assuming that the package you're installing is compatible with it. So if I say, you know, back install some code that requires NetCDF and that code can build with NetCDF 1.3 and you already have it installed, then it'll just link against the existing version. So you may get codes that require some other versions of NetCDF because, you know, another one of the things SPAC was created to address. And in that case, it'll just build you two versions because, you know, you need them. You have a code that's linked against 1.2, you have another code that requires 1.3. Well, you can't reuse the 1.2 version. You have to go and build a new one. But it won't disturb the existing install. So you could do both. And I think that's one of the main differences between SPAC and, you know, what you might use as part of your OS or if you install something with your OS package manager. So we actually do a lot of things like this manually right now. We'll have software installed locations. It'll be like Red Hat Enterprise 6, NetCDF slash 526 Intel, you know, 4.2 slash, and we end up with a million different versions. So this is a system for kind of implementing all of that and figuring that all out. It is and it probably does more than you might think if you're using them, if you're used to making directory hierarchies. So I mean, I guess, if you look at the way that different sites install software, so like if you use the systems at TAC, or do you mostly use the local systems at Michigan? I use and support on both. So I'm familiar with the way some places do things. Yeah. So like Oak Ridge, Livermore, Texas Computing Center, I guess Michigan, they try to make these directory hierarchies and it usually includes things like the compiler, the compiler version, the architecture, the MPI, the MPI version, and so on. And that's actually just, that's only a subset of the dependencies of most of the things that are installed there. So essentially, if you try to build a purely hierarchical versioning scheme like that, you're not covering all the configuration parameters of the software. So where SPAC is a little different is that the version, or I guess the configuration of something that you install is actually the full DAG of dependencies. And the version itself is just a hash of that entire DAG that includes all of that provenance information. So this software, its version, its compiler, the compiler version, its dependencies and all of that for them as well. Wait, so if it's all stored in a SPAC maintaining like a database someplace, so I can then go ask, what does this install have in terms of options for that package? So it does. The database is pretty simple in that it's really just the install tree. And under each install, we have a little .spac directory that contains a YAML file, which is just, you know, it's like a simpler that has all the provenance information. So it tells you what that package was linked with. And then we can go and query that. So if you type SPAC find and then you type an expression like, you know, find me all of the versions of, you know, HDF 5 that were linked with MPH. You can do that. It'll show you all the versions that are installed that linked with that library. And you can get more specific. You can say, show me all the versions that are installed that linked with MPH as greater than version 3. All right. Now I noticed too, I mean, you said earlier that, you know, you can link against dependent libraries of different versions. So let's say I've got, you know, package one links against some dependent package version X and package two requires, you know, that same dependent version or same dependent package, but, you know, version Y instead of version X. How do you handle, I understand how you can do that from a build system and a link system, but how does that work at runtime? Right. So SPAC is actually, it's based on the way that we already install software at Livermore, which is actually probably less, I guess, module intensive. Are the listeners familiar with modules? No, well, probably some, but let's go ahead and, this was kind of a leader question, right, into modules. Yeah. I mean, so, and actually, I mean, SPAC can do both of these things. So a lot of sites will use modules to manage the environment. So environment modules are a thing that came around in, I think the 90s, like 92, there was a paper at Lisa that talked about environment modules and how they were being used at some compute centers. Essentially, this is just a, it's modules runs entirely in your shell. And it's a way of encapsulating all the environment variables that you need to set to get a particular package to run. And usually the main one that you want there is, get this thing in your path. So if you want to use, I don't know, mpitch, you probably want mpi run for that particular mpitch in your path. And also the, it'll set LD library path and all the environment variables that you need to find the libraries that go along with that package. Okay. So then do you, does the SPAC like integrate with module files or does SPAC just make a directory that's very module file friendly? So SPAC actually does generate modules right now. So if you, if you check it out and you, and you install something, it'll generate a really primitive module file for that thing. It'll look and see if it has a bin directory and a lib or a lib64 directory, and it'll spit out a module file that just adds those to the right path, adds man path as well. Okay. And it does. So like if I, if I have package a that has like six different dependency packages, you would make a module file that includes all the right stuff. Yeah. All the right stuff to it, to a certain extent, right? What we don't have right now is some packages really want to set, you know, their own custom module things. And we're not bringing in pre-rex on the modules right now. So for example, like we won't, we won't add all the dependencies to the module file, which is, which is something we ought to fix. But, but we can very easily generate that because we have all the information that we need. Now what someone recently contributed is support for Elmod, which is, which is tax module system. So are you guys familiar with that? No, I was, I was, I was your, your head of the class because I was about to ask you if you integrate with Elmod too. Right. So we had a user, lots of Miliano Colpo from one of the, I think it's highly in compete, super competing centers. He recently sent us a pull request for Elmod support and it will generate the, the hierarchical, you know, directory structure for the modules. And it'll generate the Lua module files that, that tack likes. And that's really nice. So you can, you can easily swap between like, you know, the version of your package for one MPI and the version of your package for another using that. But really, so at Livermore we don't use modules. We tend to install all of our software so that it will run without the user having to know the magic incantation to load the module corresponding to that executable. So if you find something in a directory, you can generally run it. And the way that we do that is we set our paths on the executable. So I mean, and for people who aren't familiar with our path, essentially, if you, if you know what LD library path does, it's the search path that you set in your environment that says, here's all the directories you want to search to find dependency libraries of this executable or this, or this library. And, and our path is essentially the same thing, but it's actually set on the binary itself. So if you, if you add our path arguments, you're basically saying this binary knows how to find all of its dependencies, just use these paths. So we install everything that way. So you actually don't have to use modules to run stuff that you've installed this back. Okay, that's actually really cool because I need to do that. I mean, I've ran into it before, but I've never completely thought through what that would actually do for us. And as you're right, the packages are getting more complex. Yeah. You know, you need trialinos, trialinos needs this, but they want to use the Python bindings for trial, you know, it's just like you go down this chain of pain of trying to make it all work. And then they want to be able to run that and, you know, use Matlab at the same time. And Matlab brought its own QCC, something that so yeah, yeah, real pain. So I understand what you're doing here. The workflows are really, the workflows get really nasty because I mean, we have big applications here. We have multi-million line applications, like, you know, some of Livermore's applications have upwards of 50 dependencies. And sometimes you want to run two of those in the same environment. And they may have, they may require conflicting versions of the same libraries. And so if you do it this way with the R-path settings, essentially like each process is going to have its own search paths for the environment and they're going to be automatically set on the binary. So you can run those both in the same environment. And we were thinking you could use that, you know, either to set up like, you know, the M subscripts that you run and on the system on a daily basis, or you could use this as like a tool to set up like a Docker environment. So if you have, if you're using Docker images, you do have to build that image somehow. Docker will preserve it after you do that, but you could use this to set something like that up. So question, let's talk about how a SPAC, since the source build system, but obviously most of packages, we're getting like open MPI. Let's just use that as an example. Just a good random example. Good random example. Should I use the version that Jeff just submitted the pull request for this morning? Is that the one you want me to talk about? Yeah, during my research for this, I might have actually submitted a pull request for you, yes. But does the application like build system need to be modified, or is it more like SPAC, you kind of, so like right now, sometimes I'll make like a shell script that has all my auto comp flags and environment variables that I want set for that build. So I can kind of remember what I use. Is it something kind of like that? And it gives me wrappers around it? Or do I actually need to hook it into build system? So I have to port stuff to it. Yeah, you don't really have to port things to SPAC. I mean, you know, package managers were created and RPM kind of sticks to this philosophy that you have this notion of pristine source code. You really want your package to check out the source as it comes from the developer, because that's what most people can get. And then you would have a script that can build that. So I mean, think of SPAC as like it's an archive for all of your shell scripts that you would write to build these packages manually. And it handles things like deciding where the install paths should be, what the versions of your dependencies are, and filling in the blanks if you don't give it some of those things. So when I'm going through and I'm writing this script, does it give me like special variables to fill in? So like, you know, dash dash prefix equals dollar SPAC prefix or something like that. And it fills that in for me. Yeah, I guess I could, I could talk a little bit about what the package files themselves look and so, you know, if we're talking about the open MPI package, it's just it's a Python file. So SPAC is in Python. Inside that Python file is a class called open MPI and it extends the SPAC package class. It's got, you know, some version descriptors in there that say here's version 1.10.0. Here's the checksum for it if you download it and here's the URL for that tarball. And the other, you know, key piece of this is it has a method called install that takes what we call a spec. That's basically a description of how the thing should be built along with all of its dependencies. And the prefix where you should install it. And so essentially, like you're you're given a prefix and you're given a description of all your dependencies and you have to write a script using that to get the thing installed. And that's basically all it takes. Okay, so you said though, like all the features. So okay, let me back up a little bit. So say open MPI is a bad example, because it doesn't really rely on any external libraries. Right. Oh, no, no, no, no, it totally does. Well, yeah, they're all optional though, right? Yeah. All right, so you can still use it as an example, like, Oh, I need to have verbs installed. I need to have libnl installed. I need to have libpsm installed. Right. But it depends on what your hardware is. But you know, there are actually quite a few dependent libraries that open MPI can depend on. Yeah, I guess I was thinking of stuff normally, you don't have more than one version of PSM floating around on a SQL cluster, right? Because that tends to be very fabric and low, low, low hardware specific. Yeah, fair point. You've usually got one set of hardware drivers, so to speak, whether those are user space or kernel. Right. Okay. But even in that, say I needed to have two different versions of PSM floating around inside this SPAC spec file, but they're very nice. Yeah, yeah. SPAC is overloaded. So I do, you know, I've got my little script thing in there and I get to, you know, dash dash with PSM equals. How do I tell it which SPAC installed PSM to use? Okay, so we've gone to some length to try to abstract the complexity of dependencies from the user. So you get this install method that takes a spec in a prefix. The spec is really your full dependence DAG. So that's the directed asyclic graph of all your dependencies. But for any particular build of a package, you know what your dependencies are and each one has a unique name. So you can just basically say spec sub PSM, assuming open MPI, the package is written to depend on PSM. And it'll give you an object representing the PSM configuration. And then you can just say, you know, dash dash with PSM equals spec sub PSM dot prefix. And that gives you the location of the other dependency. So it's fairly easy to go and extract that kind of stuff. All right. And I'm shamelessly going to keep using the open MPI example here. But open MPI, like we just said, is all about the optional builds, right? The configure script will say, Oh, is this library here? Oh, cool. It's there. I'll build support for it. Oh, is this library here? Oh, it's not there. Well, then I'll just skip it. Unless a human specifically asked for it, like, Oh, build me verb support. Oh, but I can't find lib ib verbs. Well, then that's an error. And let a human figure that out. So does does this back package file have conditional logic like that? Can you do that? Like say, Oh, if they're, you know, this thing exists in the DAG, like I find lib ib verbs in there, then, you know, force that to be built in the open MPI configure script. So the answer is a little complicated. So at the moment, what we have is variance. And that's sort of something that I stole from Magports, because you can build, you know, a package with like, you can build QT plus Python support or something like that. So you can add an option to your package. So you can, so what, sorry, what was the example that you used for, for the optional pick any one of them, like verbs, PSM, anything like, you know, I see it's there. So I want to make sure that open MPI build support for that. Right. So then, I mean, you can put in your package file variant, you know, verbs, and, and then put, you know, what the default should be, whether verb should be on or off by default, and then a little description, so the user can see what that variant does. And then they can say back install open MPI plus verbs, and they'll get a version with verbs. Now, what we don't have right now, and what we're actually working on, there's a pull request in there for this at the moment, is good support for like external packages. So like, you know, if, if SPAC has verbs installed, then you can very easily say like, okay, turn on verb support, just use that. If it's on your system, and it wasn't installed by SPAC, then, you know, SPAC doesn't know anything about it. So it's kind of hard for it to find where it's located. We're working on ways to detect external packages. And this is actually something that we want to do for MPI specifically, because on a lot of systems, you know, they come with vendor MPI, right, you want to use that. So we'd like SPAC to just fire up and know about all the vendor MPI versions on your system, and allow you to link all your packages with that stack. Okay, so we're talking a lot about source installs. What about like binary installs? What if a vendor gives me something with certain options already in it, it's just like a tar file to just blow out or comes as an RPM or something like that. So you can very easily just copy that into place in your SPAC package assuming you can download it from somewhere. And you can have a local mirror of it if it's not available on the internet. It's not really geared for binary installs right now beyond just, you know, that simple notion. If you can do it in a script, then you can get it installed. But one thing we are actually looking at at Livermore right now, I don't know if you guys know this, but Livermore maintains their own Linux distribution called TOS or Chaos. I guess TOS is the Trilab open source stack, which is Livermore, Lanell and Sandia. And Chaos is the Livermore internal name for it. I think OS there is operating system, and I forget the rest. But those are both RPM driven, or TOS is RPM driven. So if we want to install something as part of the OS distribution here, we really need to make an RPM out of it. So we're looking into ways to generate RPM using SPAC. And that's actually more complicated than you might think. I guess, do you guys want me to talk more about that, or is that getting a little too detailed? Yeah, so actually, I'd be very interested to tell us why is it harder than we would think to do binary package integration? So TOS, Livermore's Linux distribution is based on Red Hat 6. And all these Linux distributions are made to get you sort of a fixed version of something, and to ensure that it's compatible with all the other versions that are installed. But you pretty much get one software stack there. And like I said before, in HPC, you typically want to install these giant combinatorial suites of things, because different codes require different versions of MPI, different compilers. We have several different ecosystems of software in TOS, the way that we actually build it and distribute it to the other labs. So we have the base OS, which is installed via RPM. We have RPMs that go in slash opt. Yeah, so in opt we assume that all the software there is binary compatible with sort of the major versions of different compilers. So if I link something with Intel 15, I'm not going to mix that with something that was linked with Intel 14. So we allow different compilers, but not too fine-grained. And then we have user TCE, which is sort of the TOS computing environment, or at least that's what we're planning on calling it, where we'd have sort of a fully combinatoric stack. And what that means is, you know, if I build a version of Petsy that links against Intel 15.1.2, and then I build another one that links against 15, or that builds with 15.1.3 and a different version of MPI, those are different. And they'll be installed in completely separate directories based on their hash. So what we want to do is we'd like to be able to take the space of SPAC packages, which is essentially anything you could, any DAG you could imagine, any sort of pendant tree that you would want to build, and project that into each of these areas. So user TCE would basically be, you know, hashed install directories for all the versions possible. Opt would probably contain per package directories, but we'd have to project this SPAC space down onto just version per basically only major compiler versions. We wouldn't include all the small ones. We'd assume that those are the same. So we're working on something to do that and to generate RPMs for all of those places so that our developers could write the same package for installing a one-off build in the file system for a user right when they need it. And then they could easily translate that to an RPM to go into the system later on. So can I infer something from your explanation there that you can actually, let's say you're building some package and you want to say, oh, I want this package built against all my open MPI installations or against all the different compilers that I have. Is that something that's, you know, that kind of automation built in the SPAC? Yeah, you can do that. So SPAC has a syntax for all of this stuff to make it easy to install different versions of things just from the command line. So you can say, suppose you, let's go with the Petsy example again. If I want to install Petsy and I want it to build with the Intel compiler, then I would say SPAC install Petsy percent Intel. And I can get more or less specific with that. So if I wanted a specific version of the Intel compiler, I might say SPAC install Petsy percent Intel at 14 and that would get me version 14. And the cool thing about that syntax is that it's recursive. So I can essentially, I can put a carrot on the command line and do the same sort of constraints on the dependencies. So I could say SPAC install Petsy percent Intel at 14, carrot, you know, open MPI at 1.3, if I wanted to build that with that version of open MPI. So what if I need to have like different C flags or something weird like that specific to my machine? Is there a way to kind of like do like a pseudo thing in there to kind of like have dependencies that aren't real? Oh, so like virtual dependencies? Is that, sorry, you're trying to get, oh okay, let me, hold on, maybe we should edit that out. Let me think, what do you mean by the different compiler flags for different versions? I don't know, I mean sometimes on some systems you find that like say on a blue gene such and such it's you know not stable with O3 or something like that. Right, so well you can do compiler flags in a couple different ways. So we just got through implementing a feature that's getting integrated where you can actually add compiler flags arbitrarily to build and those become different versions. So I could actually say on the command line you know plus C flags dash O3 dash G and that'll get added on to the flags used to compile everything in that package. And that that becomes a different version from if I built without it. The other thing you can do is you can just write the compiler flags right in the in the package file. So you know if you were going to do your configure line you would say configure and you'd set the environment variable for C flags to have your special flags right there. So that's the support for compiler flags right now. All right now is that related to I saw in the documentation this concept of virtual dependencies? Yeah so the virtual dependencies is really where your package doesn't depend on a specific implementation something it depends on an interface and so this was added for MPI. Packages don't usually you know require open MPI or MPH right so if I and actually this is one of the problems that you run into with the OS package managers typically they'll install one MPI version and that's all you get and sites want to run with both. So you could have PetSeed and in the PetSeed package file it would say depends on MPI right and that and then the open MPI and MPH package provides MPI and you could link PetSeed with either of those. Now what SPAC does that other things don't in this regime like a lot of package managers support just this basic notion of a virtual deb but SPAC actually versions it. So you can have the open MPI package provide MPI at version three when it's at a particular version and you could have the MPH package provide MPI at version three when it has version three or higher and MPI version one when it's at version one or lower and the package can depend on MPI at one if it only needs MPI one and SPAC will ensure that it links with an appropriate version of the right of the particular MPI implementations and so the cool thing that that does is it lets you compose the packages so if I say you know I only have to have one PetSeed package that depends on MPI with the right version constrained and then I can say SPAC install PetSeed Carat MPH at you know three and it'll link just fine it'll compile with that version of MPH. Okay so who's using SPAC today? So we've gotten a lot of adoption inside Lawrence Livermore it was it was created by me to sort of help students build things faster because we had a long time we spent a long time spending students up for their summer projects we've since gotten a lot of adoption within the compute center here for installing stuff for users and application teams have actually started adopting SPAC because they need to manage all of their dependencies so you know some of our multi physics code teams who depend on lots of different libraries they can now share builds for all their different packages and build the versions they need and that's saved them a lot of time. We've gotten some interest from external compute centers so we're currently collaborating with NERS to get SPAC up and running for Cory and also with Lannell to get it up and running on Trinity those are both I think XC40 Cray systems and we've had some interest from external universities as well so like Inria has used SPAC to package up some of their solvers and we've got contributors to the GitHub project from a lot of different places. I think we had a guy from Stowers Medical Institute in Kansas City who used to set up a cluster and we've had some other people who just used it to manage their own environment so it's getting some traction. I wouldn't say it's super widely adopted right now but we've had some pretty good experiences with users and we're starting to get contributors. So what's the most complicated project that you've got under SPAC right now because the whole point is to make difficult packages easy to install so what's the juiciest one you've got so far? Right now the juiciest one that we have is probably one of the Livermore Multiphysics codes which has I think 70 dependencies or something like that. We're working with Kitware to get a pair of you in there and that may take the title next. QT is in there so QT I think has 40 or 50 dependencies so those are pretty big and complicated and then I guess just you know maybe not for the number of dependencies but for the complexity of the package we've got GCC in there which is interesting because it kind of has it has different dependencies depending on what version of GCC it is so they add you know different external libraries for different loop optimizations and analyses that they do. Something I'd like to ask is there a public repo of maybe contributed packages for things that are kind of basic like GCC the foundation for a lot of things? Yeah so like Homebrew SPAC comes with a package tree so if you if you clone it right now I think it comes with upwards of 200 packages so if you if you just go to github.com slash scalability llnl slash SPAC it's there you can download it we'd really like to and we're implementing a feature to allow external package repos so that sites can maintain their own sort of site specific packages easily and so that we would have a clear migration path from that to sort of integrating your package in the mainline. So what license is SPAC under? It's LGPL. And why did you choose LGPL? We chose LGPL mainly because we've actually had an easier time working with some companies with LGPL because it if they want to contribute to our project it's it's in the license that if they want to release their changes they pretty much have to so we don't have to go through lawyers like we've had bad experiences working with some companies under BSD licenses where you know they have the option to keep the source code private and so they don't want to give up that option and and so they have to go through a whole release process for it but with LGPL we've had a smoother experience. I'm not sure that we have any particular leanings about the license we could look at re-licensing it if someone wants a BSD version. All right and then there's another question that I'd like to ask all software projects just for pure curiosity and we kind of alluded to it earlier already but what version control system do you use and why? We use Git simply because the you know for an open-source project it's way easier to allow external people to contribute on sites like GitHub. I think you know the web tools are just way better and also I like Git for the branch separation and basically just accepting features from other people is tremendously easier than it was with Subversion. Okay so how does someone want to get involved like you know contribute to the public tree of SPAC files for their project or work on SPAC core directly? So I think the easiest way to get involved is just go to the SPAC website or the GitHub page if you Google SPAC-LNL it's the first link and fork the project and start contributing use your own version and and start editing the packages to suit your environment. There's a mailing list so if you look in the readme for SPAC there's a link to the Google group if you join that we have you know I wouldn't say it's a ridiculously high volume list right now but there are there are interesting discussions and we have weekly telecons so those are at Thursday 9 a.m. Pacific time right now but we may be we may be rescheduling those and that'll go out on the mailing list. So where is SPAC in the road of development? I mean are you guys alpha quality beta quality are you production quality what's the current status? I'd say we're around beta quality right now or at the very least with the next release we will be. The packages I wouldn't say are robust across a lot of different systems right now we we test on Linux at Livermore and that's about it. We would need to test on a lot more different sites to guarantee that the packages would be you know a good experience for most everyone who downloads SPAC but we're getting a lot more contributors we like I said we started working with Kitware on on Paraview and we're adding continuous integration locally at Livermore so that the packages will hopefully be more robust very soon and we're working on Cray support so we'll have more platforms very soon. Okay Todd thank you very much for your time can you repeat the location of the SPAC website? It's github.com slash scalability-llnl slash SPAC. Okay thank you very much. Thanks Todd. All right thanks a lot.