 Welcome to another edition of RCE. I'm your host Brock Palin. I have again Jeff Squires from Cisco Systems and Open MPI Jeff. Thanks again Good afternoon Brock. What do we got today? We have with us two people who work on the rocks cluster distribution. At least that's what I think it is I've actually I'll admit I'm familiar with it, but I've never used it myself. So I'm flying blind again They are Greg Bruno and Mason Katz and they're both at UCSD. So guys, welcome to the show Hi, I'm Mason Katz. I'm work at UCSD. I've been here for about 10 years working on the rocks cluster distribution project Prior to that I worked at University of Arizona doing security writing network operating systems It was a lot of fun and prior to that I was a 8-bit embedded developer So I'm a big fan of the the of the accumulator Yeah, good old Z80 My name is my name is Greg Bruno. I have a very similar background to Mason here for the last 10 years I hired on one month before he so I always remind him about how I outrank him in seniority And but prior to that I was working at NCR where they build and can still continue to build the world's largest databases They're using teradata technology Okay, guys, thanks a lot for taking some time Can you give us a quick rundown on what rocks is meant to accomplish and therefore explain what it is when you say a cluster distribution? Expand on that So that's that's a good question because we've had this question internally for for many years. We originally our first release in November 2000 we called ourselves the Rocks clustering toolkit and that was that was a name that we put together the toolkit part was added by Greg Gregum And my own boss Phil Poppettopoulos Name should be last name should be familiar And at the time we're a toolkit because we're really a layer of tools that went on top of an existing machine And then allow you to deploy a cluster and over the years We've become more and more automated more and more turnkey and now we're really more of a distribution We're more of a cluster on a DVD so you can pop it into one node turn it into a master for your system and Just one by one bring up your compute nodes with zero human interaction But we'll get we'll get into the details of that more Okay, so what's the history where did rocks come from how to get started? Actually Mason and I met Phil Poppettopoulos prior to working on rocks in another lab That was here at UCSD So all three of us were hired into to work for a PI there where he was investigating Windows clusters So this was in 98 99 and 98 99 when he was investigating Windows clusters and so we all came on board and we're all UNIX heads So this was we were really out of our element and so we were trying to use a lot of the windowing tools in order to You know basically put a windows distribution down onto all the compute nodes reliably They had all these these programs it did like multicast it None of them worked reliably and Phil one day said you know what I'm just going to make a partition on every node And it's going to run Linux and at that time it was Mandrake and so anytime we wanted to Re-image the entire cluster Phil ran a program out there that told the nodes go back into this one Linux partition and then basically just do a DD in order to you know, Spot all the new bits down So that was really kind of the prototype of if you want to reliably build a you know back in cluster nodes at any sort of scale Linux is your horse So what exactly is is rocks is it is it middleware? Is it an operating system? Is it a distribution? You know, how exactly would you classify it? It's it's all of those rocks includes All of the software you need to go from bare metal to fully functional application ready computational appliance So rocks for example includes all the bits from smell all the relevant bits from centos It includes middleware for cluster modern such as ganglia It includes middleware for grid such as globus or did it one time? So it's it's it's everything we actually have referred to it many times It's kind of the the hypodermic is this is the hypodermic you use to inject Software onto your bare metal. So it's it's just It's the methodology and it's the technology to wrap all that up OS to application treat everything the same applications are treated. No differently from the OS It's just how do you bundle everything and get it deployed so Check technology wise is really a distribution, but includes components from operating system all the way to application including the middleware It's a fuzzy answer So run and give us give us a sample scenario, right? So you you mentioned a really high level one before you you pop in a DVD on a head node You bring up compute nodes give it give us a little more detail on on what you know a common scenario is that you see your users doing Well, so let's go to the simple answer just a just a HPC class an MPI machine We'll have people that they'd buy a few nodes from a tier one vendor They create they'd have their front-end node. They talk to their network admin get an IP address Simple stuff like that. They go to our what where our website at dub dub dub dot rocks clusters org They pop in the DVD turn the machine. They answer a few questions. It's just like installing red hat It's actually a little easier because we remove a lot of the installation screens And they just put in their IP information time zone password Hour later the front-end comes up They will run a very simple command on the front-end when they first log in called insert ethers and they turn on these slave nodes They're all on a private non-internet routed network and they just automatically come up and it takes about 10 minutes a node you do them a Few seconds apart from each other. So really in a matter of two hours you can go from a room full of 32 64 nodes and boxes to a fully functioning MPI machine and that's that's always been our target That's where we started we started funded as NSF funded and Paki program To build to make it easy for researchers to build their own build and manage their own MPI machines And we move past that but that's where we started and that's really what we're best at still Okay, and so when you say it's about 10 minutes per machine. I assume this means a disk full load So you're you're black testing in OS and some level of middleware and software and other things like that and That must include you know loading it on to the machine itself or or what? Yeah, that's right. I mean we do all all of our installations are from bare metal and so There's a there's a lot of benefits there One of them is is that when you start from bare metal or you you're actually starting from the empty set And so when you start from the empty set, that's that's known, right? You know what the machine was going to do in the when the you know the amount of software in the set is empty And then as you start to build up you're going from known state to known state to known state to known state And so with at the end the configuration and Characteristic of that machine is fully known because you started from a known state and you transitioned all the way through known states Another benefit of having the disk out there on the node is I mean simply just to cache the operating system And also to cache applications. So if you're running at scale Having that disk drive there in order to help the nodes You know relieve the pressure of the network just to go out and grab some files or some common files Or some read-only files is we found that to be a huge benefit So in the religious debate of disk list versus disc full it sounds like you guys are coming down on the side of of disc full Well, I don't I don't think it's a religious debate. It's it's it's what do people prefer We've chosen as a group just not to worry about the discless problem We we understand that people care about it and we've had we've collaborated with one group Kessler start University Good friends of ours in Bangkok who created a discless role. So they did add discless support to rocks The project there that they were with is no longer funded. So that's that's a support That's some support that went away, but for us. It's not a religious argument. We're very open to discless It's just not something that we've focused on Okay, fair enough You mentioned the word role in there. What exactly is a role? so role a Rolls a meta package really. It's it's the easiest way to think about it. It's it's a meta package, but it's not just Configuration and bits for one machine. It's a meta package for your entire infrastructure for your cluster for your data center even because in that role you have Individual packages and in the in the context of Linux. There are PMs. There's a little bit of XML in there That's actually maps directly to Whatever the automatic installer languages of the OS is in the case of Linux. That's kickstart So it's a little bit kickstart snippets and its packages and then that Those that XML that kickstart language actually decides what type of appliance gets what packages So compute nodes get MPI visualization nodes get open GL Database nodes get postgres, so it's how do you how do you suck up all that information and Put in this little tiny self self-composed self-contained ISO image Just just to follow up. I'm sorry just to follow up a little bit there with basin. He mentioned part of a role is there are a Collection of XML files now most of these XML files if you if you looked at them It's it basically has one nugget of functionality and an example. We always like to use SSH So we have an SSH dot XML file inside inside the base role and then there you'll see the configuration of SSH on you know on every node because that's a common Function that we have on every single node and then there's another file That is it's it's a graph Specification and so all of rocks is the configuration of the node is specified As a graph and we have several or I should say the root nodes of the graph or what we call appliances So one root node would be your front end another root node would be compute another root node would be this tile and when a node asks for a kickstart file we Send it over to a CGI and that CGI looks at that node and says oh this is noted IP address Whatever we look up in the database to see what What appliance it wants to be we go to that root node of that appliance and then we do a graph traversal of And this graph traversal is accumulating all of the XML files across that graph and so Getting back to what is a role a role is this thing that when you add it to the front end it actually splices itself into a Machine so if you start off as an MPI machine and you bring the vis roll on earlier And you also want to create vis tiles from that same rocks machine What happens is that that graph XML goes and it splices itself into the current base role Okay, so everything lives in a role the OS management tools the rocks meta tools themselves Everything is belongs to a role That's right. Everything. Everything's a role Okay, so everything's a role so Rolls are made up of these XML files and RPMs or tar files. I mean do they have to be a certain type of package format? Yeah, so even expand on that Do you so do you focus on Linux or do you do other operating systems as well? So like would RPMs be the Linux way and and have different kinds of packages for for other operating systems? Yes, we focus primarily on Linux, but we've added support for Solaris we had a support for Solaris one because We were funded by it We ended up hiring a new developer to our team a few years ago His name is a new bridge Indra who's done an excellent job at taking rocks which started out as a very not just Linux, but very red hat dependent anaconda dependent system and Mapping kickstart Ideas into jumpstart which actually isn't that bad because most ideas and kickstart came from jumpstart So in many ways everything that rocks was doing was being mapped back to where the ideas actually came from from underline OS um, so we To try to focus on the OS OS part of the questions. Yeah, right now we do Linux and Solaris We view red hat is kind of the dominant definition of Linux. That's not to diminish the significance of other distributions just to kind of Focus on the reality of we've us being a very small group being being government funded And we can't really justify spending our time to support rocks to yet another Linux distribution When it doesn't really functionally do very much for the users. That's not not to say it's not a great idea But it's not something we'll do in Linux. Yes, everything maps to a package an RPM and solaris everything maps to a pkg Just a package everything my maps to jumpstart jumpstart files layers everything maps to kickstart file and Linux rolls themselves are Packages from whatever the native OS is and then the XML is Really our own invention, but it really looks a lot like kickstart language and then that can be interpreted that can be parsed and Converted into either kickstart language or jumpstart language. So we don't do we don't do tar balls Because we really want to take advantage of the native packaging system and being able to do package dependencies So we do we have very easy easy tools I mean one command we could say rocks create package and it will take a standard GNU configure make install package And it'll build an RPM or even a macOS package or solaris package for you So the fact that we're focusing on RPMs and PKGs isn't really a constraint. It's just the technology We use we use RPM more of as more of a Software distribution methodology, we don't use a lot of post sections and packages or anything We're just using it to wrap up the bits. It's a it's a it's a bit transport as far as we're concerned So one thing you threw in there you said macOS package after you said you supported only Linux and solaris So what gives is that is that a hint of things to come or? It could be and we were again. We did solaris specifically Well one we're interested in because solaris brought to the table a kernel with true threads So from a technology, there's some real real Interest that we have from from HBC and even non-HBC context Mac OS we did well, I mean we're sitting here on Macbooks So it's kind of nice to every now and then build little bits of rocks on the Mac So there is there we have we have the support to build anything and rocks can be built into a macOS package And just to follow up on that on the Mac story there was a a guy pretty high up and Apple who was here at UCSD and he was making the rounds and talking to a bunch of different groups, and he said hey How about rocks on Mac OS X we said it sounds great, and he says okay And we said all what all it's gonna need is a funded developer for two to three years, and we haven't heard back from him So in what you describe I'm gonna jump back a little bit in the conversation what you described for roles It sounded like there was at least some level of overlap for package dependency resolution because that's always a sticky problem And it's and it's such a sticky problem that people have invented multiple systems to do such things like with yum And yast and and various others. How how do roles compare to that? Is it is it a different thing or do you use those underneath or or what do you do? Yeah, we use those tools underneath when we write RPMs Most of the most of the things that we author for rocks actually don't have these very strong dependencies But rocks is I don't know probably 99% third-party software that we're integrating and configuring for people So for those things that we have to compile Sometimes we specify the dependencies sometimes we don't But usually we rely on in Linux now and anaconda now it is all young so young is doing the dependency resolution But that's just the resolution on what the bits are that's only half of the story The other half of the story is what is the configuration and that's where rocks comes into play It's very easy to say I need x y and z on all the nodes in my cluster That's really easy to do you almost fantastic at that where young Falls down and it was never designed to do this is I need package x y z on my cluster And this is my IP address and I need this service on from at this machine I need that service off of that machine. It's the configuration that rocks focuses on so that XML those Kickstarter files it's really All of the post configuration that everything you would have to do after you run yum to get things working and It does things like look up variables in the cluster database to figure out what's going on the cluster and what things should be doing and Just pre-configures Everything for you and their dependencies that we care about in the XML and the pre-configuration You have to append to some files before you pin to others. You have to start some services before you start others So that's where we do our own dependency tracking. We don't really we don't address the package dependency problem itself Because the underlying technologies do that for us and we're big big proponents I'm not reinventing the wheel and if I if I could come into work and not have to write code That's a really good day so when you say Not reinventing the wheel for the standard cluster tools people and end-users used to working with MPI libraries batch systems Compilers you're not making your own. You're just kind of packaging up things like Torque SGE GNU PGI Just kind of handover work other people have done in a roll form But let's go and open MPI of course, right? Yeah, that one specifically. Yeah, sorry I had I had to is just that mandatory. Yeah, absolutely. Oh, so all those things are rolls We are completely agnostic on things like that we rolls themselves actually came about because of a conflict no not a conflict but Differences of opinions sometimes a conflict in terms of what local workload manager do people want to use We we started out in rock supporting Was it called PBS then or open PBS or Whatever name that they had I don't know. It was pre-pre-torque obviously. I guess they were just PBS So we started off on PBS and Sun Grid Engine was started to come out and we ourselves became big fans of it because we found it easier to run and We started to have a distribution in rocks where we would give people the bits and they would have PBS and Sun Grid Engine And then they'd have to run some crazy command to shut one off it was completely ridiculous and We finally figured out how to Turn rocks into a component model and that's that's really what roles are roles are the components in the component model of rocks And then we made PBS of a role. We made SG a role We're now now torque is a role. There's commercial roles that do PGI and Moab and stuff like that So it's it's yeah pick your favorite middleware and somewhere there probably is a role for it Do only you guys create roles or do other people create roles? So we create, you know, kind of the base system and some of these these ancillary Middleware things that Mason was just talking about but then we also have a guy long-time developer Who's been with us up? He's actually north of the Arctic Circle in Norway And I've been there in June when the Sun never went down and it completely freaked me out but that's a side story and So Roy Trump Roy Trump so Roy drags it up in Trump So produces the PBS role because as Mason was talking about there was a time where we had PBS and SGE and We decided to drop PBS for SGE and Roy said hey, I really like PBS So he's been the maintainer of that role for several years now there's some other Projects here at UCSD which they make their own role. So after we we come out with our distribution They bundle up their own tools and put it on. There's you know, some of those guys are doing Quantum mechanics codes and other guys are doing visualization codes with that and then there's also commercial roles out there so Mira com again, we initially did the mirror net role and We conned them into taking it back and they've they've done a great job with it They've they've been pumping that roll out for you know, the last couple years as well And there's other commercial entities out there that are also making roles cluster corp is Another one that that I know of that are making roles out there and there's going to be several other institutions and other Commercial companies that we have no idea that they're actually making them and using them day to day So actually you touched on a couple of things there too So this is obviously a wide-reaching project, which is a really really good thing So you've got some open-source guys. You've got some industry players and so on how how do you function as as a project? Do you have core developers? You know only there in your group or do you have people who submit patches to you? I mean do you consider yourself a true open-source project with developers and random places around the world or or does it mostly focus on? External people making roles or or how does that go? So you touched on a lot of a lot of things there and actually this is one of my favorite things to kind of talk about about rocks So rocks itself is is Is open source? I mean it's it means the basic definition We release all resource code. You could watch our development day to day We don't necessarily we usually say we're open source But we're not strictly open development And what we mean by that is if you go around and look at kind of the key open-source projects that have truly made an impact And in some cases even change of the world They are actually open development There's a very small group of core developers that run the project and as it scales out You have a very small group of lieutenants that are in charge of accepting patches from the community and Usually rewriting them or deciding what to do. So it's it's it's a tough line to walk It's how do you how do you go through and? We have a vision of where a rock should be and we want everybody to be very welcome to the development community So how do you stay on that vision while at the same time pulling in developers from the outside to help? You march towards that and it's something that we struggle with every day And it's something that we we really want to do much much better at but we are in terms of the core of rocks we know exactly where we're going and we're quite opinionated about it one of the reasons that we created roles to decompose all these things was to open up our development community and the the success that we had of having Roy Draxith volunteer or be drafted to become our first external role developer was huge That was a huge one and since then I think we've we've come to a point where kind of roles are now 50-50 UCSD external commercial and industry. I mean commercial and non-commercial So We're like to kind of go back. We are open source. We publish everything we do on the curio We're looking at other tools for this to kind of use better tools We're all CVS guys internally So we're trying to move trying to Change for the times and move to better systems to get get people to be all contribute better. We do accept patches We have contributed patches. We have put patches back into our code We probably have a 50% hit rate when people submit patches on whether or not the right thing to do We've had we've had good success and bad success. We had a group in Singapore who were the first guys these were actually the guys that introduced us to sun grid engine one of the guys name was on a G named Naba and He he did this is pre-rolls He put SG into rocks and converted us to it and those guys were were just an absolute blast to work with for a very long time So and then we've had we've had guys that have tried to convert patches that would They were great because the patches worked for them, but they didn't work for community as a whole It's very easy to raise solution that works for your cluster It's really hard to raise solution that works for every cluster out there including clusters. You've never imagined and that's that's what we're trying to do So that's actually sounds very similar to a lot of the things that we wrestle with in the open MPI project as Well, you know getting patches that are are good for somebody and not for everybody and so on and telling the line of Getting new people in but you know staying on vision It was you you've put to words very well what I have not been able to express well, so thanks I hope you don't mind. I'm probably gonna end up borrowing those words from you Yeah, what's it's it's really hard open source is difficult a lot of people there was so much height between around kind of cathedral and bizarre release early release often and I'm frankly all this is nonsense. It's is great for a application We don't ship an application and open MPI's on applications middleware something critical to the system If you're shipping one little web service. Yeah release early release often we're shipping a system We can't do that. We have we have to have a formal release process Where we start a beta we go in for a month of beta We do code freeze before that for a month and we're we're a for lucky a two-release a year group We're struggling we're struggling with the question and we want we're struggling with technology to figure out what technologies do you Do you use to attract more developers because we our intention is not to be closed development It's just the way things are turning out. We're trying to expand Yeah, we wrestle with the same thing. I mean open MPI I assume very much like your code base It's it's complex and there are subtle reasons why things are the way they are and it's just something you have to work in the code base for a year or three before you can Get the scope of what is happening and I assume this is similar is very true for you And so even though you have very well intentioned people submitting patches. You're like, hey, this is great I appreciate the spirit. Thank you But you know that that kind of thing and how to not turn away people and get them frustrated because of those things now I I do actually want to touch on one thing that you said in there That was very interesting to me. You said that you're using mercurial, but internally you're all CVS, I wonder if you could explain that and your rationale for that a little bit Let me let me just start by by saying Our boss philpapodopoulos has a great line And he says I've been using CVS for 20 years and CVS has never lost a file for me And so that one really resonates with me because I mean I've been a long long long time CVS user And again since we're so small That we don't run into the big difficulties of you know, the branching and then trying to to deal with the Conflict conflict resolution or conflict resolution in the code not with the people and so if I mean all you have to do is walk across To the next guy's office and say hey, you know, what are you working on to to try and manage those types of things? I mean if we were split Geographically CVS may pose a lot more problems than it does today But in terms of technology, we Well, I'll back up there was Many years ago When I worked in industry there I had I worked with an absolutely fantastic source control system by sun microsystems It was part of what they called spark works. I think it was right about the time they were switching from bsd to system five And this was a distributed source control system built on top of SCCS so obviously commercial But it had this we had we had some consultants across the country and we had the ability to Take a repository pull off an entire clone that was still hooked up for diffs And they could operate on that remote repository commit back there And then once a month they'd come over to um, this is when I lived in Tucson, Arizona they come over to Tucson and We'd all sit down in a room and move their Their diffs back into the tree and it was all the software took care of all this So I've I've been a huge fan of distributed source control for 20 years And I don't know what sun did because the software was fantastic and it disappeared from the face of the earth and mercurial get Bizarre these guys are just now starting to come back to Well tools we had 20 years ago. So we looked at all of them. Uh, we looked at bazaar for a while I really liked it. I loved that it was made only mainly python Because that's what we do most of our stuff in And it turned out to be just it was just slow. It was just unusable Get very very interested when we first looked, uh, it was just too hard. It's matured a lot We're looking at it again now Starting to do a a git role for building git appliances So kind of a hint that we may be moving towards that Uh, and we looked at looked at mercurial and mercurial That looks pretty good. It's it's well documented. It's easy to use But what we're doing right now is we're committing everything to cbs internally And once every 20 minutes, I believe we're doing a cbs checkout into a mercurial Repository sandbox clone I'm not sure what the terminology is and then that's doing it at hg add L I believe the command is to kind of update that mercurial mercurial repository and then people can check out from there So there's a 20 minute latency from from our hg repository from uh from our cbs repository And we're using the now for remote developers We're drag set used to have cbs access and we kind of pushed them to mercurial Again, we drafted them the first time So we figured we draft them as our first guinea pig on a distributed source control and it's worked pretty well And a new presenter is a huge fan of it. So he's been doing a lot of his Solaris work in mercurial. So hopefully the Hopefully the young guys can teach us. Um, how do you develop it the right way? Okay, moving on from that What is some of the um Upcoming features you would like to add to rocks that aren't there now or maybe are in, you know, your beta branch or But what would you like to see rocks do? Oh, wow, that's uh, that's tough. Um I mean where where we're focused right now is in the virtualization space So for the past two years we've been developing tools to To take the tools that people have come to to know and understand and rocks on building physical machines physical clusters To have that same tool set support virtual clusters and to have That tool chain be almost indistinguishable between the two. I mean, obviously, there's a little bit of this, you know differences and being able to specify what a vm is but after that We want we wanted all of our kickstart stuff to work exactly the same and we also wanted, you know, how we deal with distributions how we deal with roles and so, um We're We're actually funded by the nsf To do that. That's what our proposal was focused on was virtualization So our first release of that was just being able to handle virtual compute nodes Then the following release we had virtual clusters So you could deploy a virtual front end which installed your virtual clusters or your virtual compute nodes And then you could have multiple of those things all overlaid on the same physical machine And also the kind of the last part of our proposal And we've done some prototyping with is how do we bring up compute nodes in ec2 and make that part of your cluster and Phil Poppidopoulos is the guy who's leading the charge on that one And so we see that as As a very exciting opportunity where people would have their physical cluster in their lab And either using the physical cluster or they overlay a virtual cluster on top of their own physical cluster And when they for those times and they need a little bit extra cpu power That and if they have an account on ec2 and if they have a credit card associated with it That we have the toolchain that allows them to easily spin up some compute cycles out there When they need it for those peak periods So that's That's I mean, that's the that's the big area that that we're looking at these days So how much of that work is available right now? The ec2 stuff is kind of proof of the concept that's stuff that Some of that's you know, so I got a quote line from from phil. I'm going to quote a line from our boss Uh, it's if it's not a cvs. It doesn't exist A lot of that stuff is not in our source control So it doesn't yeah, it probably is somewhere Yeah, it doesn't really exist though. Uh, we yeah, we're the ec2 stuff is far out there. We're probably Every release we try and say we got every developer gets one feature to add to the release Um, I already know one feature that's going into the next release. Uh, I don't think the ec2 one's going to make it So I think we're talking two releases out, which means we're a year away Year away Not too bad. I think a lot of people have been you know, work on that problem And I haven't seen many solutions that I've liked yet So, uh, it'd be interesting to see a couple other approaches people use From there, what is the strangest use of rocks? I've noticed some of the roles you've mentioned have been kind of Not what I would normally consider a traditional hpc kind of cluster node or head or anything like that What's what's one of the oddest roles or uses you've seen Well, just to kind of tack on to that I'm glad that when you went to our site and looked at the roles some of the roles surprised you Because I think probably the conception and it's probably the majority of the clusters which are spun up with rocks. They are hpc clusters but We're actually more than that where we can't support visualization display walls We know people who are using rocks to just manage the desktops in their lab um And so it's it really it really is more than hpc. We were I mean we we do what we can to make it more than hpc But to back up to your question of what are some of the strangest uses of rocks where One message that came through on our discussion list was um, this guy said Hey, I downloaded your code I installed a four node cluster in my garage. Now, what do I do? And so we looked at this and just said I mean it was funny that the guy took some took our code with no real purpose He just wanted to get the code and he wanted to learn about clustering and he just he spun it up in his in his In his garage and he just sat there and said Okay, now. What do I do? And he was like he was twiddling his thumbs there So I that was that was kind of a strange comical use of it but kind of a really cool use of it is um a guy who I talked to and I Escapes me where he's from but what he's using the cluster for Is to do sound processing in a room? And the processing that they're doing is they want to recreate the sound of these great Hauls Where you know these these these fabulous, you know live performances have been done So say, you know, hey, I want to be at arena x y z That they they they take all the sound and we they pump it through The cluster and then it comes back out and it replicates what the sound would be like at that venue And I just I just thought that was just a a really neat a unique use of clustering technology Well, and my current favorite today is we have a Probably a 40 tile visual being built in Saudi Arabia right now Not strange but kind of interesting And that's actually going on right now What's the largest rocks install you've seen most of rock systems I'm familiar with are medium sized What was the largest one out there? Yeah, so that's a tough question to answer because the largest cluster Um, we're not we're actually not quite sure because we have a Volunteer registration page that we ask people from time to time if you're using our software Please go to this page and register your cluster there because we use that data to go back to the national science foundation To say, you know, basically to beg and say hey look Look at look at these people that are using our software that you're funding us It's it appears to be, you know, fairly widespread and it's being used You know for all all sorts of different reasons. So please keep giving us money so we can continue to pay our mortgages So we On that page. I believe there is a cluster in Germany that has about 8 000 cpus So you can imagine it's at least it's at least a thousand nodes. Maybe 2 000 nodes um, we know of a cluster out in uh, the university of Nebraska at Omaha. That's all 1200 nodes so um, oh and there's also one Roy draxeth from university of trumpsa who we've mentioned a couple times already He's got a pretty large cluster. I think about a thousand node cluster as well up in uh, norway So you guys have mentioned a couple of times, uh throughout that, uh, you know, you're you're trying to your roots are definitely hpc But you're trying to grow beyond that and you've given at least one example of you know virtualization But you've mentioned a variety of other appliances as well. Um, so what what are you doing? You know to what else do you want rocks to do? What else are people? You know commonly and maybe uncommonly using rocks for besides hpc Again hpc is our core. It's it's where we came from. It's where most of our users came from and still are Our ambitions are for rocks to be a a general purpose Uh data center controller and that's really what it is a cluster just happens to be A subset of what's in a data center, but it rocks again rocks is the hypodermic for putting bits and configuration into your system So we have the The easy the easy path away from hpc Which several of our users have taken is how do I take rocks and just run a computer lab? Because it's still a cluster. It still has a private network and we have users that will set up a nice computer lab from the undergrads Lab lab a comes in at 10 a.m. They pound away on the systems for a while 1150 hit 1050 or 1150 hits students leave And they re-image all the machines to get rid of all the stuff that students do to machines that you don't want on your network And then it's ready for the next class So that that's kind of a simple example to to expand beyond hpc, but we're trying to figure out how to generalize more I We we handle This is just one example where we've we've moved away if this cluster is An open gl machine as opposed to an open mpi machine other than that. It's so the same A database clusters or just clusters of postgres servers or minusql servers We build clusters a cluster doesn't mean hpc Okay, so Okay, and we get some contact information mailing list website irc channel Where are you guys available? Our website is www rocks clusters.org And all the information is there we have to we have a mailing list that you can see from our docs We have every week. We run something called office hours We're one of us and in so far. I think it's been 90 greg bruno Maybe 99 greg bruno Is on for a week on skype for to answer your users questions. Our main list is very active. We've got 2000 3000 2000 2000 users on our main list right now So that's Signal to noise is pretty good And just one more thing there is yearly in about mid may we put on Rocks of palooza And so that's our all hands Meeting we've we've had a couple of them here One up in amd at sunnyvale one in singapore And I think that's all and we've had four right we've had four and We haven't planned where the fifth one's going to be it probably be at ucsd But so every year we have a meeting that lasts about a day and a half or two and a half days And we try and bring some developers together and also some new users we have We generally have like a Basic track that kind of gives an overview of clusters and rocks in general And then we have more of an advanced track where people can Really understand the role structure and how to build their own roles And that that event is always a free event for academics And we usually charge commercial users a little bit so we can pay for the food and take care of everybody