 Welcome to another edition of RCE. I'm your host Brock Palin and Have again with me Jeff Squires from Cisco Systems and the open MPI project Jeff Hey Brock. Hey, you know it was it was just pointed out to me I'm embarrassed that I didn't realize this myself But it was just pointed out to me that we're we're effectively at a year anniversary here Do you realize we've been doing RCE for about a year? Yeah, this will be the 24th episode. We've missed a couple in there for holidays and stuff But no, it's it's been a good time and I'm kind of surprised it's still going It was just an idea and it's been a good time Yeah, it seems to be a good one We've talked to a lot of interesting people a lot of interesting projects out there And I think our stats are up to on the order of you know 200 downloads an episode or something like that So, you know pretty good for two guys just talking to a bunch of random people and whatnot No, no media backing no advertising budget. No nothing just two guys in Skype Yeah, and and thanks so much for Volunteering to be my co-host and gone through all these shows with me and meeting up every week and the industry This is this is interesting stuff You get to talk to a lot of fascinating people who are doing similar yet different kinds of things stuff that overlaps With my work and stuff that doesn't overlap with my work So it's a it's a great opportunity to learn new things which is kind of what I assume that our listeners are here for too Okay, it's all good stuff. So happy anniversary, Brock. Thank you very much, Jeff Let's get on to today's show. Yeah. Yeah, let's go ahead and get started We have today the PetSea project. We have Matt Neapley and Jed Brown. Um, they didn't get exactly where they're both located at but they'll probably be able to introduce themselves Jed, why don't you go ahead and tell us a little bit about yourself? Hi, so I'm at UTH Zurich right now And I've been using PetSea since about 2004 When I was at the University of Alaska Fairbanks cool, I'm Matt and Right now I'm at the University of Chicago Computation Institute and I started using PetSea during my PhD in 1997 and I was at Argonne for seven years as You know developing that full-time Okay, so Are one of you working on PetSea currently and one of you is just a user? Are you both working on it? Are you both users? So I've been contributing to PetSea a fair amount for the last like year or so and Put some new features in but I'm not an official developer. I Think I think you're official now. I think your picture is on the page But yeah, and I'm an official developer. I am one of the guys that maintains it Okay, and this is out of the the MCS division at Argonne. Is the official host of PetSea? right Okay, so how about being mathematical and computer science is just for those who aren't aware of the acronym there. I Did get that right didn't I? I hope so. Yep, mathematics and computer science So how about you give us a rundown on what PetSea is I'm a sysadmin I've installed it a number of times and I have a number of people who used it But I've never actually used it. So what are some of the features and what does it do? well, it's first and foremost a set of libraries and it's a set of libraries intended to solve nonlinear algebraic equations and Its focus is on large distributed memory parallelism and that pretty much necessitates iterative solvers So that's been the focus and it you know lots of people it's it's targeted Primarily for PDEs, but people use it to solve ODEs and variational problems and other things like that Okay When I built a PetSea, I noticed it comes with a lot of optional packages And you guys have actually built your own build system around it What kind of features do these optional packages you add on I notice another guess we had on the show before was HDF 5 project you can use HDF 5 what kinds of things can you enable and disable inside PetSea? so most of the optional packages in PetSea are at third-party solvers of some sort and Yeah, essentially every PetSea object has a plug-in architecture and so what you're doing when you enable these external packages is you're activating those plugins and then the user at runtime can choose whether to To use those third-party libraries in part of their solution process or to skip them So as a little add-on to that what is the history of PetSea? You should probably back up a little bit. Where did it come from and when did it get started? So PetSea was started by two guys Barry Smith and Bill Gropp in 1991 when they both were at Bargon and it was to support their research they were actually writing a book on iterative methods and they couldn't get any current system to Run their examples, so they built something to run all the examples for this book and then Bill went off to work on MPI and Barry rewrote it to be something that was suitable for other people to use So a side-along question in there then this actually predates MPI So did you guys do parallelism with something else before or I guess Barry and Bill did they do something? Before MPI like PVM or something like that Before MPI there was PVM from Al Geist at Ornel But there was also P4 that Bill was writing and so the original PetSea 1 Used P4 and then when PetSea was completely rewritten and PetSea 2 They used MPI because in that was and they used the reference implementation that Bill and Rusty wrote during the MPI standards deliberation Does PetSea only run in MPI or can you run it this as a serial library or like a shared memory parallel Is it only focused on distributed memory? So you can run it in serial the usefulness of building it without an MPI is Really just in breaking that dependency but it all of the all of the examples run fine in serial and PetSea uses sort of a mock implementation of MPI if you want to build it without an MPI library installed Do a lot of users do that or do most people run it on large distributed systems? over half of the people that download it download it for Windows and I Have to imagine that a bunch of those people run it in this without MPI mode because There's in my opinion still not a great install of MPI and Windows you basically download this binary Instead of being able to really build it okay, so the target audience and or just these PDEs but some people can use ODEs So what would be the benefit of using PetSea in a serial version versus using one of the you know alternate solvers that PetSea Just uses have you cleaned up the interfaces or some other helper routines you have? So solving PDEs and in general large-scale problems is a hard thing And it's really hard to know in advance what algorithm will work well for your particular problem and As you add physics or you know somehow change the problem It can also change what kind of solvers work well And so what PetSea provides is this uniform interface and a lot of tools that allow you to Construct your own solvers that may be especially tuned towards the physics that you're working on and so by getting these third-party solvers through PetSea it enables you to compose them Very elegantly and to swap out all of these components at runtime I would just add one small thing and that is that and also installs these things for you automatically So it relieves some of the burden of building these yourself Yes, the build system for a lot of these tools is really really bad So the value What I'm hearing from you is is not only the implementation of a lot of complex algorithms to do this stuff And and to do it correctly because correctness is is a major portion of the battle But also the integration of all these different algorithms, which may or may not be yours They might be sort of the sub packages. Is that a correct summary? Yes Okay, so if I'm I'm an application developer and I'm writing some scientific code or something And I have some PDEs or ODEs to solve or something like that What kind of interface does PetSea give me and what languages can I use to you know call the PetSea library functions. Well, okay, so two points I mean the language is you can use there's a library you can use cc++, Fortran, 7790, Python, MATLAB but The issue of what's the interface? I think first and foremost PetSea is going to give you linear algebra If you're willing to accept the linear algebraic interface meaning matrices and vectors Then all the solvers just work with that linear and nonlinear solvers Time-stepping and so if you buy into that interface up front then It's pretty easy to pull together a code to solve most common problems Okay, and In let me I'm gonna jump back a half a question here in in the implementation of these I assume there's a level of effort put into the performance of these algorithms as well Is performance a primary goal or is correctness more important than performance? How would you rate those two? so correctness is certainly the the most important thing we spend most of our time getting algorithms correct but a lot of effort has gone into tuning the kernels that really matter and so when writing these kind of code usually memory bandwidth is the main bottleneck and then also latency of the network and so we go to great lengths to design the algorithm and the data structures such that it's well suited to that kind of architecture and then some of the low-level tuning such as in the sparse matrix kernels There's been quite a lot of work there So when you talk about these optimizations and you kind of touched on both So you actually do design your algorithms with a distributive memory architecture In mind, but also paying attention to memory bandwidth and things like that Do you also do things like assembly language to you know really squeeze out every flop possible for some of the the you know single machine kernels You know in most things in Petsey, that's just not an issue But in the few places where it is for instance in block methods where you have small dense Operations, you know, we did Chris did write some sse to specific instruction code to make that run faster on Intel for instance Yeah, and in some of my work, I have a particular like a dense tensor product thing and I've Done a tuned, you know as using sse intrinsics in this case, which I can get better performance than through C But usually memory bandwidth is the limit and then assembly doesn't buy you anything Yeah So when building Petsey, what do you guys rely on? I noticed like a lot of these other sparse solvers and stuff they all require a blast library How important is using like better third-party library specific for your hardware or just relying on generic Petsey Which which one's really important in getting good performance when using Petsey? Blast doesn't matter that much because most of what you're gonna do in the sparse stuff is either Blast one or It's some tune thing having to do with a particular matrix format that you have So I say the Blast library is not very important. I don't know Jed. What do you think? Yeah, it's very rarely does it matter for this sort of stuff Yeah Okay, so are you looking at taking advantage of any like obscure hardware? Oh, and recall this obscure It's really becoming more common Accelerators like, you know open CL based systems or FPGAs or GPUs or whatever you want to call them Okay, well, I can I can tell you Because I work I do a lot of GPU programming now but it's my beliefs that you know you put the right algorithm on the right hardware and I think it's just quixotic to to think that you're gonna get very much out of sparse Madfak on these things You'll get a speed up of you know five times maybe and it'll be the same five percent of peak that you get on a regular Processor, so I don't think it's all that exciting, but of course will will do it Fantastic I I can associate with that greatly because we get penalized for some of the same things and MPI implementations we don't necessarily want to do the optimization, but everybody's asking for it and it's the Yeah So to provide a little bit of context here Mad has been doing a lot of work on fast multiple methods, which does map a lot more naturally onto GPU hardware Yeah Okay, let me ask you a real Quick really question. You said that you know underneath the blouse doesn't really matter all that much for for many things How much does the MPI implementation matter is latency and or bandwidth? Important or or other MPI implementation issues So for for these sort of operations, there's two important uses of MPI and so one is the the ghost or halo exchanges where you're doing nearest neighbor communication and the other are the global reductions Which you have to do when computing dot products and norms and such things within the algorithms and At very large scale probably those reductions are Are the most important thing in at least in terms of strong scalability Yeah, I'll point out If you're talking about cluster size, so so not something like ranger but something and like the argon machine jazz that that it is the underlying switch fabric is really important so Gigi just does not perform nearly as well as into the band and very constantly complains about that No, when you say performance, are you talking in terms of latency or bandwidth or both? I'm talking about flops you can extract from a sparse matrix vector product because that's generally the dominant cost and it really I Don't know why I don't I wouldn't say that I know exactly Which piece of the MPI is doing it? But I sounds like bandwidth, right, but I don't know For the for the MPI. I believe it's the reductions. It's what it's it's all latency in that Why well for for large enough, but It does matter whether you have gigi or infiniband for these moderate size clusters and you see huge performance differences You know 30 percent or something So Don't want to get into that too far, but if it a band raises the price of most systems we buy by 30 percent so It pretty much evens out. Yeah, I hear you on that. It's gonna be application and Sight-specific requirements, right? Where do you want to spend your money? Do you want to spend it in less time for your computation or do you want to spend it in longer time? You know, but more nodes or something like this is a classical cluster design issue, right? Yeah, and I mean in my opinion too. I I think those performance gains Probably are in part due to drivers. I mean some of those drivers are just bad and if the people who spend more time Writing a nice driver. We get much many fewer crashes with the mirror net stuff because they have a better driver It's weird interesting so It does highlight the importance of you know, there's multiple levels of middleware involved here And I certainly this is this is dangerous territory for me to talk about the goodness or badness of open fabrics There's a lot of good points to it But there's also a lot of good points to all the different types of networking out there for example a prior RCE cast we talked to brice-goglan from in RIA who did the open MX project And so they've done some very interesting stuff on on significantly lowering the latency of ethernet because they avoid the whole TCP stack For example, and so that would I would love to hear from you someday if you know that kind of stuff actually matters If you could get better performance on your gig E Just by you know swapping out to use a different driver effectively. I Bet that's true So I want to back up a little bit. We talked about what bindings you can call Petsy from what is Petsy itself? actually written in So Petsy is all written in C With the exception of a couple of pieces that are only enabled with C++. That's some of Matt's recent work for example but it uses an object-oriented design and so you can Well use it from a variety of other languages and you still have a real object model And so you talked about everything being like plug-in. How hard would it be to Basically add my own special solver into the Petsy framework if I wanted to So that is supposed to be the really easy thing. So if it's not easy where you're not doing our job, right? That's what it's for Okay, so how long does it take for like a new solver comes by how long does it take? You know one of you guys to add that into Petsy So it doesn't need to be distributed with Petsy in order for it to be a first-class to whatever You know client code might want to use it So for example one of our users or a vendor say can implement a matrix format and Say a preconditioner that works with that matrix format they can compile that stuff into a shared library and send it off to somebody else and They can load it without recompiling or relinking anything Assuming they're on a system that has dynamic Loaded and it'll be first-class That is as far as their code is concerned. It's the same as something that's distributed with Petsy So these are these are true plug-ins per se then you can just put a couple of new plug-ins in the right directory And they get found and discovered and use it runtime that kind of thing Right so that they'll get they'll get loaded You know by the the client program just you know picks up whatever is in a directory say and Then they'll be able to use them and compose them with other With the other pieces of their solvers Just like a native one so on the command line for example How many native solvers is Petsy distributed with currently? Geez well, I gotta look at the web page here, but I can tell you and not too long It's you know, there's probably well, I shouldn't guess I should just look here we go summary table of all linear solvers. Oh It looks like at least 30 if there's been more than 30 preconditioners We yeah, yeah, I guess if you count all of the Well, yeah, and it's all composable right so even if there are 30 or 40 individual things you can You know split the matrix in the blocks use a separate one on each block Have a hierarchy like a multigrid or a grid sequencing take higher key and use different solvers on each level the hierarchy You can Jed put in cool sure compliment stuff So you can use a different thing on the sure compliment part the other part and you can compose Them multiplicatively or additively so you can kind of build an infinite array of these things using an algebra of Operator application Most of this composition you can do at runtime Through command line options or a conflict file So even though it's a library you can Modify the behavior off the command line not just using things like environment variables or other ways I think I think the Primary way to modify the behavior is the command line And I guess that it really goes to the core of what the proper programming style is I mean for us when we solve a linear system We create a linear system solver create a matrix create effector and call solve and your program is maybe 50 lines long And then from the command line what you could do possibly as you could tell it Well, I want a separate solver on each one of these blocks Or like to divide this thing in two and do a sure compliment algorithm And I myself have had you know 50 or 60 options on the command line to try out a very Specific solver for that whereas the code did not change at all Let me ask you a question related to this. So we have a similar kind of functionality in open MPI Where you can specify all kinds of command line things and and you know change the network that you're on change the Collective algorithm stuff like that. We found that some people use that But the vast majority of our users really just do a plain vanilla run And they don't try to tweak parameters on the command line or an environment variables or config files or something like that it's really a smaller subset of power users that Actually go try and tweak things at runtime to extract all the best performance How many of your users actually do do command line or environment variable or whatever? Tweaking to you know suit their needs So I think the majority of users do this a little bit But I think a lot of users probably don't see the full power that that you actually have there But maybe in a different from your application You can easily have an order of magnitude between different command line options Because really Solvers are so problem-dependent Yeah, it could be the difference between you solving the problem and not solving the problem instead of the difference between it Going fast or slow. Ah, okay. Um, so what's what's the ecosystem like around Petzi? I mean you mentioned you briefly mentioned vendors you mentioned core stuff You mentioned the fact you don't have to ship a solver in Petzi itself that somebody can write their own and and use it internally Or ship it internally. How much of that do you actually have? You know, who contributes to Petzi and and what other? Petzi entities are out there. You know, we're gonna do something guest again. Well, okay I'll tell you something. So I think most everything Is eventually done by us because if people like the past X guys Wrote their own Petzi interface wrote their own and part of the interface is writing your can party the configure script that locates you and Downloads you automatically and builds you and stuff and they did all that and it was out there But then they eventually contributed to us and we maintain it and I think that's our model There are external things that take a while to get reintegrated. So Companies that use Petzi like fluent for instance made a lot of changes and it you know I it was it took a while for all of them to get reintegrated, but eventually they do I think I mean, there's no way to know, right? You know when when they tell us about this stuff like when You know one of some of the guy I hope I can say these companies but like One of the guys at Shell had just written amazing stuff and we would have never known it was there if you didn't give it to us, right? But there is there are a lot of external Contributors, but we are basically the maintainers That makes sense. So a lot of stuff develops in the wild, but it really doesn't get you know Well known or well used until someone can just download your package and and just build it and install it and now It's magically there Yeah Okay, so what license do you distribute to Petzi under? So Petzi has its own license, but it's it's similar to that to cause BSD. I guess it's it's very liberal So throwing a throwback way back earlier in the conversation Brock made a reference to the fact that you have your own build system and You said a little earlier too that the configure bits will go out and Download the packages and this all talks speaks to the integration that you guys have done What is since I'm the kind of the build guy an open MPI have a an inordinate interest in this kind of stuff Did you guys use standard GNU auto tools or did you do something else? How did you do that stuff? Okay, I'm really gonna control myself here because we could do a whole program on just build systems Oh, yes, we could but okay, so let me try to be nice I spent a whole year writing a complete auto comp of Petzi in that year it became clear that auto comp is crap so we We rewrote the whole thing in Python so it has you know it gets rid of use of M4 which is not in my opinion a real programming language and It you know It's properly namespaced and all and you can do dynamic loading of modules So you can just have configure modules sitting wherever and they will get automatically loaded and nice plug-in architecture And it does automatic download and build of the packages, which was over 50% of our pets. He made if not more was Hey, I can't build this package and once we just automated it We saw a huge reduction in work that we had to do with user support Where we will eventually rewrite make because Make on Windows sucks because you have to install sigwin. So that is the next thing up So it sounds like you put in a lot of effort into the system, which is actually fairly unusual but I'm glad to hear it because we did we We did things and I agree with you on some of the shortcomings of the auto tools We still use the auto tools, but One of the reasons being that we it would be a Herculean effort to rewrite it, but we had to do quite a bit of Stuff for the discovery of plug-ins and stuff like that because that is really not in the new auto tools model at all And it was quite quite difficult Yeah, that's always nice to hear another complex build system out there for good reasons Yeah, I mean the auto tools is auto tools is built so that you can write LS It's perfect for LS, but if you have any kind of dependency at all, it stinks. There's no composability at all So I want to move to something else the earlier you had mentioned a couple of users of pet C What are some of the major users of pet C that maybe some users have heard about? So there's a lot of Libraries that use pet C For example Libmesh and deal to our finite element libraries. Slepsi is a Eigen solver and Then there's there's some commercial stuff. I guess fluent and I think Chombo and P flow Tran There are Like there are some others simulators like Magpar which is micro magnetics and There are automation Sweets like Phoenix that has pet C underneath for the solver part okay, and what's some of the Largest systems that you've actually heard of pet C being used for I don't know how to really describe this in terms of a sparse matrix, but How would you describe a large system and what are some of the largest things you've ever seen done with pet C? So I know there's been systems over a billion degrees of freedom and There's been runs on at least 130,000 cores Yeah, I mean I think I think it's as big we can run as big as you want to get so the the blue gene P It argon the 130,000 we can run on the whole machine efficiently and rich mills from P flow Tran has done billion plus Degree of freedom runs on Jaguar the XT five at Oak Ridge So we haven't hit the limit of scaling yet What's the what's the strangest use of of pet C that you've seen one of the Fun things that we see as open-source developers is it sometimes your software just gets used in completely Unintentional ways that that you never imagined what have you heard about any of these for pet C? I Think we both agreed. We didn't really know how to answer this question So Something that's done somewhat commonly but is definitely against the philosophy of pet C is that people take matrices Assembled in serial from some like third-party source and then want to use that as a parallel solver and that's a it's kind of an awkward thing to do to distribute these matrices and it's of course not scalable at all because the assembly is Is far too expensive. Okay So I'm looking on your web page here and it says that pet C 3.0. Oh was released a little over a year ago What kind of things are you guys working on for the next release? What are some of the upcoming features that? users can expect You go ahead on this jet and talk about this new stuff in TS first. I think that's cool Yeah, so I've been doing some work on the time integration Part of pet C which has been historically a little bit weak So I put in a few different solvers, so one of them is for explicit systems for example For hyperbolic problems where like Euler equations some were somewhere that you have shocks and you need to preserve some monotonicity and the other stuff which is maybe more interesting to me is for stiff systems and Systems where the semi-discrete form is a differential algebraic equation So this is where you have some constraints a lot of DA use come out as like a limit of an infinitely stiff system And so I put in some general linear methods Which are new like the the paper presenting the method that I use is two years old And so that stuff is in TS and I'd like to see more people use it That's my latest stuff so, you know, I'm very interested in these fascination methods now and They're sort of cousins the pet C supports the FFTW interface as a matrix operator and This has analogs for That come more out of analysis than algebra like fast multi-pole and wavelet operators in the fast-gauss transform And I'm working with some guys in Boston Lorena Barbara to Put these into pet C. We right now. They're like kind of companion packages They depend on pet C and you can do them, but we want to eventually reintegrate them as Linear operators that sort of you can select in the same way you select FFTW and that that should happen in the upcoming year and there is some support for Structured grid manipulation and defining, you know fields over these You know complicated topological objects and you know, it's there, but it has very few users So It's not as good as it could be and so I would hope that we would make some progress in making that better over the over the next year too Okay, and I look at a little closer on your website. I see you have 3.0 patch 10 Assumably that was your your most recent release when what kind of time frame you're looking at for our next release Is this gonna be 3-1 or what are you gonna call it? How big are the changes? I guess Do you know that one? I don't know. I don't know what the timeline is, but I think it's I think various message as of a week ago or whatever Excellent we have a similar philosophy and open MPI will release when we're ready. Yes, we try but sometimes we skip So along the same lines and this is a slightly different question You know, what kind of things if you had infinite time in monkeys, what would you like to add to Petsey? You know, you're kind of your favorite pet peeves or your favorite feature that you just never had time to actually do so maybe the the most interesting to me is this It's sort of an exotic domain decomposition scheme for indefinite problems. It's so a lot of the linear systems that are very difficult to solve are what's called the indefinite and they correspond to like finding a saddle point and Yeah, a lot of the conventional preconditioners don't work very well for that. And so we have one tool right now Which are called sure-complement methods But those have certain weaknesses and Anyway, I would I would like to have this exotic system, but it's a lot of work I think Demetri is working on it a little but I Think if you ask me I am the thing that I find Fascinating and I believe underutilized that the current point is boundary integral formulations Most people by default go to a volumetric funded element method and they'd work and there's very they have very nice properties But um, you know, there are a lot of sweet spots for boundary integrals including, you know, moving domains and Stuff like this where it's really hard to manage a volumetric mesh and I'm do I'm trying to you know get some of that kind of support You know, you could look at it as well. You know, you just end up with a matrix vector problem Anyway, so in some sense we support it But there are a lot of tools that are specialized in these codes that You know make it hard for people to jump in and we have those kind of Kind of support for funded elements. I'd like to put it in for boundary elements One question I always like to ask other developers is what do you guys use for source control and why? I Guess I'll answer Okay, so we use mercurial And at some former time used a bitkeeper, but mercurial is very good That's I'm a big fan of mercurial myself. Okay, and a particular reason that you guys use mercurial Well, when we looked at everything Bazaar was unfinished Darks was very slow for large projects Mercurial with Python and there really had not get was not Didn't have the momentum that it has now I Can I can understand the arguments for get but again? We're we do a lot of work in Python. We kind of are becoming a Python shop And so that made a lot of sense for us How big it you mentioned in large projects, how big is Petsy? How many files lines of code whatever whatever metric you want to use there? It's around 300,000 lines of code Which is funny because it used to be around the 500,000 lines of code And then Barry demanded that we do a cleanup and we were able to shrink it Significantly because you know it's numeric code yet you tend to have boilerplate and a copy it over instead of really trying to reduce That's excellent that you were able to do a concerted cleanup effort like that and reduce the size of the package considerably. That's that's great Yeah, we threw out a bunch of stuff So what is the contact information for Petsy is there a mailing list a website on where you can download it? Yeah, so there's a website if you just Google Petsy P ETSC or you can go to mcs.anl.gov Petsy You can get all the information there. There's two mailing lists one for users Where we get a lot of a lot of questions, you know some of them algorithmic and interesting and some of them Just about, you know, how do I get this installed to work here? and then there's a development list where we you know discuss sort of new things or That's going into Petsy and then there's a address Petsy mate That is where most of the sort of configuration problems should be going Hmm, and if someone wanted to get involved working on Petsy is the development list the best place to get started Yeah Well, I would say pull the pull the mercurial version because I did this about it You know a couple years ago when I you know started Looking into it and you know look at how some of the source is structured and you can add plugins and Discuss on the development list. Yes Okay, guys, well, thank you very much for your time. I think we're done here and This show will be up this weekend. So thank you very much Great your time guys. Thanks a lot