 Welcome to another edition of RCE. Again, this is Brock Palin. You can find us online at RCE-cast.com. You can find all the old shows described by RSS and there's also a direct link to iTunes there. Also, again, I have Jeff Squire from Cisco Systems and OpenMPI. Jeff, thanks again for your time helping out with this. Hey, Brock. This is always cool stuff. I love to come and talk to the various people here and find out new and interesting things that I didn't know before. It's always an educational experience for me and I even get to write about this on my blog sometimes there. Did you like how I snuck that in there? The marketing overlords like me to mention my blog on here, so I have to find a way to mention it every time. No problem. This package that we're going to talk about today I found out about from a friend of mine who's using it not on the system I'm administrator for, but then after we set up the interview, but before this recording I had a user request it, so I've recently gone through the building of this package, which is a gigantic, very versatile package, but I'll let our guests describe that. Our two guests today are Mike Haru and Jim Willenbring, and they are both at Sandia National Lab. Guys, why don't you go ahead and take a moment to introduce yourself. Sure. I'm Mike Haru. I've worked for Sandia since May of 1998 prior to that was at CRE Research, so I've been in HPC for a long time. I and also Jim will find out to actually telecommute for Sandia. We both live in rural central Minnesota and I have worked at Sandia and also teach part-time. I'm a scientist and resident at St. John's University here in central Minnesota. I'm Jim Willenbring. I've worked on the project for nine years. I'm part of most of what happens in Trilino that isn't specific to a Trilino package. For example, the build system, testing infrastructure, general planning for the project. My official role is the framework and tools capability area leader. The package we're talking about is Trilinos. Could one of you guys explain to us exactly what Trilinos is? Yeah, this is Mike. I'll get started. Trilinos is several things and depending on who you are, you care more about one aspect of it or another. First and foremost, it's actually a fairly large collection of independently developed packages. Each package is developed by a small team of people, typically two to five, maybe 87 at most. Each package is focused on one particular set of capabilities that are needed by people who are doing computational science and engineering or other technical computing areas. For example, Solvers is our heritage but we do much more now. So it's this collection of libraries. It's also the framework that Jim mentioned that he manages and keeps track of that allows all of these packages to coexist, not step over each other and build together. And a collection of tools that makes the development and software engineering processes easier. And then third and maybe as important as the other two is or maybe more so is it's a community. A community of people with like interests and goals that meets on a regular basis, talks with each other, rubs elbows and shares and argues and cajoles and everything together as we try to provide this huge body of software to the scientific and engineering and technical computing communities. Okay, so I've been for a number of months now calling it Trilinos, but I guess that's not actually the correct way as pronounced. How is the package actually pronounced? We tend to say Trilinos but certainly there are others who pronounce it as you do and it's okay to be bilingual. Okay, so now I've got the free pass because it's going to take me a while to get it beat into my thick skull the right way to say it. So where does that name come from? What does that derive from? So I arrived at Fendi in 98 and there's a heritage of naming projects by Southwest names, Chaco, Hamas, Sierra, things like that. And that namespace was used up and I started thinking, well, what's another language that has interesting terminology? And I thought, well, Greek is pretty good. And so I started looking for Greek words and here was this picture of a three-stranded string of pearls and that's beautiful. I had this notion that we wanted to do individual packages and have that put together in some kind of meta-framework. And so that image has always been part of the project. And so Trilinos comes from this notion of a string of pearls. The try in Trilinos was actually our grand vision of three packages. And so we thought we were going to do three. And now we're up to, I think, roughly 57 or so. But that's where the name comes from. And you'll see that there's this Greek naming theme, not completely, but throughout the project. And so there are a bunch of packages in the project, many of them which are less pronounceable than Trilinos or Trilinos, if you will. That has got to be the best thought-out rationale from a name that I have ever heard. That is fantastic. So give us a little more of the history, right? So you started this project up and you had this grand vision of three packages. What were the original three packages? I mean, what was the scope back then and how has it evolved? Right, data classes, which eventually evolved into what we call the EPETRA package. Linear solvers, precondition iterative methods, which became the Aztec OO and ISPAC, it split into those packages. And then nonlinear solvers, which evolved into being a package we call NOCT. So those are the three basic capabilities we thought of. The whole project grew out of several impeding needs. One is we had lots of algorithms developers who were given funding to develop new capabilities, that on one hand. On the other hand, we had some very, what seemed to be, onerous software quality engineering requirements coming down to us from the Department of Energy offices. We had them telling us that there are certain processes and practices and assurances you have to make about the quality of your software. And so on the one hand, we had small teams. On the other hand, we had big demands. And so the idea was, how about if we create a kind of federation of algorithm development teams, and then together we can come up with policies and tools that address this big problem that no small team alone can handle. And so really that was the kind of the pot that put this all together. So I noticed when I was building trialinos and I was trying to build every sub-package that I could, the source code was almost 100 megabytes. And when built with all the object files around, it took about 14 gigs. This is a huge package. So how many people are involved with this project and how do you delegate who's responsible for different packages? That's a good question. So each package team has a package leader. That person is responsible. Remember, I said there are about 57 packages. So each package team has a package leader. In addition, we have eight, what we call capability area leaders who are responsible for being proactive in a particular area. For example, linear solvers would be one or meshing geometry. And then on the top is me, Mike Heru, as kind of the nominal overall project leader. So what would you say one of the primary uses of trialinos is, do you guys only work on the individual packages? Or are you actually users of it in a higher level manner and you contribute back? Yeah, we are all users of each other's capabilities. We have a set of strategic goals. One of them is that if it makes mathematical sense for one package to use another, then we create that ability for one package to use the other. So yes, we are our own users in addition to being developers. We have literally thousands of users throughout the world in addition to being the primary deliverer of capabilities in libraries for Sandia and now growing presence at the other national laboratories in the US. And many of the developers. Go ahead. Sorry, I was going to ask the next question, but if you've got more, go ahead. I was just going to add many trialinos developers are also users via other projects that they work on in addition to using other packages within trialinos. So there's a lot of higher level knowledge that comes back down into the implementation of the various packages and deep knowledge of the requirements for the capabilities. So all these different packages in in trialinos, how are they? Are they related to each other? I mean, did they share common interfaces like one package? Well, let's say I have two or three different packages. They're all different implementations of the same interface or are they just related through the build system and they just provide different capabilities for different users? Or what is the relationship between these? Well, there are kind of. Okay, I'll take this one. In a way, there's almost an old trialinos and a new trialinos and then there's packages that don't quite fit into that mold. But many existing trialinos packages are based on the ePETRA object model. So they understand what an ePETRA matrix is or ePETRA vector is and it uses that common interface. And as long as your package can understand the ePETRA object model, it can make use of various solvers and preconditioners and other tools. Now the newer trialinos, ePETRA is a package that does computations only in double precision. There's also a ePETRA object model that the templated version. And this is kind of the, as I said, the newer trialinos and there's a host of packages built in much the same way on top of that. But then there are also a few packages that maybe don't use these models natively, but perhaps can understand them. But that's a primary method of interoperability, I'd say. So you're referring to objects a lot. I take it trialinos is built in C++ or Python or some sort of object-oriented language? The majority of it. Yeah, the majority of it. We certainly tend to use object-oriented principles and C++ is the dominant programming language. But there's also some C and we've actually aggressively started adopting FORTRAN object-oriented capabilities in a package called FORTRALINOS. So what is the breakdown of your users? So in the MPI community, mostly what I hear is, oh, yes, the scientists and the engineers and the, you know, not natively computer people per se prefer FORTRAN because FORTRAN as a language is absolutely fabulous at what it does. It does numerical computations very, very well. But, you know, from what you're saying that you're actually going after the OO kinds of languages, the C++, which at least in my world I wouldn't natively associate with scientific computation. And also the object-oriented features of newer modern FORTRAN and stuff. So are you seeing big uptakes of that in the user community? Well, so it depends on which user community you're in. In the engineering community, and Sandia's heritage is engineering. So there's almost no FORTRAN used in the applications at Sandia. And so it's very natural for us if we're servicing the needs of the Sandia community to use C++. Even outside of that, there's a growing community of applications that are not written in FORTRAN. And even if you are a FORTRAN application, most applications are able to interact with C++ in a meaningful way that's not too difficult for them. So I think it works pretty well. Certainly I think there's an uptake issue in certain application areas. But over time, and I think actually the migration to many core-based GPU-CPU type of nodes is going to accelerate this change. So what about the argument of any type of trade-off between performance and being really low-level and ease of programming and abstraction using object models? Was this a discussion when deciding what Trialinos and the architecture of Trialinos was going to be? Yeah, sure. And if you look carefully at the design of Trialinos, we have a hierarchy of interfaces. And the very lowest levels are where you have the maximal control over performance-impacting aspects of software design and use. And then you can increase, you know, go up to more and more abstract levels. And you don't lose a whole lot, but what you lose is maybe some control over the path taken through the software. And so this multi-level approach really allows you to pick the level of abstraction that you want in the use of what Trialinos provides. So there's another well-known package coming out of the National Labs that, at least from my high level, Sis Edmund, please build this for a user perspective. Petsy, what is the relationship between you guys and Petsy? Yeah, that's a good question. Petsy predates Trialinos. And in fact, you know, when we started the Trialinos project, even before it was Trialinos, there were people at San Diego using Petsy. There were people using another package called Aztec, which was a precursor and now is actually one small portion of Trialinos. And so we are certainly aware of Petsy. Petsy provides a lot of good capabilities. It's a highly regarded collection of libraries. Their focus has been purposefully on PDEs. And so they do PDEs very well, both structured and unstructured, linear and nonlinear solvers. And so it has a nice following, particularly in the FORTRAN and the science community. It has a focus on being a homogeneous collection of software capabilities. Even if they're pulling something in from the outside, they will integrate it into Petsy, and so building it isn't a challenge. Trialinos has taken a different approach and tried to provide a community environment for people to come in and make their contributions in forms of these packages. And so two things have happened. One is on the edges between packages, there's some interesting dynamic stuff that goes on, and it can be a challenge to build pieces of Trialinos because of this. On the other hand, because we have this loose confederation of people, we've been able to grow the project very quickly and provide capabilities that we just couldn't do if we were trying to have a more centralized control over things. So Petsy, the capabilities of Petsy are very good, really focused on PDEs. I tend to tell people if Petsy has what you need, it's probably going to be easier for you to use that. But if you think that you're going to grow outside of what Petsy provides, you might think about using Trialinos. Or if you already need some of the things that we provide that Petsy doesn't provide, then you should consider using Trialinos. So I noticed when building Trialinos that there was actually an option to tell Trialinos where your Petsy install was. Are you actually adding your own abstraction layer or will Trialinos use Petsy for certain functions? We have our own abstraction layer and we have made a commitment to that if you're an existing Petsy user and you've committed to using Petsy's data classes, then you should have access to what Trialinos provides because all of our solvers and preconditioners access matrix and vector data via abstraction layers. And so we can certainly wrap Petsy's data structures and you have access to everything that Trialinos provides. So yeah, it's a nice thing that we also do that with another popular package called Hyper that provides scalable preconditioners. So yeah, it's a good thing to do. We're all good friends with each other and are trying to leverage each other's capabilities as best we can. Petsy also wraps several of Trialinos capabilities in particular ML and they provide access to some of the Zoltan capabilities, each of these as packages in Trialinos. So let me ask this in terms of my home cord and Bias and whatnot. How many of your packages are paralyzed versus how many of them are serial? In particular, do they use MPI or OpenMP or what parallel abstractions do they use? Yeah, good question. So all of the packages that make sense to run with MPI are enabled to use MPI. We actually don't have explicit calls to MPI except in one class that is a parallel abstraction an implementation of our parallel machine interface that uses MPI. So we have using object-oriented principles, we've isolated MPI-specific dependencies to one data class. And of course our examples and other programs need to use it. So MPI is heavily used. We also support OpenMP in the EPETRA package and we are heavily involved in development for data classes and algorithms that use CUDA. Thrust, which is a layer that sits on top of CUDA, threading building blocks from Intel and certainly P-threads and other types of parallel programming environments. So if you built an MPI abstraction class to use everywhere else, does that cause any difficulty trying to say I have an existing application and I want to use a solver and Trialinos has it? So how hard is it to kind of bring Trialinos into an existing traditional C or Fortran or C++ that's directly calling MPI? It's very easy. In fact how you construct our, we call it EPETRACOM and there's equivalent one called TPETRACOM. How you build one of our COM objects is you pass in your communicator, your MPI communicator and then we access the parallel machine through your communicator. Okay, so then related to that, say I wanted to use all your lower level building blocks of Trialinos and basically make my own custom package. How difficult is it to just kind of start creating my own add-on to Trialinos? We put a lot of effort into making it easy, but that doesn't necessarily mean your mileage may vary. I really don't think it's that hard. I think it's much easier than building up your own software stack because you just have to implement that interface. If you have an enormous piece of software existing then that's going to take a little bit of work to integrate the EPETRA or TPETRA object model into that. But the intent here is that you're providing all this infrastructure so that random Joe physicist or mathematician or whatever can just focus on their solver code and the rest of the stuff is kind of handled by magic, right? By the rest of the infrastructure and things that you're providing. Going back to what you said earlier that there's all these really strong demands coming down from management and you need to have very high quality software, but that's kind of beyond the average developer. They want to just do their thing that their expertise in and then all the rest of the infrastructure just kind of provides the rest of those high level guarantees. Is that kind of the rationale here? Yeah, it is. The overall design principle that we adhere to is don't repeat yourself. We like to make, if there's a capability that exists within what we provide, you should be able to easily tap into that and build on top of that, adding the delta that is your unique specialty or your unique need and then leverage everything else that's available. And part of the beauty of object-oriented programming and abstractions and inheritance is that it's fairly easy to do that in software. And depending on the level that you want to add to within Trelinos, if you want to add a new solver or something like that, you can actually leverage the Trelinos build system and make it look like a Trelinos package fairly easily and then you could just install your package and library along with all the other libraries. If you're building strictly on top of Trelinos with an application, then you can just install Trelinos and use it, but there's a interoperability to allow people to add capabilities and really whole new packages into Trelinos pretty seamlessly. So going kind of along that and also going back earlier in the conversation, you said community is very important. How does your community function? How do you have all these independent teams together and get the team leads together and say, all right, it's time to make a release and who makes decisions about where the interface goes and all that kind of stuff? How do you guys function? Well, we're a fairly distributed team. Certainly the bulk of participants are in Albuquerque, New Mexico, Sandia's main site, but we also have a number who are in Sandia and California in Livermore area, and we have myself and Jim who are in Minnesota and people at various other labs and universities. So we're distributed. A lot of our communication occurs via mail lists. We heavily use those. We're migrating to more interactive, wiki types of setups, but that takes some time to do. There's nothing else for security policies and issues related to that. We also have an annual Trelinos user group meeting that has a developer day associated with it, so we meet face-to-face on that day. That's usually right around the first part of November. And then we also have a spring developer day that occurs in April or May depending on schedules. And then we also have monthly leaders meetings for the team. We have an advisory group that involves the external advisors. And then we also have a board, what we call a capability leader. These eight people and we meet fairly regularly again by phone. So there are all these opportunities for exchange of information that occur. And there are many heated and passionate discussions that occur on the mail lists or by phone. And that's where a lot of the kind of community aspects come out. Where we negotiate for good ideas. I think the biggest thing that we bring to the table in this community spirit is a meritocracy. There's nobody who's in charge of the project because they were imposed by management. It's really all about who has the best idea and the best idea gets to win. And we're going to work to solve an issue until the best idea emerges and then go with it. So I'm curious, who exactly is using Trelinos? Of course, you guys at the labs are using it. What are some commonly known packages that people may be familiar with that are currently using Trelinos? So our user base is very broad. Some of them we can't really say who they are. Probably you can understand why. But I'll give you a few examples. So the climate community has actually started to adopt Trelinos in a fairly serious way. So across several different components of the community or systems model project in the atmospheric area, ocean modeling, sea ice, ice sheet modeling, all of these are components in the community climate system model. And they all have been charged with doing what are called multi-decadal simulations. They're explicit by nature or semi-implicit. And so time steps are a serious bottleneck. They can only take so big a time step. And so they have to transform their formulations to what are called implicit approaches that allow much larger time steps so you can go out multiple decades much more quickly than if you have to take these tiny time steps. So we've been involved quite a bit in the climate community with development of solvers and integration of solvers into those codes. We're also quite involved in the nuclear engineering community. There are several large projects that have started up recently. There's a project called CASEL which is intended to extend the life span of nuclear reactors that exist in the United States and others. But I'll stop for a moment. That's funny. My training is actually in nuclear engineering and that's how I found out about Trelinos was a guest talk and their package was built on top of Trelinos. Yeah. So then what kind of scalability for the MPI portions I assume every package is a little different but what kind of scalability are we reaching with Trelinos? Yeah. So most of the packages obtain their scalability as long as the algorithm itself is scalable via our underlying data classes. And so the scalability is actually very good. Trelinos has run on the full scale of Jaguar PF which is the largest general purpose machine on the planet right now. We haven't run on the Chinese Tianhu One system that's GPU based but we actually could. We have the infrastructure in place to run on that kind of machine. The biggest limitation many of our users face right now is that a number of our production quality packages that are very robust, well understood, are limited in the size of problem they can solve because their integer size is 32-bit. And so that limits the size of problem that they can solve to plus or minus 2 billion. We have new capabilities, this T Petra package stack that Jim mentioned earlier gets rid of that limitation. You can have an arbitrarily large problem but there are some people who are production oriented built on the older packages and that's actually the biggest limitation to scalability. All right. So as a software developer, I always like to ask some questions about the code base and the management and things like that of other projects. Just add a genuine curiosity because sometimes there's genuine passion in some of the decisions that are made. Some people don't really care but some people have very, very firm reasons for doing so. So looking on your website, I see that Trellino's 10.0 switched from the GNU Auto Tools to CMake. And I was just curious as to if you could tell me a little bit about why. We had several reasons for switching actually and the decision was carefully thought out. A paper was written about it and it was discussed for well over a year, I think. But the strongest drivers, one was improving Microsoft Windows support and specifically being able to build Trellino's outside of Sigwin. Another significant reason was to improve dependency tracking. The Auto Tools really had problems figuring out exactly what needed to be recompiled after changes were made to the code. And sometimes seemed to compile too much and sometimes not enough and that really caused problems. And if you're building all of Trellino's and if you're working on one of these low-level packages like E-Petra, you really need to recompile everything before you commit to the repository. It really got to be a mess with the dependency tracking of the Auto Tools. Another was improved shared library support. We had kind of hacked shared library support into our Auto Tools system. And there's just a little less maintenance. It's a little easier to add an additional package. Fewer lines of code to take care of overall, I think. Also, the C-test and C-dash that you can use with CMake very seamlessly, the testing and then also the dashboard where you can display nightly test results and such. That was a big win for us because we had our own homegrown system. And it had some nice features, but it had some pretty severe limitations. Now we can do things like have test timeouts, make sure an individual test doesn't run too long, make sure a build doesn't run too long, make sure that builds and successive days don't stomp on one another if it runs an entire day. Cool. Those are all very good reasons and very well thought out. Let me ask you this. Also, what is your source code repository technology of choice and why? We used to use CVS and we've moved now to Git in the last couple of years. And the reason for the switch was that CVS isn't very efficient when the code base gets really large. CVS certainly has some nice features, but now with Git we can do lots of great things. You can collaborate with people much more easily. You can check things into separate branches and work with a small number of people without making your changes public and then bring those back. You can have gatekeeper situations if you want to much more easily with Git than CVS. There are a number of reasons there as well. So what kind of licenses Trilino's under? It's mostly licensed LGPL. We're migrating toward a BSD license. So just to decode that, LGPL is the GNU lecture general public license, which is an appropriate license for embedded software. It has a copy-lifting mechanism where if you make changes to Trilino's code proper, you're obligated to make those changes available back to the open source community. But if you embed Trilino's in an application without making modifications, then that's okay. You don't have to make your application open source. We're moving to BSD strategically because we're finding that some of our industrial partners and vendors now who support Trilino's. For example, Cray provides Trilino's as part of their scientific library's offering. They pre-compile it for their XT series of machines and they provide a little value added by boosting the performance of some key kernels. We find that these people and industrial partners are major companies in the U.S. that are using Trilino's as part of their software foundation are very uncomfortable with the copy-lifting aspects of LGPL where BSD doesn't have anything like that. So we're moving over time to BSD and we're going to move all of it as much as possible. There are some hairy edges where that's hard to BSD because we get far more value out of these collaborations than we do out of the principle of maintaining open source. Maybe you could clarify a little bit about, since I come from one of these large companies who tend to prefer BSD-type style open source licenses, what do you mean by migrating to, is there legal paperwork and things that you're trying to get all proper or what does that mean? Well, so when you go in, so we're doing a package by package. Again, these packages are modular pieces of software. It's a great management tool for lots of things and one of them is migration of licensing policies. So little by little we're going through and taking each package and we're declaring you are now a BSD package, not an LGPL package. And so that's what I mean by migrating to BSD. And yes, it's paperwork and we can't just say we want to switch from an LGPL license to a BSD license. There's more legwork associated with the move than simply changing our mind. The other thing we're doing at the same time is we need to recognize that there are community members who want to make contributions to trulinos and that who don't want to give up their ability to assert copyright. And so we're moving to a model that will allow external institutions to put their copyright into the source code that they contribute to trulinos as long as they also agree to licensing it under the BSD approach. So I want to back up a little bit and ask a different question that I'll splice in. So what's coming in future versions of trulinos? What do you guys plan to do? Lots. As I said, it's a community. Each member of this community has their own set of strategic goals and ambitious activities that they're trying to lead the community in their particular area of algorithms and software for scientific computing. So we have leading edge efforts and everything from data classes for scalable mini-core systems, algorithms for scalable mini-core systems to the latest in linear solvers, block iterative methods, recycling methods, so-called communication avoiding methods that's the best in nonlinear solvers. Now we're moving towards optimization, embedded optimization, uncertainty quantification, you know, stochastic PDEs and, you know, scalable meshing and all of these different capabilities. And because of this community model, we can grow simultaneously in lots of different directions. If I could say our ultimate goal is to provide a solution that's high fidelity to a particular problem of interest with error bars. So we can, so the goal is not just to provide an answer, but to also give you a sense of confidence in that answer. So what's perhaps the strangest use that you've ever heard of Trellino? Something that you really didn't intend. You hear that somebody's using your project to say, wow, okay, I wouldn't have thought of that. Well, Jim, you had one. Well, mine is really more of a fun use. Maybe it wasn't intended either, but professional racing teams use Trellino for their design, designing their vehicles and such. Yeah, so a particular Formula One race team that we won't mention has heavily used Trellino to improve the design of their vehicles. Other uses I know of digital dentistry. We had a download from a company whose sole purpose is to provide digital capabilities in the dentist's office. And so that's very interesting. Also, we've been getting into areas, people who are doing medical imaging who try to deep blur images, not in some kind of big computing facility but right in the clinic. And so all of these many core capabilities for GPUs and CPUs that we're developing make possible. You can cart in a Dell workstation with a GPU and you can do things in the clinic in terms of modeling and simulation and improving the quality of the diagnostic devices that physicians are using by using Trellino. So what's the website for Trellinos? So Trellinos is available from trellinos.sandia.gov. That is the main portal to the project. Everybody who gains access to it typically starts right from that page including the developers. And if there are things that you don't like about the website or you want to see changed, we're always open to try to improve it. Okay guys, thank you very much for your time and this will be out soon. Okay, thank you. Thanks Joe. Alright, thanks. Alright.