 Welcome to another edition of RCE. Again, this is Brock Palin. You can find us online at rce-cast.com, where you can find our entire back catalog of over 100 episodes on research computing, high performance computing, and scientifics, anything. I have again with us Jeff Squires from Cisco Systems and one of the authors of OpenMPI. Jeff, thanks again for your time. Hey, Brock, yeah, this is always great stuff. I always learn to learn new random things over the people and projects that we run into on the internet. So with that, I don't think we really actually even have any announcements today. So why don't you just tell us what we got on tap for today? OK, so today we have two people from the, I think, it's referred to as DEAL2 project. So guys, why don't you go ahead and take a moment to introduce yourself? Yeah, so I'm both Kent Dangers. I'm a professor at Colorado State. And some 20 years ago, I was stubborn enough to not like the software choices that were out there and just decided to write my own. And that turned into a 20-year-long project that they're still maintaining at DEAL2. And my name is Timo Heister. I'm an assistant professor in mathematical sciences at Clemson University in South Carolina. And I've been involved with the DEAL2 project for a couple of years now and became a maintainer when I did my postdoc together with Wolfgang. So give us the elevator pitch. What is DEAL2? So DEAL2 is a member of the great lineage of software libraries that we've been doing in computational science. So think about something that starts with BLAS and LAPAC. And then linear algebra libraries came along like Petzi. And truly, not that we're on this podcast before. And then sometime in the early 2000s, people started to build finite element libraries. So finite element is something that we use typically to solve partial differential equations in continuum mechanics. So for, let's say, solid mechanics problems or for flow problems. And there's sort of a common set of tools that you need to have to build these sort of codes. So you need to have meshes, for example. You need to have description soft, what we call finite elements. You need to have linear and nonlinear solvers. And all of these sort of things is what we put into DEAL2. So I should say that it's a library written in C++. It's got about a million lines of code. It's probably one of the bigger ones in this field. It's got a fairly sizable community of a few hundred people who use it. That's sort of the 30,000 feet view of DEAL2. All right, so for those of us who are not in the field, give us a little more on what are these finite element analysis things? What are partial differential equations? And why do we need software packages for them? OK, so partial differential equations describe things in physics. They describe relationships of functions, and for example, in forces and other kinds of things. And the finite element method is a way to solve these differential equations numerically on a computer. And the interesting thing about finite elements is that it's even in the name, it's finite element analysis, is that it has a very strong mathematical foundation. So now you mentioned, or actually Wolfgang mentioned, a bunch of different numerical packages out there. Why are there different ones? And why do we have packages for these anyway? Why is it hard to write software like this? Yes, and it's repetitive and redundant if everybody does it themselves, right? That's why we have these packages and have them since, well, 1950s in Fortran, right? So what we do in DEAL2 is not redundant to what, let's say, truly nice and petty do in the linear algebra realm. And it's not redundant to what, let's say, Blas and Latak do for the dense linear algebra, right? They all built on top of each other. And I think what we as a community have just learned is that over the last 30 years, we built libraries at higher and higher levels, right? That we started with dense linear algebra and then we started to build libraries for sparse linear algebra and for parallel linear algebra and then people started to write libraries for partial differential equations. And now there's libraries for a certain quantification and these sort of things. So they built sort of a stack where you can describe problems at a higher and higher level. And of course, all of them are, let's say, their own speciality, their own special area of computational science where you have to have specialized knowledge. So not everybody understands how exactly to implement, let's say, a GMRES iterative solver. And not everybody understands how to implement finite elements. And not everybody has to because now we have all of these libraries that provide these sort of functionalities in a more or less generic way that people who are not specialists can use them. So you've thrown around a lot of different things. You mentioned PDEs, but just walking around floors at places like super computing and stuff, people will normally think of finite element as FEA. And that means just structures calculations and user codes that we're familiar with for structures calculations. Sounds like that's not really the case. Sounds like FEA is just a method to solving a specific set of equations. Yeah, that's right. So the finite element method is a way to solve all kinds of partial differential equations. It could be in fluid dynamics. It could be in solid mechanics. It could be geosciences, magneto-hydrodynamics, any kind of multi-physics problems, nuclear physics, astrophysics. And I could go on and on. So all kinds of physical phenomena that can be described by these PDEs. So give us a little bit of background. You're talking about well-developed libraries and code reusability. What prompted you to develop a new FEA software library? What's the history? Yeah, so I started this when I was doing my diploma thesis. So that's something like a master's thesis in Germany in 1998. And at the time, you know, a lot of these libraries that we have today, 18 years, 19 years later, they didn't exist, right? I mean, Petzi was just so invented, lots and lots of it was there, but there were really not a lot of libraries for this out there. And a lot of research groups were sort of starting to get into the field and were writing their own software libraries. And in the group that I was in in Heidelberg in Germany, were a couple of libraries that were written by people who were really good mathematicians and maybe not such good software designers. And so I'm one of those people who grew up in the late 80s as a teenager. So I had access to computers from when I was 14 or 15. And I had spent a lot of time programming. So when I became a master's student or get a PhD student, I sort of had a background in software design and I saw that the software libraries that are out there were not all that well-designed because the people didn't have a background in it. And then I probably also was more stubborn than most others. And I decided, well, to heck, to hell with this. And I just decided I'll rewrite the functionality that was already there and put that into a software library that I could build and shape and form the way I wanted it to. And that turned out to be a really useful thing because at the time, like I said, there was nothing there that was, I mean, let's say the World War Web wasn't really there. So when we put this onto a website because we could, so that was in 2000, we put it on a website because that's what everybody was doing at the time because websites were new. And it turned out that people actually liked that and downloaded it and started to use it. And so we built a whole community of people around this that now contains a few dozen people who contribute code every quarter or so. And so it was a little bit of an accident of being too stubborn to use what was there and wanting to do it right and then come up with a design that apparently seems to work even 20 years later. So before we go too much further, I do want to ask, where did the name come from? Deal.ii. Yeah, so you have to attribute that to somebody who was 26 or 27 at the time, maybe not a professional. So there was a library that was called Deal at the time. There was the Differential Equations Analysis Library. And that's the one that I replaced. And so we needed to come up with something that had a different name, but I liked the acronym Deal. So we wanted to make it Deal 2 and then we just needed to come up with a catchy way of doing it. So we said Deal and then a Roman 2 behind it, that's where that name comes from. Gotcha, okay. Well, let me ask another history question then too. So back in the late 90s and whatnot, that was kind of the bad old times for C++ compilers because there was so much new going on and the C++ compilers were a mishmash of supporting different versions of various features in the language and they weren't always very stable. I remember, because I did some C++ work in the MPI area in the mid to late 90s and it was kind of difficult. What inspired you to do Deal 2 in C++? So that, I mean, I knew the language. I had started using C++ in, I don't know, maybe 93 or so with compilers that were even worse. You could download the standard template library from HP at the time. And there was no compiler that could actually compile it or at least I didn't have access to any. But I liked the ideas that were in there and it was, I mean, you're exactly right. It was difficult at the time to use that because with every version of the compiler, some code would break. Every compiler that you try understood a different subset of the language. So it was more challenging but it was also clearly a language that everybody was going through. I mean, you know, you see that libraries like Petsy, for example, that are written in C are still object oriented. And I mean, C++ sort of codifies this way of programming. And I think in the PDE world, that's just what everybody was doing. So I had access to a library that was called Diffpack. It's time that came out of Norway that was written in C++. MPI is easy to use, VSC, Petsy was written in C. So a lot of that part of my community was actually going towards C++ and it was not the controversial choice. And I mean, every finite element library that's come up since then that is more or less generic is written in C++ as well. So is DL2, it's developed in C++. You've mentioned it's a library. Is it also a standalone application that can be called as a library or is it just a library you need to code or is part of other larger tools? So DL2 is a library. So it's not an application. And we see it as something that provides building blocks that you need to write your own program. Of course, we include tutorial programs that do something and it's used by other people that write specific solvers to solve specific problems. But DL2 itself is just a library that provides building blocks. Okay, so if you had to say there was a common use case for using DL2, what would you say that was? That's actually really diverse. So one of my favorite jobs is to go through the list of publications that cite us. And you find that there's people from fluid dynamics to solid mechanics to crazy applications. I mean, really things that we had never seen that we had never heard of. And that's a lot of fun to see that. To add to that, I guess, many of us, if not most of us that are involved in the development use it to do numerical methods research or finite element research, set the blocks together and try out new methods or solvers and so on. Okay, and actually that's interesting to touch on right there. So in terms of the research itself, this may not be something that many people are even aware of that exists because in their mind, when you write a solver or you write an equation, you're just literally taking the symbols off of a sheet of paper or a blackboard and inputting it into a computer. But that's not what these solvers and things are, right? No, it's quite complicated. And I guess one of the issues is that you need a very wide range of different things to actually show that something that you're doing is working. So I mean, the library doesn't just do finite elements. To do finite elements, you need geometry, you need measures, you need linear algebra, you need parallel computing, you need visualization and so on and so on. So are there common like stacks that provide that? Like what would be either an application that provides all of that? It's kind of like an end user application that someone can consume or a common set of tools that work together that DL2 is inside of the solver and they probably don't even know. Yeah, so one example is a code called aspect that is to do a simulation of mental convection. So it's geosciences. Both of us are also working on that. So that builds on DL2 and provides an application interface to do mental convection simulations. Another example is OpenFCST that is a group that is developing a code that does fuel cell simulations where also under the hood DL2 is used but it's not visible to the user. So take a break here for a second. I actually wanna jump back. I think there was another follow-up question that I should have asked right after the last one where I was saying it's not just copying mathematical formula into a computer, you have to actually implement it, da, da, da. And then you gave the answer and then Brock jumped in there. And after a moment I was like, oh, there's something I should ask on a follow-up. So we're gonna go back and splice this back in after the fact here, but pretend it's right after that. And I wanna ask you another question here. So give a B to silence and I'll ask you. Okay, and kind of what I was going for though is that there's actually a lot of skill and well, not just skill, but also research because it's not always known like to take this formula, it takes an iterative method and what is the best iterative method? And then actually how to implement that on a computer is actually a difficult problem sometimes too because just of your kind of constrained in the way of Von Neumann architecture works. And so the canonical example is if you do a matrix, matrix multiply in the trivial three loop method that is horribly inefficient and terrible. And so there are much, much more complicated ways to do matrix, matrix multiplication when you're running on a computer because those are much better matching for the underlying architecture. And so I would imagine that is a lot of what DL2 is that there's a lot of expertise and cutting edge research on how to solve these complex partial differential equations and other types of problems in super efficient ways that your average developer doesn't know how to do. They would rather just call some kind of function in C++ and say, hey, here's my inputs, here's the structure of what I'm doing, please solve it for me. And then you go off and that's where all the magic actually happens that you have spent years literally implementing. Is that a fair characterization? Yeah, I think that's actually pretty accurate. I mean, a lot of it, let's say, in these sort of libraries, a lot of it is that you need to find divide data structures and the algorithms that work on it and then you provide the user was a place where they can jump into your implementation of the algorithm and the data structure but they will never see how it's actually done. That's exactly what all of these libraries are about. And to cross over into my realm because I do a lot of parallel computing kinds of things, at least in some cases you can actually parallelize the solvers behind the scenes too. So it's a type of thing where you can, the user calls a function saying solve this equation and if they just happen to have, say, a 32 server cluster behind it, they might get the answer in that much faster time. Does that kind of effects happen as well? Yeah, so we provide interfaces to parallel solvers and I mean, it requires you to write an extra 50 lines of code or so to say something about well, so what kind of data do you actually locally need? So typically you split a vector across machines but then each processor needs to know a little bit about some pieces of information from other processors. So ghost elements. So that part you need to say what parts do you actually need but the rest of it all works automatically. So if you want to split up a mesh with a hundred million elements to 10,000 processors that happens all under the hood. But there's of course a few 10,000 lines of code behind that. So can I just, well you just did there, sounded a lot like partitioning to me. So can people just pair this up with something like Metis or something like that to be able to take their mesh and partition it and does that come into a format that deal can understand? Yeah, so well typically we start with meshes that are not partitioned if we do it ourselves. And you can either do this with Metis, so deal two has interfaces to Metis but what we typically use is a library that's called Pforest that is specific to what we call adaptive refined meshes. So Pforest really has tested all of these things out to 10,000 or 100,000 processors and that's exactly the sort of thing that we need for partitioning of meshes. Okay, so what I'm really hearing here is is deal two is really scoped to, it is just a finite element solver package and all the other things you need to be able to take your real world problem, a physical geometry or whatever it is, discretize it, turn it to a mesh, apply properties, partition it, distribute it among processors. That's all other functionality that comes from other projects out there like Metis as well as is on the user's part to make that functionality specific for whatever their physical properties are. Yeah, so we implement a lot of the specifics of these data structures in deal two. So oftentimes you need to store information on a cell, for example, and that's something that we implement ourselves. But for example, partitioning is something that can be described in a very abstract and generic way, and that's just the sort of thing that we typically import from other libraries because there's people out there who are so much better than this than what we could possibly do, right? It would be a shame to re-implement what they already did. So you talked a bit about the parallelism there. What kind of parallelism do you support? Is it shared memory parallelism or distributed memory parallelism like MPI style or what? So we do all of this. We use MPI for the very large computations because that's basically the only thing you have out there if you want to do scientific computing on distributed memory machines, but then we also support threads on multicore machines. So if you want to do things on typical laptops, let's say with eight cores, then you would use multi-threading for that. We've played a little bit with GPUs and that in our community has not really been found to be very useful so far. So you mentioned GPUs there. Have you tried any other type of accelerator like FPGA, Xeon Phi, something like that? Something like different architectures. Yeah, so we've played on the Xeon Phi's but I think ultimately for the sort of things that we do, not just we but the community has struggled with this because we often work with really complicated data structures and ultimately I think it comes down to if you have simple data structures then GPUs and accelerators are a great tool. But if you work on, let's say, sparse matrices and complicated mesh data structures, we'll be fine if that is just really difficult to put them onto GPUs in any way that makes it efficient. I mean, we can match the same speed that we have from the CPU, but we don't really gain very much. So do you guys get a chance to play on some of any of the new and upcoming architectures? We talked about the Xeon Phi's, we talked about accelerators, but what about things that are coming down the horizon like processors in memory and 3D memory and do SSDs like the fast RAM kinds of things? Do they play an into effect? And I realize this goes more into the software and computer engineering side of things rather than the mathematical research side of things but just the fact that the architecture of some of our compute machines is actually changing a bit and does that affect how you write your methods and algorithms? Yeah, so in some ways, of course, I mean, we try, for example, to parallelize with threads, many operations that we do. I think a lot of the technologies that you mentioned really require sort of access at a lower level on simple data structures. And so the typical place where I would say this plays a role is, let's say, in linear solvers where, let's say, we created the linear systems in DL2 and then we handed off to libraries like Petsy and Trilanos who may hand it off to some other thing that works at a lower level that can use all of these sort of things. But that's sort of how we see that ecosystem working that at least we help, that we can offload these sort of things on two other libraries. So how does one, as a user or a consumer of this library, how do you learn about these kinds of things, not only from the mathematical perspective but from the API perspective and whatnot too? Because I would imagine documentation takes you so far, but then how do you get to the next level to become an actually proficient DL2 user? Documentation is a really important part of the DL2 project. We spend a lot of time on writing documentation in various forms. We have now, I think, 57 tutorial programs that highlight certain things and explain parts of the library and how it is supposed to be used. And we have a lot of other documentation about all the functionality of the API and so on. That said, Wolfgang has been recording video lectures and he can probably tell you about that. Yeah, so this was an experiment that did for a class of mine where I just wanted to flip the classroom and then it sort of got out of control. So I recorded these lectures that, you know, the idea was to present the material but then also to show interactively how you work with DL2. And as a matter of fact with other tools as well. So I show how to use Git and how to use visit and pair together. We recorded something, a pair of you. So it was more the idea to give newcomers to the field the opportunity to look over the shoulders of people who've been doing this for many years and how to see how they interact with something. And that turns out to be a really useful tool apparently to learn how to use DL2 as well. Let's look a little bit more. I'm not familiar with FEA myself. I was actually one of those people who thought FEA meant you're just simulating cars crashing into walls. Besides numerically solving a PDE with FEA, is there an alternative? Like what would you say if someone's trying to solve a PDE is FEA the only way to do it numerically on a computer or are there other methods out there and what are they? There's many different ways on how to numerically solve PDEs. So you might be, you or the listener might be familiar with the finite difference method. That's probably the easiest to teach and the easiest to get into. But there's many other ways to do this. There's finite volumes and there's also meshless methods that don't base their discretization on meshes. So mesh free methods, radial basis functions. There's iso geometric analysis which is maybe less known, but finite element is just one of the methods. Is finite element like extremely generic and applies for like many different forms of PDEs and these other ones are more specific or like how would someone choose to use DL2, a FEA library versus a different library that's implementing one of these other methods? Sure. So I mean it depends a lot on the equation that you want to solve. So I would say that over the last 20 years or so we've basically figured out that the finite element method is really just a generalization of finite differences. And you can see it also as a generalization of finite volumes if you want to. But nevertheless, so for problems, for example, there are high-speed flow. So everything that is strictly hyperbolic, I think today we understand that the finite volume method is probably the way to go. Whereas for everything that's not hyperbolic, so for static problems or for problems that, where there's transport of information in every direction. So let's say, you know, elastic static problems or elastic dynamic problems. So for these sort of things, it turns out that the finite element method is actually pretty good and can be applied to a lot of different applications because it sort of gives a conceptual framework of how to approach these problems. So let's go in a slightly different direction here. Let's talk about something I think you made an offhand remark about earlier. The software engineering aspect of all of this, the design and the testing and the validation of it. Because when you create these numerical libraries, not only are you doing it more efficiently, but you also do want to make sure you're actually giving correct answers. Can you give us a little insight on how you do, you know, how do you prepare for a release? How do you actually verify that your software is giving correct answers across all these, you know, variety of different types of solvers and methods and things like that? Sure, that's a very important part. There's several ways to approach this. There's several kinds of tests that are happening. There are tests that you can do based on the theory of the finite element method. So you expect certain convergence orders or certain behavior and you can verify that that works. But we also need to do like the standard software testing of the API if the data structures are correct and so on. So we have a test suite with about 8,000 tests that we run continuously. So we do continuous integration. We run nightly tests with many different configurations and different libraries and make sure that stuff that worked at some point still works. Now, do you find that your software performs the same way on different types of architectures, such as, you know, every 18 months or whatever we get a new chip from Intel and AMD has an entirely different line of chips and do you support things other than Linux? Do you support OS 10? Sorry, I asked a bunch of questions in there but just more on this engineering side of things about, you know, the verification of, you know, I get the same result if I run the same program under OS 10 on my Mac versus over MPI and a cluster of, you know, 32 servers. It is difficult. And to add to that, it's not only the different architectures, it's also just different compilers, right? You compile something with the Intel compiler and it will do different optimization. And the problem is that all these computations that we do are floating point computations that might be slightly different. So testing these is a little bit of a challenge but there's ways to do that with tolerances and so on. And yeah, Wolfgang, do you wanna mention our cross-platform support? Yeah, so I mean, you know, it's a challenge to make this happen but by and large, I think we're actually pretty good at this. So we have a lot of people who use Macs. We have a lot of people who use any different number of Linux distributions. We've run on basically every cluster there is. So, you know, you adapt to this and you have your tools and your procedures and ultimately you make it happen. It's just software maintenance. It's not exciting stuff but you figure it out. So what's the largest example of someone using DL2 in practice? I'm not quite sure what you mean by largest, largest SM, most number of users or largest in the sense of the biggest parallel run that has been done. The biggest parallel run that's been done, well, actually probably not the number of processors, probably the number of elements, right? The total size of the problem. Like how far does this thing scale? Okay, so the biggest runs that we know of are probably done by a colleague and good friend of ours, Martin Kronwischler. And he's been running on the Supermoc in Munich in Germany and on something like 150,000 cores and the biggest problem. So when you talk finite elements, the interesting number is the number of unknowns is in the order of 30 billion. And so you're solving, so you're creating and solving a linear system that is 30 billion cross 30 billion. So a common question that we ask a lot of our guests is what's the strangest or most unexpected use of DL2 that you have seen? Yeah, that's fun, right? When you do open source, you never know what people use your stuff for. So there's a couple of papers that I really like. One simulates a deterioration of statues in Rome due to air pollution with DL2. And the other one is a paper that I think appeared in Nature that simulates the growth of plant roots. Also not tempting that I would have expected our software to be useful for them. I'm sorry, just real quick. The growth of what? I missed that. Plant roots. Oh, plant roots. Yeah, so I think it was wheat or Arabidopsis or something of that sort. And what license is DL2 distributed under? LGPL. Any particular reason you guys chose that? That is a more complicated story. So it used to be the QPL license, and I don't know why Wolfgang and his colleagues originally chose that, but we switched to LGPL a while ago because we wanted more people to be able to use it and make it more compatible with other things in the ecosystem. So normally we ask what is written in Bay Area. As you mentioned, it's written in C++. Do you have native bindings for other languages, Fortran, Java, Python? We've got a couple people who are writing interfaces to Python, but that's slow. So the issue I think that we encounter is that a lot of DL2 is written with templates and templates don't really match so well onto language or math well onto languages that don't have templates. And so I think that's one of the issues that we face that it doesn't match very well. So what's coming in the future? What are some new features if we're not gonna expect Java bindings anytime soon? What else can we expect? Yeah, so I mean, it's a community project, right? And so what you can expect is what people implement, right? So everybody implements the sort of things that they need. That's sort of the majority of contributions that we get. And then sort of in the small circle of maintainers, there's a few projects that we want to work on. One of those is that we want to have parallel multigrid solvers. So when you want to solve really large problems, then multigrid is the best methods to solve linear systems, and we need to support that in parallel. And then there's a few other finite element-specific things. So something that's called the HP finite element method that we want to provide in parallel. We provide it in sequential computation so far, but not in parallel. These are sort of the big things that we want to do. So tell us a little bit more about the community around DL2 developers and users. Yeah, that is a topic that is really close to our hearts. And it turns out that the programming aspect, the software design aspect and so on moves further and further into the background. The bigger the project becomes the more users you want to attract. And well, it's just a matter of trying to keep the project going. You need to attract developers that not only use your library but actually contribute back and step up and maintain it and help out. And this has been a long process for us and we've been working quite hard in trying to figure out ways how to motivate people to contribute back. And I think if I may add it, I think we're actually really good at this too. That's the number of people who contribute code today is vastly larger than it was five years ago. And I think we have a really nice supportive environment where people come to this project and participate in a joint group of learners and work together. And I think that works pretty well. So guys, thank you very much for your time. What's the website or email where people can contact you and find out about DL2? It is DL2.org. So D-E-A-L-I-I.org. And we're running our development on GitHub. So that's where most of the things are happening. Okay, thank you very much for your time. Thanks guys. Thank you.