 Welcome to another edition of RCE. Again, this is Brock Palin. You can find us online, subscribe to the podcast RSS feed and all that other fun stuff at RCE-cast.com. Also, if you could leave us a review on iTunes, that definitely helps for getting the word out for the people who may be interested in the kind of material that we offer. Also, iTunes almost shows a certain number of back episodes. The entire back catalog is available at the website. So again, that's RCE-cast.com. Today I'm flying solo actually, so going to roll right into our guest. Today we're going to be talking about Elmer, which is a suite of scientific tools. And so I have with us three developers of Elmer. So guys, why don't you go ahead and state your name and give us a little bit of background. Okay. Hello, my name is Peter Roback. I'm developing Elmer since 2001. I'm a physicist in background and let's say my special interest lies in solution or multi-physical problems. And there I have made most of the contributions. Hello to my name is Thomas Zwinger. My background is physics and mechanical engineering. I work here at CSC as a senior application scientist. And I mainly do applications of Elmer either in microfluidics or in geophysical problems, mainly in ice sheet, glacier flows and micro meteorology. Hello, my name is Mikko Bückling. I'm an application specialist here at CSC. I've been working with Elmer for a couple of years now. My background is from computer science and from mathematics. And so like my specialty is numerics and HPC in general. So I'm like a helper for the physicists here. So real quick before we go into Elmer, where and what exactly is CSC for people who don't know? Well, CSC is sort of the national supercomputing center in Finland and we have lots of other activities as well. But from maybe your audience, this is the main activity and then Elmer software is then a natural companion to this supercomputing activity. Okay, so can you give us a little bit of overview? What exactly would you say Elmer is? Well, I would say that Elmer is a multi-physical finite element software suite. It comprises of several parts. So we have solver, we have the GUI and even a post processor. And it's both for end users, but you can also use it as a library if you are developing your own models. So how exactly, give us a history. How did you guys get started on this path of making this package? Also, there's a number of other solvers out there. Why did Elmer get started? Elmer actually got started in 1995 as part of a Finnish national program in computational fluid dynamics and at that time there wasn't really open source tools available. So it's certainly not a special purpose nationally, but later then we went to open source. So maybe my colleagues can contribute. Well, I step in. So the open source that was done in 2005 where it was released under the new public license too. So officially the release was a new public license too or later. In 2012, if I recall correctly, we made a small adjustment to that in that sense that the Elmer solver library, so that the solver suit itself was published under the LGPL such that commercial codes or applications could plug into it. One major step in between I think we should mention was that in 2008 we added the Elmer GUI. So the graphical user interface which is based on Qt library to the package was a step taken in order to make it more popular, more easier to use for beginners so to increase the learning effect with Elmer. And from my point of view personally, I want to add 2012 where we officially established the Elmer ICE package, which is an add-on package for a huge scientific community of Elmer, the glaseologically-induced inclined community, which deals with the simulation of ice sheets and glaciers, which is very important for climate research nowadays. So you mentioned there's a library in the GUI. Do you have your own language in the GUI or how exactly does one kind of program the GUI? Well, the GUI is quite lean so it's maybe 20,000 lines of C++ and it's programmable in the way that you can define the menus using XML files, but Elmer doesn't really try to be something like free friend++ or any code where you could sort of write down your own equations, so you have to sort of still do a little bit of programming if you want your own equations put in. Okay, and then Elmer itself, what's the primary language that probably the primary library and solvers written in? Well, most of the packages or the internals are written in Portran 90 or later and then with some added C parts due to some legacy code, mainly and mostly because at the time that the software was written you couldn't really load shared objects with Portran, so there are some C layers there for doing these things and this was for the solver part and for the GUI part that's just C++ mostly. Okay, so if I'm developing an application and I want to use a routine from the Elmer library, am I restricted kind in the languages I can call it as or are there bindings for things like Python or other higher level languages or alternative languages to Fortran? Well, currently we don't have an API for calling the routines from anywhere else than Fortran but this could be quite easily actually added. We are actually now doing the ISOC bindings for generally just cleaning up the code and using ISOC bindings there so you could actually quite easily add ISOC bindings for external calls from C or from Python. If I may add to that, so there is an internally developed language if you want to say so, it's called MATC which enables in the solver input files in those files to define the runs to program simple mathematical context there. Not much difficult things. If things get a little bit more difficult, we recommend that you start writing a Fortran routine which very easily can be dynamically linked to the system. Okay, so in terms of the actual programming model, is this code parallel capable? Is it serial only? Is it MPI capable? What's kind of the focus of the type of solvers it has? Well, I could say that mostly the parallelism comes from MPI so it's a distributed memory code in that sense. Lately a lot of threading features have been added with open MP to the code but that's still in development so it's mostly MPI. Are those threaded features meant to do like hybrid mode using threads on multicore processors and also combining it with MPI or is it currently kind of one or the other? Well, our target is hybrid. So something that probably slides in here, is there any work to enable the use of accelerators, you know, GP GPUs, open CL, Xeon Phi, anything like that with Elmer? Currently Elmer cannot really use any GP GPUs or accelerators but as for Xeon Phi's we have actually ongoing work with this and actually CSC just recently, this was made official a week ago, we joined the Intel Parallel Computing Center program. So within this program our target is to optimize Elmer for Xeons and enable usage on Xeon Phi's. So a lot of work will be done in regards to this particular point. Okay, so let's shift gears a little bit and let's talk about the type of science that Elmer actually focuses to solve. You mentioned a couple, can you give us a couple more examples of how Elmer's being used and the types of problems that the library aims to solve? Yes, Elmer is suitable in solving all kind of partial differential equations but of course most of them come from the field of computational engineering. So we have fluid mechanics, we have solid mechanics and computational electromagnetics and heat transfer. So these are the main fields. But there is even some quantum mechanics and more let's say seldomly used fields. So how hard is it to extend Elmer though? Does it have some sort of plug-in architecture or something like that? Say it does a lot of what I want to do but my background is nuclear engineering and if I want to add more quantum type behavior to Elmer, how would you want to go about doing that? Well if your problem can be described by a partial differential equation then it's quite simple. So there are existing examples how to write your own PDE and let's say a generic advection diffusion reaction equation would be maybe 100 lines of code. So you can start from that and write your own equation and hopefully get a tailored solution. If your problem is not a partial differential equation then it could of course be much bigger job but let's say standard problems are quite easy but of course difficult problems tend to be always difficult. So what packages that people might be more familiar with or run across that Elmer would maybe compare to as an alternative to be explored? There's other PDE libraries out there, there's Petsuit, Rolinos and a number of others. There's commercial applications like Comsol and others. How would you say Elmer fits among these in the landscape? Would you say they're complementary, competitors? Solve similar types of problems with a simpler higher level language for certain types of science? What's kind of the focus? Okay so if I compare to other financial libraries like D2 or LeapMesh++ then I would say that those are more oriented for developers. So Elmer is also an end user program and on the other hand Elmer is written in 490 and newer let's say dialects. So they have a little bit different target audience as developers go because people tend to prefer either of the languages. Then if one looks at competition among software then I would say that for instance the instance open form is complementary in the way that Elmer is not that much used in high Reynolds number flows but Elmer then takes let's say the lower regime of the Reynolds number more easily. And if we compare to Comsol, more to this extent of course their Comsol is much nicer to use but we have some advantage for instance in the parallel computing so I don't maybe want to see the commercial software as direct competitors. It's sort of you either choose the open source or the commercial sort of route and then you choose your tools according to that. Okay so Elmer has a high level language and also lets programmers think in a little bit higher level constructs in some of these other libraries but Elmer itself does provide a library. So what are some packages out there that can use Elmer under the covers for doing solving or you know optionally use Elmer? I don't think there are other programs at least not that well known ones that use Elmer. So it's the other way around that we tend to use many of the libraries under linear algebra but Elmer maybe is primarily it's an end user. Let's say software that is easily extendable with new equations. It's maybe not ideally a library that you would plug in under your hoods you know let's say other software so we have a little bit different profile than some libraries. Okay you mentioned that Elmer uses some libraries itself what are some of the things that Elmer leverages to you know provide performance ease of use coder use things like that. Could you explain in more detail I mean what are you particularly interested to? I more want to know what tools or well I don't want to get a tool so much quite yet but I want to know what software you guys leverage so you don't have to write as much. I got the point yeah sorry. Yeah so basically external libraries we plug into a lot of them. So the most prominent and actually essential so we cannot run the code without it. The LAPAC BLAS and UMPAC and we ship them actually with the source code. In order to make sure that if you combine it from scratch that it's included. And if we go to the parallel computing side on an optional basis you can include a hyper mumps mumps is a very versatile parallel direct solver Trilinos SuperLU and Bardiso whereas Bardiso is more for the multi-threading. In special applications like the the ice the classical applications we also have some solvers within the interfaces to net CDF which is the common standard of input output for climate related problems. And if we turn to the post processors there is a legacy post processor which is called Elmer post which is not further developed from our side but that one relies on TCLTK. So quite old stuff and the newer post processor that is included embedded in the Elmer Graphically User Interface Elmer GUI is based on VTK and the QWT. So this is basically the whole range of external libraries we link into. Of course I have to mention also QT or QT itself which the Elmer GUI relies on. If I may add to the post processing so we consider nowadays ParaFU as the good tool for doing the post processing. So basically giving this away out of our hands ParaFU is a very versatile open source tool which reads in the VTK output which Elmer is capable of writing. Oh okay so you've moved to using kind of like the standard VTK file formats and so anything that can handle VTK files ParaFU visit others would be able to be used as a post processor with Elmer? Yes I maybe should emphasize that we used the XML based newer version of the VTK format so and it's used in binary so it's rather compact compared to the let's say previous ASCII version of the VTK format that we had a few years ago. Okay so you have a GUI but you also have an MPI and you work in HPC center. I assume you can run the Elmer solver without the GUI so it supports batch environments just fine? Yes. Okay so what do you see a scientist's normal workflow who's using Elmer would be like from beginning to end what would be the like those of you who are the physicists on the call how do you use this to solve real science? Yeah maybe a tramping so I have a lot of experience a lot of experience within these glaciological applications in the ice sheet places so what we usually do is mashing with open source software like GMSH very good software for getting measures unstructured measures some colleagues from France then used also a program which is called YAMS for the redistributing adaptation of these measures it takes in input from for instance satellite surface velocity data of ice sheets in order to identify the areas where by higher shear rates it would need higher measure resolution and then we do a lot of mashing actually we go from food grains to 3D within Elmer so there is a capability of a parallel mashing of already partitioned measures so we base Elmer is based on the domain of the composition so each separate process is running on a separate domain so if you can avoid partitioning in 3D which we do then you can have a real advantage to avoid that bottleneck in parallel computing and that's what we do basically vertically extrude our measures if it's possible already on the amount of process of course we are using so that's on the pre-processing side and on the post-processing side basically we already explained you can power a few which you can use power a few which also can be used in parallel mode if you want to if you have larger systems to post-process and Elmer itself of course it's completely running in parallel so that's basically workflow so you start building a geometry and also the mashing with open source like GMSH eventually refine it with another program read it into Elmer and post-process with power a few. Does Elmer have any type of capability to do say AMR adaptive mashing such that say you have crack propagation or something like that and you want to refine around that region can it do that and if so how does it handle load balancing? Well unfortunately we can do adaptivity but we cannot handle load balancing so adaptivity was developed in Elmer more let's say about 10 years ago and we sort of got stuck into the load balancing so we didn't have the resources to continue the development of the let's say parallel adaptivity approach so I must say that in that respect Elmer is not let's say among the most spectacular software so our adaptivity is purely serial at the moment. Okay and then in terms of mesh formats he mentioned GMSH which I'm going to have to track down myself but what kind of I assume Elmer as a library has the ability to read different common mesh formats what do you say some of the most common ones are? Well as said GMSH has its own mesh format we support also the universal mesh format for instance which is written by Salome package and then we try to support also mesh format so certain commercial packages but there we often see that the commercial mesh formats they are not easy to find specifications on the format so we have to often guess a little bit how the format is built so for instance we can read in meshes from console multi physics from some older version of Gambit and so on but sometimes meshing getting the meshes out of commercial course may be quite frustrating. So in terms of community involvement though so say I wanted to add a mesh format or something like that do you see a lot of that do you see a lot of community contribution to Elmer or is there kind of like a core group with maybe around the edges? I would say more of the latter so we maybe code ourselves 90% of the features but that was a good example the let's say different mesh formats they have had some contributions so for instance yesterday I got in a patch that implemented some new elements to the console multi physics mesh reader so yes we do get some contributions but of course not as much as we would like to get and we are very happy to get those so we are agile and try to let's say get all the contributions that we get we try to also incorporate them into the official version. So what's the community actually look like is there a mailing list forum how basically how can people get involved with this? Yes there is a discussion forum it has more than 1000 members and there is some active discussions it could be even more active but but you can find pretty much on every topic some discussions if you look at the search features so we also have a wiki where actually if people ask for being switched free can actively contribute also for the classical community especially which I already mentioned is the largest scientific community using Elmer we have a special portal which is called ElmerEyes portal where they can have tips and contacts and also news related to this very specific scientific field and also on that side for that community we are running a mailing list so where we can broadcast new developments meetings etc. I could also add that we have a group in LinkedIn which includes maybe 250 members currently so we try to be active in many different fields so. Okay so channeling Jeff here since he couldn't be here today what you already mentioned that Elmer's license under GPL v2 so what code revision system do you use and why? Well currently what is in use is SVN from Sourceforge that's mainly I think that's mainly for historical reasons Sourceforge was just picked up as a possible place to put the code we are now considering moving to using JIT but that's just thought currently so it's not set in stone if we will make it a switch or not. Okay so how far can Elmer scale like what's the largest problem either in elements or CPUs ran that's been solved with Elmer? Well the finite element assembly it scales always ideally so that is not the bottleneck but the bottleneck is in solving the linear systems that result from the finite element assembly and the type of equation well then dictates quite a lot how well we can scale we have demonstrated scalability for instance up to 6000 cores using FETI which is the finite element tier and interconnects method. Quite suitable for elasticity problems for instance but I would say that we can scale up to thousands in many fields but you have to have a big problem if you want to run in thousands of cores so maybe at least 10,000 or 50,000 degrees of freedom for each core so if you want to run with 5000 cores you better have a problem which has some hundreds of millions unknowns in it. Okay so what future features are coming to Elmer? Yes so currently we are working quite actively in computational electromagnetics so we recently implemented a new type of element an edge element that can also be distorted so it can be well you can provide on but maybe I give this to you. Well I think that what Peter is trying to say that mathematically the mapping doesn't have to be up fine from the reference element to the actual element so that's what you're trying to say. Yes and this opens let's say new fields in computational electromagnetics because this element is quite rare I don't think there are many open source tools that has this feature for example. Also recently we have added features for gluing let's say boundary conditions even if they are non-conforming they may be rotating and also I think one important feature that we are hoping to see in the near future is new features in the multi-threading so maybe Nico can say more. Yeah I mean I think that I mentioned this before but we are really looking into hybridizing the code and exploiting Xeon files for instance so like as either as native code processors or as offload devices or some key parts of the code. And actually related to this this development due to difficulties in cross compiling with the old build system we are making a switch from the old order make based build system to to see make and this will happen in the near future. Yeah from the geophysical application side there the tracing of some quantities within the systems for instance pollution in glaciers or flows general or also the estimation of age of ice layers for instance. So these are purely convective type of equations very nasty to solve actually with the collerquing method or impossible parking. We have a discontinuous collerquing solar which is already implemented but I would say it's still on a beta level. But some person from British Antarctic Survey contributed with a semi Lagrangian solar which I hope sooner or later will make it in the distribution. And that would be very nice tool for those kind of applications. Okay guys well thank you very much for your time what are the coordinates for finding Elmer downloading the code and generally just what's the website. Well you can find it under the CSC home pages so that would be www.csc.fi slash Elmer. Or if you want to come to the what I would call a general home base of Elmer it would be just Elmerfem.org where you find basically all the links to the relevant pages around Elmer. Including the special scientific community applications. Okay so Elmerfem.org. Elmerfem.org affirmative. Okay guys well thank you very much. Thank you. Thank you.