 Welcome to another edition of RCE again your host Brock Palin and you can find me on Twitter at Brock Palin all one word And you can find the show online where you find all the back shows at www.rce-cast.com and Also, there's was an issue with the iTunes feed where a bunch of shows were missing that is now fixed So those of you who are looking in the iTunes catalog and it looked like we hadn't updated a show in a long time They should all be populated now Yeah, we're gonna blame that on the RCE gremlins. They appear every once in a while, but this one this one was definitely their fault Yep, and that is my co-host Jeff Squires from Cisco Systems and also one of the authors of open MPI So Jeff thanks again for taking some time out of your busy day Sure, and I got a throw in the obligatory ping from My blog out there get to it from the RCE webcast It's on blogs.cisco.com and I just write about general MPI HPC networking kinds of things So please feel free to drop by and and send me your thoughts So what are we doing today today? We're talking about XDMF It's kind of like a file format, but it uses Guess we've had on here before for actually storing a lot of the data So I need some clarification of exactly what this is so I'll go ahead and introduce our guests instead of trying instead of Hacking up exactly what it is our guest today is Jerry Clark who is from army research laboratory And they were kind enough to you know, let him speak with us about this So Jerry why don't you take a minute to introduce yourself and give us your background? Okay, sure. Hey Jeff Brock. My name is Jerry Clark. I'm from the Arma Research Laboratory. I work in a computational science and engineering branch And what we do I don't know if you know a lot of that general research laboratory, but it's a cutting edge of a lot of Basic and applied research and you know to help the soldiers is what we're really after And we do a lot of analysis and design which is inherently Multidisciplinary So we do a lot of things like computational structure mechanics fluid mechanics, you know multi-scale physics types of things And what we're trying to do is we're trying to avoid The stove pipes that regularly occur and not only in those disciplines, but in codes in specific In other words a single code may have it may have its own pre-processor its own visualization system And it just got ridiculous after a while. So what we're looking for Was initially just a common data modeling format that we could use for Visualization and pre-processing over Multidisciplines so we didn't find anything that perfectly suited us out there so we decided to develop our own and what we came up with it's called the Expensible data modeling format Xdmf. I guess it could have been a E Dmf, but Edmas didn't sound right. So we went with Xdmf and it's uh, It's it's a lot of different layers on it. I guess it's like, you know shreks It's like an onion a lot of different layers on the top layer It's just a common data modeling format uses HDF 5 and XML and if you use that you can pull it into a lot of Visualization packages and and you're off from there if you want to go further into it You can use it to a couple codes and do a lot of other things the interesting thing about Xdmf is Kitwares involved with this somehow who we've had on this show before what exactly is a relationship between you guys and Kitware Long history with Kitware. We were involved in in pair of you from the very beginning And and that came that developed from our our interest in VTK in general So as as Xdmf was developed Sort of one of the first things we needed was a reader in pair of you So we work with them there and they kind of got a little more interested in Xdmf and helped contribute and helped get the word out There a little bit for the types of things that could be used for and it just sort of snowballed and rolled from there Cool So let me let me go in on part of what you said earlier You said none of the existing systems out there did what you need and so you went and rolled your own So what kind of features were you looking for that that others don't support? So what are the unique attributes of Xdmf? Well, we needed something that was Platform independent and we had used hdf before and a lot of the systems that were in development at that time I'm thinking particularly of DOE's safe and hdfe os they were using hdf as well So hdf was pretty much the the way to go it seemed For a cross-platform data format, but then to describe the data model, you know, what types of things are in there? You know, what is my topology? What are the names of my scalars and things like that? There there were a lot of very complicated systems We wanted something I guess more would be described as a data catalog Then a data modeling format if you're a purist about it, but we needed something easy to do to easy things And we came up and that's why we came up with Xdmf. We use XML because it's easy to parse It's it's easy to Big down into the guts of it and you know see what's going on And we use hdf5 because it gives us cross-platform Platform independence and it gives us good high performance in both scalar and parallel applications All right, great. So that actually leads straight into my next question What kind of platforms do you run on and I was gonna say do you run serial and parallel? But apparently you do could you give any insight on that? Yes, we like I said it would the first part was make it easy to do easy things So we attacked, you know serial machines at the time a lot of you know a lot of SGI a lot of Sun At right now Xdmf pretty much runs when everything runs on Windows runs on max on the on the parallel systems part of the army research laboratory has a distributed shared resource center where we have you know Machines with thousands of course parallel file systems things like that So Xdmf has to run there for the types of codes that we're interested in a shocks visits codes Multiscale codes, so it is particularly designed to work in those situations. I was mentioning layers before An interesting part about that is built into Xdmf is a distributed shared memory system So you can use this in your code to make it look like your code is writing to a Common data model in format as a file system But it what it's actually doing is writing into a distributed shared memory system Not only does this allow you to run time visualization on a code But it also lets you a couple codes from a global from a global perspective as in each parallel code Does not have to be cognizant of the decomposition of the other code, which gives us a lot of interesting capabilities Well, that is kind of cool. So Do you a natural follow on to this then is you know How do you foresee this scaling into the the massively multi-core kinds of environments because one of the Issues that server vendors like myself are looking into is you know, how do we you know accommodate the changing landscape of having a Whole pile of cores inside one box rather than what customers have typically been doing in the past is buying lots of systems To get lots of processors now It's all in one box. Does your system kind of accommodate the the change in communication architecture that's going to be required to do You know shared memory kinds of communication and and Numa kinds of communication rather than network-based kind of communication or Even more complicated, you know the the combination of both the new nuh so to speak non-uniform network architecture in the box and And out of the box. That was a long question That was a long question as luck would have it Part of part of my branch is also looking into At what we call it advanced computing for tactical situations and that involves a lot of looking at FPGAs and cell processors. We're really really concentrated on there. We're getting a lot of speed up is GPU Calculations so we that's been going on for a couple years and we have a good experience with that so What we've used xDMF for is to use that distributed shared memory To couple both the multi-core Calculations and calculations on several different GPUs so we can use this distributed memory concept To allow the GPUs, you know to do their thing to take care of their hierarchical memory But all and then right into the distributed shared memory system along with the multi-core Systems as well, which lets you know, there'd be one spot where you can grab the data Where everybody can get to it and it's all synced up So the DSM is smart enough to know that it you know If if that section of distributed shared memory that is being handled by a core lives on that processor You know obviously doesn't send a message. It's smart enough to do a shared memory, right? But if it has to go off processor to get something that's smart enough to know, you know, where the piece of memory is the the shared memory system itself, you know is byte addressable the neat part about it is that we've developed an HDF 5 driver that sits on top of that shared memory system So your code itself can use the HDF 5 API to write into the distributed shared memory system As opposed to using you know puts and gets Into the system itself. What's nice about that is it lets a lot of codes not have to be rewritten Because they all have some kind of file IO capability and they can use that section of their code To you know get and put information Into distributed memory system as if they were writing out to an HDF 5 file So I have a I have a question buddy xdmf then Well, I guess that's what we're talking about. We're talking about xdmf So xdmf actually has code that comes with this some of the examples I was seeing of how to use it actually involved just writing your own xml file and writing directly to hdf 5 So it was more like a description language that then you could write an Implementation of to be able to understand but there is an actual xdmf library I can build and link into my code and get this distributed shared memory functionality Yes, there is um on the on the top level of all you want to do is use it for visualization al3d from livermore does this they they were writing out their hdf 5 files So they just had print statements that printed out xml There is a c++ class library that comes with xdmf And it's wrapped for You know scripting languages like python tickle tk. It's also wrapped for for java So if you're using any of those languages, you can get direct access to it All access to the distributed shared memory to reading and writing xdmf files is is in there. So At that level You've got a lot of functionality that at your fingertips We're also in the process of writing a new xdmf access c++ class library to Take advantage of you know more modern things like smart pointers and visitor patterns for writing out in parallel Also, what the our new System will also allow you to do is to use xdmf for more of an internal element database for hpc codes Um for you know finding element codes and things like that So when you write data to xdmfd Is it pretty much always like structured grids or something like that? Like you say like this is a grid or is it really freeform where you can describe anything? Um, it's it's not all structured grids. So, um, basically in xdmf You have a computational domain domain is is busted up into one or more grids Each grid has a topology and a geometry Those topologies can be structured. Um, they can be unstructured Unstructured in that, you know, they can be tetrahedrons, hectahedron or they can be mixed Topologies as well. So you have a concept of a grid And you have a concept of attributes on that grid Which are either node centered cell centered edge centered and they can be you know scalars vectors tensors You know all the all the usual suspects So it's it's a way of writing out and reading back in Both structured grids and unstructured grids and mixed apologies And a lot of information It's the concept is that it's part of the system is that it is It separates the heavy data from the light data the heavy data is your Gigabytes pair of bytes of calculated values, you know your positions The light data is both your data about the data. What topology are we talking about? What are the names of the scalars, but it can also be Small values, let's say I had total energy on an entire grid. I could write it out there So that's we're considering light data and that's the part that's an xml So as an mpi guy myself, I have to ask the natural follow-up question What what are you using as the communication infrastructure underneath? Are you using some middleware package like mpi or or something else or did you go native and write this stuff yourself? well Back in the day We started out with sockets And that that's had some problems. So eventually we moved totally to mpi So the distributed shared memory system is is written totally on on mpi And so the answer to your question is yes as for its communications. That's what we're using mpi Good love to hear it What what kind of systems do you support? What kind of networks are you tied to any particular mpi implementation? Are you nice and mpi implementation agnostic or how does that go? We were we were fairly mpi agnostic until we got to the point with the distributed shared memory Where we needed to be Where we need to be able to handle threads as well. So we had to be thread safe. So For the just if you're going to use the distributed shared memory system part of it You have to use a thread safe version of mpi other than that We just used pretty basic calls And we don't take it. We don't try to tie ourselves to a lot of Any advanced features that may or may not be in any implementation So what's some of the most complex xdmf definitions that you've seen like either lines xml or Combinations of different complex formats described xdmf is totally hierarchical In that you could have you know Grids of grids of grids They can be Collections of grids as well. They can be collections temporarily and or spatially. So probably the most complex one we've seen are Calculations that are run on thousands of processors For a a large amount of time one of the big work courses workhorse codes here is something called cth Which is out of sandia national labs? And if we write if we run on the thousands of processors for you know Hundreds or thousands of iterations you're generating a lot of data and each one of each one of the processors could potentially you know Dump a data set for every iteration and you have to collect them temporarily and spatially Within the xml So the nice part about it is you can have these collection files if you want to look at the whole data set and have either A pair of you or end site or visit Read those things in parallel Or if you if you're having a problem with a A certain section of the grid or a certain group of nodes you can look at those Individually so it turns out that The more complex it is you can you can also use the xdmf and the visualization tools For debugging purposes as well So if you want to do parallel io with a xdmf, do you actually have to describe how your data is chunked up? Inside xdmf if you're doing parallel io to a single hdfi file And then conversely does xdmf support the one file per process or one file per group of process Where you can describe the overarching data In a single xdmf file That is actually part of motivation for our new Our new implementation In the in the current implementation where you know the the idea and the focus was to make it easy to do easy things The one file per process Was pretty much to rule and while there was very generic ways to Read and traverse a an xdmf file Writing it out was A free-for-all you could there's many different ways to write the same data out to xdmf So in the new version, we're putting a lot of convenience functions In there that lets you both write, you know one file per process or coalesce them into groups to reduce the number of Files you have out there which can get problematic if you have you know thousands or tens of thousands of files out there And it also supports the parallel Well, hdf5 where you have one hdf5 file and you're and you're relying on the hdf5 infrastructure to To do your parallel IO Originally, I should mention too that originally the xdmf supported on the heavy data side It only supported hdf5 We're trying to abstract out a little bit now to let you have other other different kinds of heavy data One being a flat file, maybe a compressed file Also, what we're supporting now is An sql heavy data type. So your heavy data descriptor can just be an sql statement that Magic happens and goes goes gets your heavy data out of a database Whoa, that's cool so Wow that actually just spun me off in an entirely different question I'm going to ask a different question. I thought I was going to ask then so If you're talking about going into sql these days, are you talking are you even envisioning going into Cloud kinds of things like say using amazon's cloud or other people's clouds to do Computations and or data storage and retrieval or is that kind of a different sort of you're sort of reading my mind You're like psychotic. I mean Again on One of the things another thing our branch is involved in is what's called mobile network modeling And we're trying to mobile model tactical networks out on the battlefield We started using and and this effort started probably about a year or two ago And you know xdms was built for physics based modeling and simulations And it and it didn't have some of the things that we needed for mobile network modeling In particular the idea of like a packet of data Or things you need for this parallel discrete event simulators like an event that happened So we are adding some things on to xdms And we're using xdms as the basis and adding and creating a new data modeling format To handle these types of data as well So part of that will be exactly what you're which you're mentioning Going out into the cloud Part of the motivation for putting the sql heavy data type in there Because a lot of this data resides in in database as it's collected So we needed some way down the in the bowels of xdms to get hold of of those types of sql data So we are looking at for this mobile network modeling Doing you know multi-tiered Applications cloud type applications Where we can both monitor and monitor Simulations emulations and experimentation of mobile network modeling As well as replay events and do statistics and get all the information we need out of there So cloud computing is is big in our future. That's for sure Well, I have to say I'm all for it for for two reasons that Actually they serve a vendor I we all love the cloud computing buzzword in in everybody's definitions But I was also a signal officer in the army for 10 years And so I am well familiar of the complexity of these mobile networks And so the modeling stuff is is is sorely needed. But anyway that aside Could you give a little insight into What is um, so you gave a description earlier of of heavy data versus light data What what is the degree of complexity? In in the difference of writing say a driver for the light data versus a driver for the heavy data Of the uh, we had a driver for the for the for the light data You know typically we're uh, we had almost exclusively used xml in our new Implementation, we're also going to have a light data driver for json So, you know, we're doing we're going more towards the cloud and we find that that's uh, that's going to be very useful for us A lot easier to a lot faster to parse The heavy data drivers, uh, we it's fairly it's fairly abstracted. So you on the on the upper levels So, um, you know writing a different type of heavy data Driver really only involves, you know, over over over writing a few class methods. So it's not It's not too difficult, you know, it's only software. How tough could it be anyway? I just have to say I I appreciate your sense of humor. They're fantastic No, that's what all the hardware guys tell me in cisco. That's right. That's great stuff. How can you guys know use it? Oh, okay So let me go off in a slightly different direction here. So what is ARL's goal in making this Open source, it seems like this is a, you know, a critical piece of software That could be useful, you know, both tactically to, you know, our government's interests But in giving it away, we also give that capability to others. So, you know, what is the uh, what is the rationale for going open source? well The rationale for going open source, um, and um, it's it's really to be a multiplier what we're developing Internally is something called a computational science environment Xdmf is at the center of that So what we're trying to do is provide to, you know, our researchers and our designers a capability That is You know, something they don't have standing on the shoulders of giants of things that have already been developed you know, this computational science environment gives you things like parallel a o and and um Input parsers and the world really does not need another input parser. So What we're trying to do is have our our our critical codes that are developed be developed on top of This computational science environment, which takes care a lot of the plumbing and things for them So by doing things like xdmf What we get is we get a multiplier First of all, we get features that we don't have time to develop or put in that are Added by external External entities for example the compressed tar file The compressed tar file heavy data type was generated externally and just sent to us and we got code for very can't be that What we're since we're really after the computational science environment to provide to our researchers Anything that can use xdmf any extra tool out there that we can pull in and pull into our Analysis and then apply to specific problems. They're army specific and and and meant to you know We're really after his help is is helping the soldier So by putting xdmf out there and letting a lot of tools not not only letting it get debugged, but also a lot of External tools that are out there be pulled in we get a huge multiply factor That we could not we could never have funded all this development internally So I want to back up a little bit Something you said earlier got my head spinning Your ability to use like the the cloud and other things This introduces an interesting idea You could actually as long as you had a public facing service and you had a xdmf driver You could transfer terabytes of data to somebody Through email because you would just send them the xdmf file that describes the data And where to download the data and when you actually went to read it then it would download it Is this anything that like is handy or is thought about before or is this considered impractical because of wan speeds? no, you know Every once in a while Even a blind hog finds an acorn and you know by going with xml and having the data described in xml The the shape of the data the shape and location of the data described in xml You know not only through email, but a lot of low-end tools can go parse the data without ever touching the heavy data So they can figure out you know How much storage do I need how much bandwidth do I need is this even possible? They can go get parts of the data so by Describing the the type and the shape of the data in xml as well as in the heavy data It opened up a lot of things for us I'd like to say we thought of that in the beginning of the development process, but We didn't we just came upon it. So I guess uh things are the way they are because they got that way That's interesting because there's something coming up nih has this open data mandate and NSF is looking at implementing a similar version where researchers need to make certain forms of raw data available Uh to other people and what they could actually do is instead of having like a web portal They could just have a service where you could just give these xdmf files and people just Run those and that automatically opens up the data from whatever remote resource They've made the data available on instead of having to have all these complicated systems and web interfaces and such We mentioned the we mentioned net dmf before as we do it and that's exactly how we're using it in the cloud the A web service will give the xml actually the json representation of the of the light data Down to a uh a javascript widget and you you go through and you pick the events you're interested in And hit go and then it goes back and and gets the heavy data and can show you uh Paths and communications on these mobile network models So you're only getting you're only touching the piece of the heavy data that you that you're actually interested in Not all the gigabytes or terabytes that are there That's neat now all we need to do is figure out how to kind of take the Striped server transfer of grid ftp and apply it to this so that over next generation networks We can actually move you know 40 gigabits per second across you know four servers off of a parallel file system across the WAN to somewhere else Just need to buy bigger routers from sisco Sorry had to do that So you mentioned that xdmf is is a multiplier for arl that there are a lot of things in there that uh You know arl itself did not have the time or resources or funding or what have you to to develop And and I have to say that's that's kind of an echoed statement across a bunch of of projects that we talked to including my own So I wonder if you could talk to us about some of the things that uh, you know have been developed outside of of say the core Group at arl and who else is involved in uh in the xdmf community and and what have they done? um, okay, they some of the things that have been uh Developed outside that we would not probably have thought much about Um, we're we're pretty much uh, we were pretty when we started this project. We're pretty much new to xml So uh being able to put out some kind of xml dtd About xdmf was was about as far as we could go. Um, so people down at the um, federal university re re o decision arrow Actually came up and developed all the way on their own a xdmf schema Um, which uh, which was helpful in some of the things that they were doing To view xdmf files that were were sent to them to make sure that they were valid and and and worked well probably one of the more interesting uh, and and potentially productive uh things that will come out of this is uh, some people's um, uh, john bidis come and uh, drone Sumania at the swish national computer swish swish national super computer center Um easy for me to say, uh took the uh, the dsm part of xdmf and uh, sort of reworked it and souped it up and uh, are using it for Some runtime visualization and we're hopefully uh, we're we're putting it together and it may turn out to be a a standalone little Functionality that we could build on and our new Um Our new xdmf uh implementation can can take care of can take advantage of that So, you know putting things out there in the open source. There is a uh, a huge advantage to it for us as a multiplier So, uh, what's all your usual development environment look like you an emacs vi guy or what do you use? Well, you know, I'm showing my age. I'm a vi guy. I tried to uh, I tried to change I tried to I tried to be modern, but I you know found myself always going back to the back to the colons and And messed up a lot of code trying to trying to go to other things Uh, the new guys that are out there, you know, we we always get it on influx on new people In fact, we just opened up the crates and unpacked some new guys a little while ago You know, they're all into these uh integrate devir development environments and and things like that They're not too impressed that we can get a pretty far with vi and printf So, you know, they're a totally different environment Part of our computational science environment is not just the the software. It's also the the the entire development environment So we provide, you know, both the The repositories the hardware to to run them on Um, plus the the automated testing for things like either C test or Hudson to automate it automatically test our code As we development. So, um, we're trying to bring uh, you know, a modern development environment To uh, to a lot of the projects that are around here so they can take advantage of that through the computational science environment So these crazy young kids forced any of these, uh New source code repositories on you like git or mercurial We we went kicking and screaming, you know, cvs is all we ever needed Um, but uh, somebody's young worker snappers they forced us to go to get Uh, God, I'll admit, you know, every once in a while they get a good idea so Now we're we're totally we're totally using git and uh as a as a spin off of that It turns out that uh, it's actually a pretty good system. We're trying To to use to do our remote distribution to all our desktops So, you know, instead of pushing everything we can use a use a git job to just pull it from a central repository And then only replace the parts that use as opposed to blowing down, you know Tens of tens of gigabytes to every workstation every night. So They come up with a good idea every once in a while So what's the craziest use of xdmf you've ever seen something that you totally did not expect or Just like wow that actually works. I'm impressed I'd have to say it was with some of the with some of the gpu work Where we were using um, you know, some people were putting together Uh, a distributed shared memory server um, running across multiple cores with gpu's doing some kind of So either some kind of radar signature cross-section or doing some kind of like n body calculation And that was updating distributed shared memory and we were so we were doing, you know GPU calculations and runtime visualization on a running code All through xdmf and it was like wow, I you're just like you said I can't cannot believe that actually worked So they were actually passing data between the simulation code and the visualization code by using the in-memory file mapping chain Yes, that's exactly how it was working. Um, it was written in CUDA and the uh, you know the Each of the gpu's would they would they would do that every they would do their calculation Every iteration they'd update positions or whatever information inside the distributed shared memory system So the visualization would was turned out to be really nice You know Is the visualization could run at its own pace and just take a snapshot of whenever it got to it And it didn't hold up any of the gpu processing There really had to be no hand shaking Going on it was just writing out the shared memory and the visualization system could pick it up whatever got around to it So it really helped a couple some of these things which opened up our eyes to You know potentially doing that um in a lot of different Coupling situations for coupling of different kind of codes from different disciplines having them, you know run at their own speed As long as you got the ram Exactly So what kind of you mentioned a little while ago that uh, you're doing a new implementation Is that just a new evolutionary version or maybe even a major dot release or is this actually you you started uh, blue sky and Taking all the things that you've learned and started over and what kind of features are coming in this um, like it's it's a uh, it's it's a new version. Um, it's uh Uh, the the older version was getting a little long in the tooth and uh, some of the you know Some of the more modern things like smart pointers were you know weren't in there So no, it I wouldn't say it's a it's a blue sky. Um, the the main focus of it is to Clean up some of those things. You know use smart pointers to to handle all the ownership issues Make things just a little more abstract and easier to use uh in parallel So, you know implementing visitor patterns and things like that and probably the the biggest part of it Was to be able to use it as a internal element database for some of the hbc codes We're we're developing. Um, you were doing we're getting into a lot of work in multi-scale multi-scale type applications And that we we we'd like a way to have the internal representation Of the data that's internal to the code and the computation Closely reflect what's out there on desk and not have to move things around And reorder reorder a lot of data. So, um, that was the the real motivation to doing that So what's the uh website and where we can get the information xdmf mailing list, etc So you can get everything at www.xdmf.org All the information you need on how to get it and how to get on the mailing list and all Should be right there So that should uh be a pretty good start for it and uh, a lot of a lot of interesting questions on the on the mail list And a lot of the a lot of things get answered and a lot of uh interesting applications are out there as well Okay, well jerry, thank you very much for your time and this show will be out soon. So thank you Okay, thanks. Jeff. Thanks, bro. Appreciate your time. Thanks