 Hey, Jeff. Greetings, Brock. How's it going? OK. So who do we have with us today? Well, today we have, and I'm going to say his name in American because he's a French citizen, and I'm sure that I cannot pronounce it properly at all. And I think he'll be able to correct us once we get him on the line here about how to actually pronounce his name. But his name is none other than Bryce Goglin. And he is the primary developer for the Open MX project. This is actually a fascinating project. Now, I'm very excited to hear all about it. I think there's some really interesting stuff that we can talk about that directly applies to high performance computing. Yeah, this is something that normally goes into the networking layer of an HPC cluster. So this is definitely your cup of tea, much more than mine. Absolutely. This is right in my bailiwick, and I think this is really cool stuff. OK, cool. Let's go ahead and get Bryce on the line. Bryce? Yes, I'm here. Hello. Greetings. Thanks for taking some time out for us today. Thanks for being here. I wonder if we could start off here. How do you pronounce your name? Bryce Goglin. OK, you're going to have to forgive us ugly Americans. We're going to stick with the American version. We're going to call you Bryce. I hope that's OK. Sure, no problem. OK, thanks a lot for putting up with us. So first off, something we really need to get out there. A lot of people are probably asking, what is the relationship between the Miracom MX and Open MX? Because they are related. Yes, they are related. MX works only on Miracom hardware. So Miracom does some specialized nicks, so specialized network interfaces for HPC, which can do a lot of stuff like zero copy and low latency communications. They have their own stack to do that, which is called MX. And Open MX does the same with regular Ethernet hardware, so you can take your laptop with a random Ethernet hardware. It will just work fine and give you the same abilities, with, of course, a bit lower performance, but I'm working on that. So is it just emulating some of the stuff, Miracom stuff doesn't hardware? Yes, exactly. We have a Linux kernel module, which does basically what the Miracom nicks will do in hardware. So everything in user space is the same, but everything that Miracom does in hardware, we try to do it in software inside the Linux kernel. OK, so when you say lower performance, it's really just because you're not using hardware offload, you're just using a commodity Ethernet nick, right? Yes, because these nicks can do some stuff, but there are quite a lot of things that Miracom nicks and also some other hardware can do it as well, like Infinibang. But these nicks can do a lot of stuff that regular nicks cannot do, for instance, zero copy and things like that. It's much harder when you don't have hardware support. OK, so if we had to distill this down to the difference, you're doing in a software kernel module what the Miracom nicks are doing in hardware. Yeah, that's the idea. Yeah. And so when you say it's lower performance, you know, about how much lower performance? Well, they're choosing. If you look at what people will usually look at, which is latency and bandwidth. On the latency, it's very easy, basically. With Mx and Miracom hardware, you can get two microseconds easily these days. With OpenMx, it depends on your hardware. So if you have a very, very, very nice Xeon or Opteon, whatever processor with a high frequency, you can get six or seven microseconds with OpenMx. If you have a slower CPU, it depends. It can be 15. And it also depends on the nick, because if the nick is slow, of course, the packet is not going fast up to the host. So the latency can be worse. But if you have very good hardware, you can get to six or seven microseconds. That's great, because a typical TCP latency could be anywhere from 20 to 100 microseconds, right? Yeah, right. OK, so with OpenMx, you're just talking with a software change, with the hardware that you already have, you can get as low as seconds, but probably with people with commodity nicks, probably somewhere in the 10 to 20 or 10 to 15 microseconds. Yeah, cool. I noticed that you mentioned for getting low latency, you mentioned CPU performance more than actual nick performance. Does any of these specialized ethernet nicks, how much do they affect performance over CPU? Well, it's really a matter of the host performance. So CPU and maybe the memory bus and IO bus, this is the main reason for getting good latency. But if you have, I don't know, fast ethernet nick for something like that, the latency could be worse, because just the nick can be slow at processing packets. So you can maybe lose a couple of microseconds. But most of the microseconds are related to the frequency of the host, because that's where the actual processing of the packet on the receive side especially is done. So it's really a matter of the host in the end. OK, so if you have a typical HPC class of server that's got nice fast buses and nice fast processors, you're probably about halfway there, right? You just need to have a reasonable enough ethernet nick. Yes, exactly. OK, good. So you could get a reasonable enough ethernet nick that's potentially much cheaper than some of these proprietary offload nicks as well. Yeah, basically you can just buy a random server from whichever vendor and you get some, well, not that good, but not that bad, gigabit ethernet nick. And you can get 10 or 12 microseconds easily from with this hardware. And this nick is included on the motherboard. So you have nothing to buy apart from the server in the beginning. Very cool. Very cool. So actually, I heard you say gigabit in there. You didn't even say 10 gigabit. So this works off any ethernet, right? Not just 10 gigabit nicks. You just need Linux to support it right now. OpenMix only works on Linux. And we need the driver to be available on Linux. And that's it. So what exactly is needed to make OpenMx work? Is it something that lives in user space or is it something we have to? Is it a kernel module? Yes, you have to load a kernel module because basically you need to tell the kernel that when it receives the packet with this type of ethernet either, it needs to pass the packet to the OpenMx stack. And only the kernel can do that. So we just need to load this kernel module which processes the packet on the receive side. And then once you get everything decoded in the kernel, you can pass it to the user space library which does basically just give it to the MPI layer and we're done. So then can a nick that you've put the OpenMx driver into, we still run regular TCP traffic for like NFS and management on that same interface? Or do we have to have a separate wire for management and for message passing? Yeah, because in Linux and all drivers that we have in Linux, you can receive multiple types of packets at the same time. You can have IP and IPv6, for instance. But you can have IP and OpenMx at the same time. There's no problem. You can receive a lot of different packet type at the same time. That's actually a big difference between OpenMx and, for instance, Gamma which is another old project working on doing HPC on a regular ethernet nick. With Gamma, you have to modify your driver to do probably better performance at OpenMx, but you break IP. So basically you need to use one nick for HPC and one nick for everything else, which can be a problem for people. And you need to patch your kernel and other things to do that. In OpenMx, we don't need to do that. We just load the kernel module and we have another feature. But everything else still works. Yeah, very nice, very nice. So actually, that's something that my company is quite big on, this whole converged networking kind of thing. So you can have different types of message passing on the same wire. And that's very important to a lot of people. Just one wire and I can get my high-performance MPI out there and my commodity TCP traffic or NFS traffic or whatever it is. Let me ask you this, why did you create this project? Why, how does Miracom feel about this? So actually, Miracom was involved in the process creation because they wanted, basically, to have MX work on other hardware because it works very well on the hardware, but they want it to be available to more people because they think their design is very good. And I quite agree with that because their API is very similar to MPI, for instance. And people want to do MPI anyway. So they want to have MX available everywhere. And OpenMx is the way to do that. You can have the same API. Actually, you can take MPCH or OpenMPI. And even if you build them on top of MX, you can just link them on top of OpenMx. It will work fine. So it's a way for them to have a lot of application available in the world. And they work on MX, and now they can work on all hardware in the world, basically. Well, I certainly like that, being an OpenMPI guy, that we don't have to do anything. And we just magically work with both MX and OpenMx. Follow-up question on this. So I still didn't quite get the zen of why Miracom wants this. Doesn't this cut into their sales? Or if I read a little bit deeper into your answer there, is OpenMx and MX wire protocol compatible, such that if there's an open source version of it, it actually does benefit Miracom. Because then there's more software out there that can talk to their hardware. Yes, you can have OpenMx on one side and MX on the other side if you have Miracom hardware. It works. But I feel it's not very important in the end, because most people will have just one type of communication and hardware on their fabric in HPC. But if you want to have some specialized hardware on one side and regular hardware on the other side, it's fine. Actually, it's what's happening in BlueGinP in Argonne. In all BlueGinP from IBM, you have regular, BotCom 10G nicks on one side. And on the other side, you have Miracom nicks running MX. And if you want to make this guy talk together, you need to have something with the same wire protocol. So what they do right now is running IP on the Ethernet side and do IP over MX on the other side. And it works. But if you do IP for this and you just run PVFS on top of IP, it's a bit of a waste, because PVFS can do better things directly on top of MX. So what they want to do in BlueGinP in Argonne is just remove IP and have PVFS on top of an MX, talk to PVFS on top of MX. But this is very interesting. But most people really don't care about that, because they will have the same hardware on both sides anyway. But yeah, you can have the same wire protocol in OpenMX and in MX, as well as the same interface as the application layer. OK, well, that's pretty cool. Well, the reason I'm kind of harping on this whole, what does Miracom think about this is because I've actually had some of our customers say to us, well, you know, OpenMX sounds great. But how long is it going to be before Miracom sues them and the project goes away? And it sounds like, you know, since you said they were involved in creating OpenMX and they've kind of given it their blessing that, you know, they're actually happy about it. And there's not going to be an intellectual property issue. No, there's no problem. We have an official contract with them. They're actually giving us some hardware to help us do the development. So there's no problem with that. They totally agree with our work. All right, so that's actually an important message, particularly to the enterprise kind of customers, that this is a blessing, a good project, and it's not something that running them is going to open them up to litigation at all. Yeah. I have a quick question then. So you said it's just a kernel module that can be installed so I can use stock kernels from Red Hat and that it basically works with anything that supports regular MX. Is there any other gotchas? Like, will it work with bonded interfaces for, say, supporting something like Luster? Like, I noticed you mentioned PVFS. But can you actually put OpenMX on top of a bonded interface? Well, in Linux, if you have a bonded interface, you see it like a virtual interface, but you can manipulate it like any other interface. So you can run OpenMX on a bonded interface. It doesn't change anything. I think I've only tried once. The performance wasn't very good, but I think the Linux channel bonding performance isn't very good in the end. So I haven't really looked at that yet. It's not clear yet if I should do the bonding in OpenMX or let Linux do it. That's something I will look at maybe one day, but you can do it with a Linux channel. But with my particular bias, we have OpenMP. I do the application level bonding for you anyway. Right. This is actually a kind of a common question. Should we have the kernel do it or should we have some upper level protocol thing? And my particular bias is clear, but I think there's probably interesting things to look at in the kernel level as too, but it's not my particular area of expertise. I think Bryce would be able to comment much better on whether that's worthwhile or not. Well, I heard on Linux channel bonding, it was really bad. And I think OpenMP does the multi-interface stuff pretty well. So maybe it's just enough for us, and I don't have anything to do. I don't know. For example, one of the machines I work with is split across two different geographic locations. And normally we use a scheduler to keep jobs for running across that. But can MX actually work over a routed network or does it have to be like switch network? Like what's the limitation? How far out can you actually have open MX? So if you have some, if you need to route your packets between some fabrics, you have IP basically. And we don't use IP because we are another stack which is basically parallel from IP. You have TCP and IP on one side and you have MPI and MX or open MX on the other side. It's totally different and we cannot do any routing in MX. It has been designed for small clusters which are basically all machines in the same room. So we don't need any router or anything like that. So you know it doesn't work. You're mirroring the architectural decisions of MX itself, right? Yes, yes. And that is, it's just frames over Ethernet, just Ethernet data. Nothing higher level, not even UDP, IP, anything like that. No. No. Okay, so actually I wanted to ask this earlier and I kind of missed that question that what makes open MX so fast? You know, why is it faster than TCP? And what I'm gleaning from your answer is that you're just chopping out all kinds of CPU processing on the server that you've avoided the whole TCP stack, the IP stack, the UDP stack. You're just sending and receiving frames over Ethernet. Is that accurate? Yeah, right. We don't want to have all the TCP stuff for congestion and long distance and I don't know congestion window and stuff like that which is very expensive and we don't care about that because we're supposed to have a short distance network, pretty much no congestion on the fabric. So we don't care about all that. We don't want to lose time and CPU cycles to handle this. So we just bypass all this stack and write our own stack and do just what we need and make it fast, that's it. Yeah, that actually, I have to admit this is the only piece of software where I've done the podcast before playing with the software and since you can run the TCP IP for like your storage and stuff on the same interface, I'm really gonna have to try this out. This sounds like a real like, I don't know, it almost sounds like magic that you can take our existing hardware and in software pretty much cut our latency down by 3X, 4X, I really need to try this out. Of course, that also means that if I have to go rebuild all my MPI libraries now. But that's okay. It depends if you build your MPI libraries for MX, it's fine. Well, all we have right now is TCP and IB. But most of our cluster is just Ethernet. Like I noticed you mentioned Gamma either earlier and you have to patch your network and back when I first look at that, I only worked with like Intel Nix. We buy hardware at random times and some of them show up with Broadcom and some of them show up with Intel Nix and some of them have NVIDIA Nix. This will work with all of those because it works with just Ethernet, right? Right, yeah. So basically you're just talking to the Linux Ethernet driver. So me being a user level software guy, the way I envision your stack is that you've got some user level library that just talks to kernel stubs, that talks to your, you know, the MX kernel driver which then just talks to the Linux Ethernet driver, right? And actually, I don't talk directly to the driver. I just talk to the Linux virtual interface for all drivers. So that's why I don't need to modify any driver. I just use a regular interface that IP and IPv6 and other protocols use as well. Yeah, so this is really compelling actually. I mean, you take your existing hardware, it just goes faster, it works with any Ethernet Nix and you can combine everything on a single wire. That's very compelling. Yeah. All right, let me ask another question here. So OpenMX has been around for a while. What's your current version? So it's 105 and we're working on doing 1.1 in the next months probably. What can we expect to see in 1.1? So 1.0 was basically the first table without all performance improvements that we could do in the first table release. So 1.0 was fine, but for instance, we didn't have a lot of very high performance support for 9K MTU or 1500 MTUs, the things like that. So in 1.1 we're going to change all this so that we can use the wall wire and use larger packets. If you have 9K MTU, we don't want to, basically I need to explain what MX does because MX doesn't have really a notion of MTU or things like that, like regular Ethernet fabrics. And on Ethernet, you need to respect this same maximal transmitting unit, MTU. And since we are wire compatible with MX, we want to send packets like MX does, which is just send 4KB packets and that's it. And if you have a 9K MTU, it's bad because you're not going to use the wall wire capacity basically. So what we have in OpenMX is by default, you are wire compatible with MX, but if you don't care about talking to MX directly, you can just disable wire compatibility and use larger packets. And in OpenMX 1.1, we're going to do that for real. In 1.0, we didn't really use everything we could. So the performance is going to be much better for 9K MTU and 1500 MTU in 1.1. And we have a couple additional features, but it's fairly technical data, so I'm not going to list all of them. That's it. There's always more work to do, isn't there? Yeah. So Bryce, who's using OpenMX? I know this is kind of a hard question to answer for an open source project. I get asked this about my projects all the time, and all I know is that people have downloaded it. But do you know of any real-world installations of OpenMX out there? No, I'm not aware of any, well, I'm aware of some people testing it, but there's no prediction machine using it from what I know. And I hope I will be aware if it was the case. But yeah, it's not easy to know what people are doing with it because you only get reports when they have bugs. And well, maybe there's no bug, but I guess people could sometime just tell me, yeah, it works fine, thanks. And I don't know, maybe ask for improvements, but I don't think anybody's using it for serious stuff yet because it's still young. I think the first table release came out maybe six months ago, so it's not clear that people are already ready to use it for production. So right now, I think some people are using it on small networks and just doing some benchmark and thinking of just stopping using TCP and switching to OpenMX. And we have some interesting plans, like there's a university in Northern America, I'm not sure I can give the name and location yet. They plan to use OpenMX on a 2000 node cluster. It will be basically some regular servers with one gigabit Ethernet hardware and with a 10 gigabit Ethernet core in the fabric. And they want to do OpenMX on this 2000 node. It's supposed to happen within almost on two, I think. And we have BlueGNP, but BlueGNP is very hard to use because the hardware is very specific and the performance is bad so far. So I have a lot of work to do there. If I want to actually get BlueGNP to switch from IP over MX and IP to OpenMX and MX. This is something interesting, but a lot of work because BlueGNP is really different from a regular server that we have in our clusters. What's the biggest cluster that you've ever tried to scale out to with OpenMX? Oh, it's fairly small actually. It's probably 12 nodes and with four, eight jobs per node. So it's very small. We don't have a lot of large cluster in France and actually it takes a lot of time to do this kind of experiments. I'm actually talking with my lab to get an engineer working on this kind of stuff because I just can do the development and do this kind of testing on a per day basis. There's nothing that's keeping people from using this in production. Like would you say it's pretty good to go, pretty well baked? Well, it seems to work fine, but when you have to load a custom kernel module, you need to be rude first and still it's hard to trust that my code is going to work because if it's failed in the kernel, you're going to crash the machine. Maybe I don't know, disturb other applications and maybe corrupt your hard drive. It could happen, we never know. So I can understand that people are not very comfortable with loading this custom kernel module. So that's probably a reason why it takes time for people to... Well, to be fair though, I mean, every proprietary high speed, large bandwidth, low latency network requires some kind of kernel module thing. So it's always a little bit of a leap of faith. And to be honest, for those of us who are not application developers, even just the MPI is a big leap of faith itself because nobody wants to dive into that stuff. So my personal bias is that if it's below your application, it's a leap of faith, regardless of whether it's a kernel module or a library. So I think what we're saying here is, let's have people try. You want people to try this, right, Bryce? Yeah, right. Yeah, so let's try it at scale and shake out the bugs and get this into a mature platform that's good for us all to use. Yeah. I'm sorry, there's no bugs, right? There's no bugs. It's perfect. I got the kernel panic yesterday, but the problem is... Oh, don't admit that. Don't admit that. The problem with kernel module is that you have a lot of highly patched kernel out there, like Red Hat or other distributions are going to patch a lot. And sometimes you find the special kernel with some slight modification and you try to support it. So I just need to wait for people to try a new kernel and let me know where something goes bad, but apart from this kind of stuff which should be fairly easy to fix, you just need to get the kernel sources and see what's going on in there. Everything seems to work fine these days. So I'm quite comfortable with running it from my own application at least. So quick question. How did this project get started? You said you have Miracom's blessing and they've supported you with some hardware. Did you work with them at all before this where this idea came up? Or did you start working on this and then they got on board? How did you get involved? So it's starting a long time ago actually. In 2001, I did an internship at Mericom in California because we had some people in my school laboratory working with them. And so I went there, it was fun. So during my PhD a couple of years later, I wanted to work on similar things. So I just set up a contract with them to develop in MX because then he did basically a programming interface in the kernel for Luster or maybe to the NFS-RDMS and like that. They didn't have a kernel interface. So I work on that during my PhD. It was fun. So I went there for postdoctoral position during a year because in France we have to go abroad for a year between the PhD and getting a nice position, research position later. So I went there for a year and came back to France and we talked with them to set up this collaboration and that's it. So I know a lot about MX internal. So it's much easier for me to take some stuff from MX and implement them in open MX. Maybe it's different because the hardware cannot do the same kind of thing, but at least I know a lot of the semantics and internal stuff, so it's much easier for me to do that than probably anybody else. Still saying clean from an intellectual property perspective. Yeah, there's no problem. We have discussed with them about that and they're okay with that. Okay, good. So a related question, what's the license of OpenMX? So we have a kernel module and like many kernel modules we are GPL, it could be something else like dual GPL, BSD or things like that, but for kernel module it's basically the same. It's open source and compatible with the kernel module or it's not, OpenMX is compatible and GPL and that's what it exactly is. In user space we have our library which is LGPL. So some people are afraid of GPL, so don't be afraid, please. LGPL is not like BSD because you have to provide the source of your code if you distribute it, but you can link your proprietary application with LGPL. There's no problem, that's why LGPL exists. It's not like GPL which has some viral contamination problem. LGPL doesn't have this kind of problem. You can link with OpenMX and you're fine. So it's open source and compatible with proprietary application. Okay, who else is involved in OpenMX? Is it mainly you or are you looking to make a community out of this or what's your plans? So I've been alone for a couple months. Now there's an engineer working with me in France and I get some support from Miricom and external people from time to time when for instance they find that it doesn't compile on their kernel or stuff like that. I'd like to get some help from other people maybe thanks to your interview which will be, I will get some proposal from other people. It'd be great because this is very interesting but it's kind of hard to work on this kind of software. So if some good people are interested in working on that I would be very happy to get some help. But so far it's a little more complicated than say a Google Summer of Code project but you're looking for someone who's into kernel, internals and networking protocols and things like that? What kind of qualifications would somebody need to be able to work on this project? If you know how to program the Linux kernel it's very good but you don't need that actually because a large part of the code is actually in user space where you deal with where you have request queues and you want to post request if a packet gets lost you want to requeue it. So you have a lot of management of requests like that which is very similar to things that could happen in an MPI implementation. So if people are familiar with implementing MPIs they probably can work in the OpenMix library as well. There's nothing specific about network drivers or protocol. It can be good if you have an idea of how it works in the low level driver because you want to use them in the right way to make the thing go fast but you don't need to know a lot and since most of the code is already there you don't need to write everything from scratch so you can learn by just looking at the code I guess. It's not very hard to understand I think. Okay well that's cool actually that's a little surprising to me I wouldn't have guessed that so that's really good information. So let me, I got one other random question that I forgot to ask you earlier here. So we talked about how OpenMix is just using the ethernet drivers and well actually the general virtual device driver down in the kernel there. Could you potentially get any benefit if you did have a hardware offload capable kind of card like if it's a you know a tow engine or something like that a TCP offload engine or other kind of fancy hardware? That's actually something I'm looking at these days because that's a very important question and people want to put some stuff in the NIC but I'm not sure they are doing what I need actually. For instance in HPC NICs you can do zero copy because the NIC is able to decide where to put the data in memory directly. Regular NICs cannot do that so you have to copy the data. So some people will say you need this RDMA basically capability in the NIC if you want to do high performance communication. In OpenMix I didn't do that. What I did is using something that Intel added to their servers maybe three years ago now. If you buy an Intel server these days you get something called AOAT which is input output acceleration technology and thanks to this hardware feature you can offload some memory copy which you will usually do with the CPU and thanks to that you don't do zero copy but you have zero copies that are processed by the OS basically. So it's almost the same but you don't need anything in the NIC and since this AOAT stuff is available in all Intel machines nowadays and I think I would say to people don't bother implementing an RDMA in your NIC for instance just use a regular server and you will have AOAT and the performance is almost the same. There are a lot of other features that could be interesting. ToE I'm not quite a supporter of ToE because first they are not supported by Linux because there are some problems like if you want to filter packets with a firewall in your host but everything the TCP stack is in the NIC it doesn't work. I think there are some problems with forwarding as well so TCP offloading is not something I really like. I'm more a guy who will do stateless offload which means you put some features in the NIC but you don't want to have a lot of sense TSO which is transmit segmentation offload for TCP or Alaro which is a large receive offload for TCP are some very simple features that you can put in the NIC and it helps performance a lot but you don't have a lot of changes to put in the NIC and in the Linux kernel. I think I like this approach which is put some very basic stuff in the NIC and try to help open a mix with this kind of feature which I did adding a lot of features which are very expensive like RDMA or ToE. So in open a mix. That's fascinating actually because when I said ToE as soon as it left my mouth I'm like oh why am I saying that we're talking lower than TCP here anyway. So yeah what you said actually makes perfect sense and you actually perked up on my own little internal bias radar there for open MPI. I think I need to talk to you about some stuff after the interview. So in open a mix we cannot use TSO or large receive offload or things like that but I'm actually working on looking at what I can put in some NIC firmware. So basically I'm using Miracom NICs but you can put them in a Ethernet mod which looks a bit confusing for people but let's say it's a regular Ethernet NIC I can modify the firmware and add some very simple feature in there to see if I can help open a mix and actually if you add maybe 10 lines to do multi-cast, not multi-cast to do packet filtering for instance say if your packet arrives and it's going to your first MPI application send the corresponding interrupt to the first core because you know the application is running on this core and you want to avoid having part of the processing being on one core and then the application run on the other core because of cache problems. We stand lines in the firmware you can do that and you reduce cache problems by maybe a factor of 10 and performance is much better. So I want to look at what kind of small feature like this packet filtering we can put in the NICs and help open a mix and I have some idea about interrupt coalescing and things like that that could be improved thanks to very small features that are easy to add to NICs. So you mentioned IOAT and that was an Intel thing. Are you actually working with Intel directly or are you working with any of the other NIC vendors to make OpenMX support it better? No, so IOAT is available in the Linux kernel for maybe two or three years now. TCP uses it, so I just took the TCP code and adapted it for OpenMX and that's it, there's no need for Intel to help me. We know how to use it, it's fine. I'm not in contact with any vendor yet because I haven't published any numbers about what you can get if you add some support for OpenMX in the NIC. So once it's done, actually it's a research paper that I submitted but it's not accepted yet. When it's done, maybe I could ask some people to actually put this kind of feature in their NIC. It's already in MiriCom NIC because I can talk to them and of course they agree to insert this kind of stuff but other NIC like the one you get if you buy a random server out there, you won't get any OpenMX support but once we have some numbers to prove people that it's interesting, maybe I could get that. Okay, so along those lines, you talked about some really interesting things upcoming, what do you see as the future of the OpenMX project? I mean, is there a set of features and requirements that you're working towards and then once you have that, you're done and the software will be complete and it won't really need any more development or do you foresee a never ending source of things to investigate and improve on or where do you see this going? So for sure my to-do list is very long these days. The main reason is that when I started implementing this, what I did, as I said in the beginning of the interview is that I took what's in MX NICs and put it in a driver and we're done. Now that I've been doing that for a year, I know that some things are not really where they should be because there are some things that you can do in the NIC but if you can't, maybe you don't want to put them in the Linux kernel, maybe you want to keep them in user space. So after 1.1, maybe in a month or two, I think I'm going to rework part of the stack to move something maybe from user space to the kernel or from the kernel to user space so that you don't have a lot of synchronization between the kernel and user space, for instance, which can be expensive. Also, maybe I can reduce some memory copies by moving the way I'm doing the matching in the library, things like that. So I'm now using something back from my own work and rethinking the way we should have implemented that in the first place. I couldn't have guessed all these results first so it's good that I tried one way and I failed and now I'm going to improve things. So I think OpenMix 1.2 will probably change a lot of things in the way we are emulating the NIC and where we are actually emulating it right now. And later, I'd like to put OpenMix as a Linux kernel but there's actually some work at MiriCom for doing a new MX release, a new major release. Right now it's MX 1.2 and they're going to do MX 1.3 maybe in a year or so. It's going to change a lot of things and I will have to make OpenMix compatible with that so maybe I'll do both big changes at the same time. So support the new MX 1.3 and rework everything I said at the same time because both of them will actually need a lot of work so maybe it will be at the same time. And once this is done, I think I'll try to put this stuff in the Linux kernel so that people don't have to compile their own custom module anymore. That's great, it's one less reason to not use OpenMPI. You know, if it's just there, then there's a good reason just, oh, we'll compile and use it. Great. All right, let me ask you some pedantic questions. I always love to ask other developers these questions. What editor do you use? What's your development environment? It's not the I, it's sometimes Emacs and sometimes some very quick, actually I like Nano which is a very small editor that some people use and when I have two lines to change, I just use Nano because I don't want to launch Emacs and wait a couple of seconds. So it's Emacs for long-standing changes and modifying a lot of stuff. But for quick changes and just modify two lines and recompile, I just use Nano most of the time. But I'm an Emacs guy, apart from that. So you don't use too much of a development environment. You're an email, you're a text editor and a terminal kind of guy. Yes. Yeah, same as me, that's good. From my own personal opinion. All right, other questions I'd like to ask everybody. What version control system do you use for your software? So I like SVN. Well, I use CVS for a long time but I was really bored about not being able to rename files without losing this story or stuff like that. So I don't use CVS anymore. So nowadays I use SVN, but mostly because people know SVN and don't want to use a distributed version controller. But for my own work, I use Git most of the time because I've been working with Linux kernel and other projects that use Git. So what basically for my OpenMix work, the repository is SVN. And I use Git SVN on top of it because I like Git and that's it. Git, another Git user, yes. I know OpenMPI is going to switch to Mercurial one day but it's probably similar to Git anyway. As I understand, they're pretty feature complete. Yeah. Or feature compatible, sorry is what I meant to say there. Yeah, no, I never used SVN or anything beforehand. I'm gonna try out this Git thing and I've been using Git. I've never used CVS or SVN and now I use Git. So I'm probably wanting to strange your people out there. So Bryce, I'm gonna kind of wrap it up here. The website for OpenMX. So it's open-mx.org. And that dash is important, right? That dash is important in the name, right? Yes. Okay, and so you can just download everything there and there's documentation and a mailing list? Yes, exactly. You have a long FAQ which covers all problems I've made so far but maybe some people will add something. And yeah, you can download everything you want there and there's some documentation inside the tab or two if you need but the FAQ should be fine. Okay, well, thanks a lot for your time. Thank you for your time. Yeah, I'm definitely gonna go try this out myself. Cool, well, again, Bryce, we appreciate your time. Thanks for telling us all about, well, what I personally think is a fascinating project. I hope this gets very popular and like I said earlier, everybody tries it, we shake out the bugs and it becomes great stuff that the world just automatically uses. I hope so, thanks a lot. All right, thanks. Bye. Thanks a lot for getting hold of us. Thanks. Don't. Whoops, I might've cut him off just a little early there. I thought we were all done. Okay. That was cool. Yeah, no, that was, that's neat. Like I said, it's the only software that we've talked to so far that I haven't actually tried ahead of time but I'm definitely gonna go try it afterwards because like I said, we're mostly running Gigabit Ethernet and with our same hardware if we could get a couple thousand core up running on OpenMX and our MPI library of course is OpenMPI and we can just recompile that and everything should be magic latency, wonderful. Yeah, so I've talked with, well, I've never actually physically talked with them but I've emailed with Bryce quite a bit and a lot of the questions I was asking him I already knew the answer and whatnot but we've actually had to do a little work to make it so that OpenMPI would do both MX and OpenMX so he had to do some work and we had to do some work and but you know, after that, it just works. I mean, you just compile a link against it and OpenMPI doesn't care. So it's pretty cool. I did some internal benchmarking myself with some really good 10 gig NICs. Let's see, what was the latency? I think I was getting around six or seven micros. Regular Ethernet hardware. Just a regular, well, they were really good 10 gig NICs and this was like, I don't know, at least six to nine months ago. So back then they were pretty expensive NICs and they probably still are now but if you could take a regular old Ethernet NIC or perhaps even something that's on your motherboard and get in the range of 12 to 15 micros, man, that's just compelling. That's a lot of dollars saved right there. Yeah, I know when I've profiled our system, we've got quite a collection of users supporting the entire university and I think most of our users would benefit more from lower latency than a higher bandwidth so those latency numbers are just really compelling in my world. Yeah.