 Welcome to another edition of RCE. I am Brock Palin. You can follow me on Twitter at BROCKPALEN and we accept questions and I post shows when they're up there, too. I also have again Jeff Squires from Cisco Systems and the open MPI project. Jeff, thanks for helping me out again. Yeah, you also post on your Twitter whenever you're mad at your sprint account as well to Yeah, yeah Hello everybody, Jeff Squires from Cisco Systems, and you can have a look at my blog. It's out on blogs.cisco.com It's the performance blog. I talk about MPI and general HPC things out there always willing to hear your comments and questions on there and We have a link to that off of the RCE web page at RCE-cast.com And there you can see a form of all the shows we've done, get old shows, subscribe to the iTunes feed or the RSS feed or download an MP3 directly for putting on your phone or other mobile device. You can also nominate other show topics on there. We're always looking for new things that you know, neither one of us are aware of and we'd like to hear from what you guys would like to hear about. Yeah, we always love to hear from the fans because you know every once in a while someone will come up to me or I'll get a random email and say hey love the show and so you know we love to hear your feedback what we're doing right what we're doing wrong and if there's anybody in particular you'd like to hear from so you know give us a shout out on the blog, on Twitter, email. You ran into a fan at the Open Fabrics conference, right? That's right, last week. It's actually a guy I've known for a while but I had no idea he was listening to the podcast a colleague of mine down at Oak Ridge and so yeah, it's kind of cool. No, that's always need to hear about that so and also good to know that you're still keeping up and feeding everything from Open Fabrics and everything else. Yes, it's the day job keeps me busy. Yes, okay well let's go ahead and get to the topic. Today we have the Condor Project at the University of Wisconsin but we have an interesting mix of people here. We have Greg Thane who is at the University of Wisconsin as well as Jason Stowe who's from Cycle Computing. Curious to see how they're affiliated with the Condor Project. Guys, why don't you take a moment to go ahead and state your name and introduce yourself. Good morning, my name is Greg Thane. I'm a software engineer at the Condor Project at the University of Wisconsin in Madison. I've been with the team for about five years. I work on what's called the Condor Flightworthy Group, the sort of core Condor Group. I've worked on a number of different aspects of Condor including the parallel universe and a lot of performance tuning and other parallel applications on top of Condor. I got a bachelor's degree from the UW longer ago than I care to admit but in the wild world of startups but then came back to Madison to do neat distributing computing stuff. Hey Brock and Jeff. I'm Jason Stowe. I'm founder and CEO of Cycle Computing. We got started also about five years ago with Condor. I actually worked in a Disney movie that used Condor to render all the frames of a feature animation and from there decided to start Cycle to help companies be able to take advantage of the Condor's full feature set. I actually knew Greg since about the time he got started and we're one of the two official kind of corporate partners of the Condor team helping out with new features and helping users kind of start using Condor internally in their own environment. Okay so can we get a description of what Condor is? You know it's resource manager, scheduler, kind of describe. There's kind of a lot of things which makes it sort of confusing to explain and a brief sound bite. I think sort of the one thing that holds it all together is Condor is a set of software to implement the idea that we call a high throughput computing, HTC as opposed to what a lot of people think about in the world, HPC. So we really think about the difference between a high throughput versus high performance. So Condor is a bunch of programs, a software stack, to implement high throughput computing. So the core of this what everyone knows is the Condor job manager which is a batch scheduler like many other batch schedulers. But there's several layers of software on top of that that leverage the scheduler and interact in different ways. Fundamentally if you look at it from a user perspective, Condor helps describe on a policy basis what should be running across this high throughput environment. So it's I like the analogy Greg's used before around a sprinter versus a marathon runner. Condor definitely favors the marathon runner viewpoint where you want to maximize the utilization of resources over a long period of time. And essentially Condor provides resource allocation where it figures out exactly what should be running where in that high throughput environment. I've also heard Condor described as a cycle scavenger. Something basically you put in labs and kind of recover wasted resources in places besides traditional clusters. Is this a good description of it or is this just like one of many things Condor does? That's kind of the way Condor got started. And a lot of people sort of still know us mainly as a cycle scavenger. It fits in with the bird motif. In the early days, Condor only ran on desktop workstations. And we would scavenger cycles when the machine was idle. And it's kind of interesting we started out. I think our first platform was the Vax running Ultrix. And those machines were very slow, especially by today's standards. And people are very concerned that if Condor was running a job on their system, it would absolutely only run when it was completely idle by their definition. And their definition of idleness would change a lot. So this really drove us to implement very flexible policies for implementing cycle scavenging and allowing the owner of the resource to determine what idle was. It's not entirely obvious what idle means to any particular person. So we need to implement a lot of flexible policies to do that. And this is sort of the way that Condor got started. And it's still, even though we've grown a lot and we do a lot more than simple desktop cycle scavenging, it's really part of our DNA. And something interesting, I think, is that this idea of what we call opportunistic computing is really still applicable today, though perhaps on larger scales. We're not necessarily scavenging from desktops or from Vaxes, but you can apply this at different scales. You can scavenge from a cluster or from a campus. So even though this desktop cycle scavenging is just a small part of what we do today, the idea really lives on in all kinds of different ways. And it's a very applicable idea that can be used in a lot of surprising different ways. And you know, Greg, the second part of that, too, is that not only do we, I guess in our observations, about 60% to 70% of usage is actually on dedicated resources and not on harvested resources. We prefer to call it harvested as opposed to scavenged. The key aspect of this, though, in addition, there's also new platforms to be harvested as well. So we've been seeing use cases where we take VMware clusters, where VMware is great at server consolidation. It allows you to, you know, remove physical hardware footprint. But at the same point, it doesn't necessarily mean that you're going to get the most resources out of the hardware that's running your VMware environment. And so deploying Condor into those kinds of virtual machine pools is actually a great way to increase utilization of the underlying hardware that you've deployed. So it's not just that there's different scales, but also different environments entirely nowadays, especially for doing harvesting, where that policy management capability that Condor has is very, very helpful. Okay, so that was interesting there. You actually use the pronoun we, meaning that you guys actually work together with University of Wisconsin. So could you guys, could you describe what your relationship is? And you know, how do you guys work together? All core developers or, you know, what are the roles? Right, so Condor is an open source project. And like many open source projects, we reach out to the community and we can't do it all ourselves. We're very happy to get contributions back. And I think Jason and Cycle is one of the first corporate companies to donate code back to Condor. Is that right, Jason? Yeah, that sounds about right. Yeah, we've definitely been actively involved in trying to add new features where we see users want them. But we also try to supply the beer during Condor week every year as well. Other than that, I think we're mostly involved in evangelizing. Actually, the Condor team guys are actually doing most of the code lifting. Okay, so that's cool. That actually sounds like, you know, working relationships that we see in various other open source projects, successful open source projects. That is, could you, you already gave one example of where, you know, using Condor in places that people may not traditionally assume it, you know, particularly if they're stuck in the initial impressions of Condor from a couple years ago of the desktop harvesting and whatnot. So you mentioned virtual machines. What are some other scenarios that you see people deploying Condor in today? How do people use Condor in modern environments? Right, there's a lot of flexibility in deploying Condor. So there's a lot of sort of neat ways that people do it, which you might not think of. Continuing with the bird theme, one of the neat ways we deploy Condor is something we call gliden, which is just part of Condor, the execute side, the demons that run on the execute side, which can be submitted as a job themselves to another batch system. So in this way, you can sort of dynamically deploy and make kind of a Condor overlay network. So if there's another batch system or often a set of batch systems, and perhaps they're different batch systems, you can't submit to them all in a homogenous way, you can submit Condor and build an ad hoc Condor pool with gliden. So that's one of the neat ways that you can deploy Condor. Yeah, we've also seen folks do Condor as a way of doing power management. So the most recent two versions, two stable series of Condor 72 and 74 have included different functionality for powering down resources when there's very little activity on them. So you can actually take advantage of Condor's policy engine to say, if this desktop's not doing anything, we'd actually like to power it down between the hours of 8 p.m. and 6 a.m. so that it's hibernated and then Condor can reactivate the machine. And one of the unique things about this is that you get the power management features that saves people money right up front and saves power. It's a very good way of doing green computing. But at the same point, when there is work to be done, you have a grid schedule or a resource manager already installed and ready to accept work. So you can actually use it as well to run calculations. And that's been kind of a backwards view, I guess, of things we've seen before. Most of the time people install Condor to run Calcs. And then other features are a benefit, but the green computing is sometimes turning out the other way. So you're saying that sometimes people are using Condor literally just for making sure their machines get shut off and turned back on for different things and not necessarily being used as a resource manager and scheduler? Well, it still is a resource manager. It's a resource manager in terms of consuming power. And generally in each of those environments, we've always seen them run some form of calculation in the end. It's just that the initial motivation may be more of using an open source policy engine that can actually manage whether machines are hibernated or not is a useful feature in and of itself outside of being able to direct the proper workflow to work there at the right time. Both of those are kind of related, but still different use cases. Okay, so what's the most common way you see Condor used today? You said that you're seeing mostly actually dedicated hardware. Is it set up more for serial jobs? Does it have the ability to help an MPI launcher, multi-core, accessing different types of specialty hardware like accelerators? Is Condor aware of all these things? So, Condor is primarily used to schedule sequential jobs today. We have the ability to schedule parallel jobs, but that's not really in our strike zone. Primarily, I think Condor is used to schedule serial jobs often on dedicated or a mix of dedicated and opportunistic resources. There's been some neat work done at Marquette and some other places to run GPU jobs. So, because of our very flexible policy language, we can describe both jobs and machines that can access GPUs. So, that's kind of an interesting thing we're going forward to do. Now, you mentioned earlier, well, you just said a second ago that parallel jobs are not really in your strike zone, but you also did say earlier in your background that you've done some work on the parallel universe in Condor. So, could you explain like does it sort of work or are there shortcomings that make it not too workable in real-world use case scenarios or what's the current state of affairs? I have to ask my bias being an MPI guy. So, right. So, Condor can schedule MPI jobs. The scheduler is pretty simplistic. You either get typo or first bit. It's not nearly as sophisticated. It's like the Maui or Torx schedulers. So, we think that these serial jobs are really where our emphasis can be, but if you especially have a mix of parallel and serial jobs, maybe you want to backfill one with the other, that's potentially a place Condor can play. I think if you just have parallel jobs, Condor is probably not the scheduler for you. Again, this really gets to our high throughput computing worldview. High throughput computing is really mainly about running serial jobs. Okay, fair enough. One follow-up question on that though. So, a lot of what Condor does or at least my perception of what it does has been the migration of serial jobs around or at least that was a significant part of Condor's history. Is that still how people do this or has the migration towards more dedicated resources kind of obviated the need to do that? What we've seen is that as people build dedicated clusters, the concepts of overflow and opportunistic computing exist there too. For example, very frequently on campus you'll see several departments each having a rack of computers and the usage is very bursty, right? Often before a conference paper deadline there'll be a huge demand for the machines, a demand which will often overflow any group's clusters where the other clusters on campus may be idle at the time. So, there's a great need to migrate jobs from one cluster to another. So, I think that these problems still exist but often at kind of a bigger scale and then not just on the campus scale but again at an even bigger scale the open science grid does this in spades. And we've actually seen in companies as well the same kind of use cases where you may have a mixture of serial and parallel jobs. There definitely are cluster environments like, you know, Jeff, I'm sure the ones that you're familiar with on a day-to-day basis run quite a bit of MPI work but frequently we find that there's a mixture of throughput-oriented applications and performance-oriented applications and that's one of the things that Condor really excels at. So, again, running back to my bias here, do you have any MPIs that support Condor better than others or are there any that take advantage of your checkpointing capabilities and migration capabilities? Not right now. We tend to be MPI agnostic. Of our MPI users, they're pretty diverse in terms of MPI implementations. We still have a lot of people using MPI, even version one from time to time, and of course open MPI is coming on strong. But there's still some LAM users in there too and there's just a wide diversity of MPI. Every time you run into a LAM user, please tell them to stop. I can say that. I was the LAM guy. We've been trying to get people to stop using LAM for years. Yeah, we've seen the same thing with open MPI. It's been adopted quite a bit and thankfully in the corporate environment, I've yet to really come across a real hardcore LAM user. We've mostly been in the same kind of mixture where it's mPitch and the newer versions especially and open MPI especially. Okay, so talking about running between different Condor installation. What's a Condor installation called? Was that a flock or am I understanding that correctly? We like the term pool, unfortunately. I think that's in contradiction to a lot of other standard industry usage. There's also the term flock, which is probably used less. There's a technical term Condor flock in which is a way to migrate to federate a lot of different pools. So usually we talk about Condor pools. Okay, so a Condor pool is normally just like a set of compute hosts that talk to a single server and flocking is kind of moving jobs between servers and federating these servers together. That's right. Yeah, there's a couple of other differences between a lot of schedulers from a scalability standpoint. So most you mentioned a bunch of execute nodes talking to a single server. There is generally at any one point in time a single central manager. That's the piece that does the negotiation between jobs that need to be run and resources that are available to run them. However, there also can be multiple schedulers within a Condor pool. So as opposed to a Sun Grid engine, for example, or a job tracker on Hadoop, Condor has the capability of having many managers or job queues, each of them capable of operating independently to service jobs against execute nodes. And what that leads is higher throughput. So you have many schedulers potentially coordinating job flow, as well as being able to do execution of jobs in greater volumes. Because again, if you need to be able to handle more jobs than a single scheduler can have, you can actually have tens or hundreds of schedulers at a pool with no issue. Right. That's really been our scalability strategy is our central manager is a stateless, pretty low CPU as low CPU requirements. And we can add schedulers to the system almost indefinitely. So we can really scale by adding lots of schedulers to a fairly large pool. Okay. So how do you handle like users on authentication? Do they just, you know, users need to exist on all systems? Or how does this actually help if like to say two separate institutions want to combine their resources? Right. User identity is probably the biggest problem when federating two different pools. One interesting thing about Condor, I think, is we have the ability to use what we call run accounts. When the job is actually run, the UNIX user ID under which it is run can be run by a dedicated account that's local to that host and per slot. There's a lot of advantage of this. One is it's very easy to track all the processes on that exact host. One thing that Condor does, I think exceptionally well, is if you run a job, a UNIX job that forks or spawns threads, when that job exits, when the head process exits, Condor can kill all the sub processes, even ones that have forked, even ones which have gone out of their way to try and hide from the system. And being able to run the jobs as a dedicated run account is a very important way that we can do that. Now, obviously, when you federate, you need to have identity management. And there's a couple of ways Condor can do that. The most sophisticated is we support the GSI authentication. So it's a X400-based certificate-based identity system, and you can, on a very fine-grained basis, describe who is allowed to submit, even from remote sides. Okay, I have a question back on something you mentioned earlier, and it actually relates to a question Greg asked from Twitter. How can Condor be controlled by a different scheduler? Like, could we actually remove the Condor scheduler and control this Condor resource manager with, say, Moab or Maui or the Slurm scheduling engine or anything like that? Oh, what an interesting question. Not directly, but there are ways to, via Condor G or even Glideon, to submit from one to the other. So I think Condor resources can't be directly scheduled, but you could build up an ad hoc Condor pool and federate pools with other schedulers that way. I think one of the nice things about Condor organizationally is because we're open source, because we're not commercial, we don't really have a sort of all-or-nothing approach to other vendors. We try and federate and work with other systems as it makes sense for our users. We're not trying to sell you, and it's not a zero-sum game. Greg, we've definitely seen the same kind of use cases where folks will run, you know, this gets back to the question of should you run Condor on an HPC environment, in particular, if you've got another scheduler. We see that all the time, whether it's folks like Purdue University, they've got some very large MPI-oriented machines that also run Condor to do scavenging and kind of fill in the usage cracks with high throughput work. There's very, very, very large number of use cases around running Condor as part of LSF and some grid engine and a few other scheduling environments. And even in an upcoming release, there's already some integration with Hadoop and Condor, and I know that's one of the things that we're most excited about kind of going forward in future releases is that that work and that functionality. So, USA and a lot of installs that we're basically installing Condor alongside, say, PBS and Moab, and there'll be a group of serial high throughput users submitting to Condor, and they may be running on the same resources. And then does Condor just kind of migrate the jobs out using Condor's checkpointing functionality? Oh, and could you explain the Condor checkpointing functionality? I think Jeff alluded to that, but I think it was clear. Yeah, I think if there's one bit of magic in Condor, it's the checkpointing stuff, and maybe that's one thing we're really known for. Condor can optionally checkpoint jobs for jobs to be checkpointable automatically by Condor. There are a few restrictions. They're the kind of restrictions you might imagine. The jobs can't have an open network connection. We can only checkpoint one process so the jobs can't fork or have threads, and there's a few other minor restrictions. But if your jobs fall into these categories, then you can link them with the Condor checkpointing library, and you get processed checkpointing and migration for free, which I think is really neat. And this is sort of the way we started out. And with this checkpointing, all these jobs don't have to checkpoint. Condor can periodically save the contents of their memory, send them back to a checkpoint server or the smith machine, and then restart them elsewhere. I think this is really what Condor is known for. However, not all jobs are checkpointable, and a lot of people with this high throughput or this sort of backfill or opportunities to computing run what we call vanilla jobs, non checkpointable jobs, and just let the system kill the job when it needs to preempt. And a lot of people worry about this a lot, but I think that if you have the chance to finish your job, you might as well give it an opportunity, even if there's some chance of what we call bad put of the job being killed and needing to restart from scratch. So how did you guys, how did you get here? So we've talked a little bit about the beginnings of Condor with the desktop harvesting and whatnot. But how did you, how did you get to where you are today? Because this is a very successful project that has existed for, gosh, when did you guys start? I mean, this is well over 10 years now, right? Or 15. But how did you evolve and how did, how did you get to where you are today? I think the key bit of Condor is it's always been very user driven. Condor started, I'm embarrassed to say this in 1986. So it's really been going a long time and there's a lot of different aspects of Condor. I think this being user driven is really how we, how we migrated. We started out on campus and we had a lot of very influential users in the computer science department. And then we moved on to physics and then off to other sites. And I think when we start getting commercial users, and maybe Jason can talk to this a little bit, that's really, I think where a lot of our features started to bloom and we got a lot more traction when it wasn't just a university project, but people making money depended on Condor for their livelihood. Yeah, I know when we worked on the Disney production for the wild, it was a feature animation. You know, Condor offered both an exceptionally cost effective and the Condor team, a very talented partner for that production to be able to, so we did, I think we had a number 84 on the top 500 list at the time, 1,000 Linux cores, 256 Mac external cores, and about 300 workstations that we harvested off of. And it was, it was a great thing to be able to run. I think we ran about 75 million renders through the system at time. And since then, you've seen a lot of companies actually. So odds are, if you have an annuity retirement package that includes annuities, you're most likely the risk for hedging on those annuities is calculated using Condor. JP Morgan Chase has several thousand cores that they've described running on trader desktops that they harvest every night. I know that their annuity risk is generally on a dedicated server model. We've also got folks like Eli Lilly and Pfizer that have done, you know, essentially drug target and other forms of drug discovery pipelines, including next generation sequencing. We've seen a lot of work being done on sequence analysis. You mentioned working alongside PBS, the guys at Conoco Phillips have done work in that area around Condor and other schedulers cooperating. Electronic Arts uses it for running renders across different servers and workstations. So there's quite a lot of different traction in different industries where people are, people are adopting it commercially to actually do production work. So you've mentioned it twice now. And so I therefore have to ask, you said that Condor, you know, your origins were when you were using it to render a feature length film. Can you, can you say which one? Yeah, it's a film called The Wild. It's a, I'm actually forgetting what year it was actually released now. It was in I think 06 that it was released. It's great for kids if you follow. But yeah, it was very, the rendering was absolutely beautiful in the movie. The guys that worked on all aspects of the pipeline were great, but we had a lot of really big requirements. So, you know, millions of jobs for crowd scenes, for example, were a somewhat common event where you had to be able to handle a very massive scale. We added accounting groups into Condor as a feature in order to meet our prioritization requirements. So essentially, most schedulers operate on cues or on users as the priority element. And in Condor, you can actually make it any arbitrary designation you want. So you can, you can create groups of users based on, for example, projects or in our case, we grouped jobs by team that was working on them. So, for example, lighting versus character filing. Another option is to use any arbitrary unit of work, you know, for example, in a movie, a sequence and shot are kind of the designation of any individual camera move. So, you know, you get five or six seconds of footage that are the camera pointing at one character, then pointing at another character. That's a shot. That could also be a unit that you want to prioritize based on this. Some shots are more important than others, for example, because maybe the director has to look at them at the end of the day. So that was an example of a feature that we kind of bore in there based on heavy usage. Okay. And so we digressed on this a little bit, but I wanted to go back to one point. The evolution of Condor from an academic project to be, you know, a real full-fledged, real-world, everybody can use it kind of package. And I just want to say, at least in my experience, it's actually fairly rare that there is an academic project whose target is actual users. And it's not only a vehicle for papers, but it's actually, you know, intended to be used in the real world and solicit feedback and research that way. Was this one of the original goals of the Condor product, or did it just kind of happen that way? I don't think it was one of the original goals, but I think as we developed more of a need to do software engineering, I think we realized that this is an area that the traditional computer science community has really not done a good job at. You know, I remember when I was in school, every single week, I'd write a main, you know, and write a program and deliver an executable. And I think I write a main about every 10 years when I'm a professional programmer, you know. And I think what Morone realized is that there's a really strong need to train students, graduate and undergraduate, in a lot of the nitty-gritty details of software engineering. And I think Condor has been a great asset for the university for doing this kind of real-world software engineering training. So you mentioned Morone there. You want to give a little more attribution? Right, sorry. So that's Morone Liffney, who's the PI, and he's the person who came up with the Condor idea, and he's our head Condor. He's a full professor of computer science at the University of Wisconsin, and he started the Condor project in 1986 and has kept it going since then. And I know from a user perspective, Jeff, that the mirroring is great at both helping us out with new features and trying to make sure that they're structured in a way that'll scale across, you know, the different user communities that we have. He has a keen eye for design just as a, you know, from a third-party perspective. He very much eats, breathes, and sleeps the philosophy of high throughput. That's something that, you know, is, I think, really unique about him, is that his vision for essentially keeping machines busy all the time with the right type of work, and in a way that's easily administratable or policy-specifiable, if those are words. That's something that I know he's excelled at over and over again. So Condor has been around for a long time. What's Condor actually written in? Is C, C++? You know, when it was running out into Vax, was it some weird assembly or some other strange dialect? Sadly, no. Condor started out, I think, like a lot of projects of its day in C, and sort of migrated over time to C++. I think today, almost all of it's in C++. There's a few C functions lying around. There's the occasional bit of assembly, especially in our checkpointing libraries, to do the really low-level Cronkey stuff. And there's a little bit of Java code to interface to some external Java libraries. But primarily, it's C++. Okay, and then what platforms does it actually run on? One of the common uses I see is actually getting cycles off of Windows machines. That's right. It primarily runs on Windows and Linux, although we do support some of the other older versions of Unix, but we see less and less demand for that. We also run the Mac, which is becoming more and more popular, especially to use a Mac laptop, perhaps, to submit to a cluster, maybe, of non-Max. When we say we run on these machines, we're very heterogeneous, so you can submit from one kind of machine to a pool of a completely different architecture. This is often common, and I think some adjacent stuff has done this, where for reliability reasons, you might want to have a Linux central manager or maybe a Linux scheduler, but because your jobs may be Windows jobs, you may have a pool of Windows or perhaps a mixed pool of Windows and Unix machines. That's definitely a common architecture. We see that run over and over again, especially when it comes to harvesting Windows workstations, the UTC mentioned, Brock, that's very common. Yeah, so there was a condor pool in Terragrid from Purdue, and that thing was actually really big. As decent size as the largest I've ever seen for condor, it was 14,000 CPUs in that pool, and they were constantly saying how many they had Linux, how many they had Windows, how many they had running Solaris. What's actually the largest condor install you guys have ever seen? I think it's kind of hard to measure, especially as we federate pools, but I believe that currently Purdue has the largest condor pool, and I think right now they're over 20,000. So we do scale up. Actually, they're at 30 plus K right now. Purdue is actually kind of interesting in that they have a campus-wide mandate, I believe from their chancellor, that says that every computer on the Purdue campus must either be running condor and scavenged the cycles off of it, or be turned off at night. So we think this is a pretty forward-thinking approach to have a very campus-wide directive to be more green. All right, so this might be kind of a difficult question to answer, given that you support such a wide variety of running environments and scenarios, but just from your personal points of view, what's the strangest use you've ever seen of condor? Or strange is not necessarily the right one, but perhaps most unexpected, or wow, I never would have thought of that kind of scenario. That's an interesting question. A thing called a Google Vanity Search, where you type in your own name to Google and see what comes up. And I think if you want to take the level of vagueness one step further, Google has this great product called Google Alerts, where you can sort of do that on a periodic basis, and when it finds something new it emails you. And I do this with condor, and I recommend anyone who works on an open-source project or any software project to do the same thing. And it's just amazing. Every week, Google will find a new condor user that I hadn't heard of that we had never talked to. Just a couple weeks ago there was one, a research group in Singapore, and again, this is a group that's never talked to us, never asked any questions on our email lists, posted a paper wherein they had used condor to analyze various different strategies for their Navy to repel pirates around the islands. So they developed a different strategy where to put the big ships versus the little ships and they modeled pirate strategy and naval strategy and try and come up with the best strategy for fighting pirates. So that was completely unanticipated and not what I would have thought that any condor user would need to use condor for. I thought I was going to remark on Google Alerts, but I just have to answer the pirate thing. That is just so cool. That gives a whole new meaning to Pirate Day when all you condor guys go into work wearing eyepatches, I think. Yeah, I don't think I can... Talk about Pirate Day has all new meaning now. So yeah, I also have to plus one on the whole Google Alerts thing because I do that myself for my open source project and also my competition. Well, mine are probably there. But it's also nice to see that, you know, my project happened and then the other guys too. But yes, it's a fantastic way being an open source guy myself to see where it's going because exactly as you said, you don't necessarily know that people don't talk to you particularly if they download your software and it works. But they might write a blog entry about it or post to it on some forum that you've never heard of or something like that. It's a great way to find out about it in the wild. And I also use them as a way of someone, you know, is ranting about open MPI. They couldn't get it to do something or other. Well, I'll just go sign up on that forum and answer their question. It's actually very useful. Yeah, we haven't actually seen anybody doing pirate protection or various swarly strategies. I think the key ones that we've seen are all what you would expect, but might be unexpected. So the great thing about Condor week every year and this year, it's on April 12th up at the University of Madison, Wisconsin. As you get to hear all different kinds of use cases. So people have come and talked about how they catch students running MP3 encoding applications on their desktops at their university running Condor. We've also had folks talk about doing song analysis, so taking the, you know, the net result of audio files and trying to figure out which songs are near each other. I think the research is from, you know, from pirates to musical similarity. And everywhere in between, you can imagine use cases, corporate use cases generally tend to be stuff that might seem boring, but is actually kind of, you know, real world stuff you use every day. So, you know, whether it's, you know, kind of dividing designing new products, new chips. In terms of Altair uses Condor to do the FPGA work that they do, they test software on the back end using Condor. We end up seeing a lot of use cases around risk analysis and around, you know, protecting annuities and retirement contracts. That's a very common one. Another one that is maybe a little bit, a little bit strange is kind of a lot of the work that's going on with genomics. That's been something that I know has been a big eye-opener for us, has been essentially the work on, now that the Human Genome Project's done, people are creating instruments that at reasonable costs, i.e. thousands of dollars, can produce a genome and the target is to be able to do this in around a thousand bucks, you know, within a year or two. And essentially processing all that genetic data is something that, you know, the type of gene searches they do and the impact it has on, you know, essentially being able to discover autoimmune diseases, you know, like your diabetes, your MS, et cetera. It's amazing, you know, the different application areas that you bump into every day with a project like Condor. So as an open source project, how do you guys manage? I would imagine that the majority of, I think you said earlier, the head of the living is done back at Wisconsin, but do you get random patches from people? Do you have other people around the world looking in your source code, making suggestions, submitting bug fixes, things like that? Assuming that you write bugs, which you probably don't. Just undocumented features. That's right. Yeah. So as Jason alluded to, most of the heavy lifting is done at UW Madison. Like many open source projects, I would say now maybe 10 to 15 percent of the code is written by people outside of Madison who are paid to write the Condor code. We have a good close relationship with Red Hat. Red Hat has hired a number. It's like six or seven engineers who are full-time Condor engineers who contribute back to the Condor project every day. So just like the Linux kernel, it's not necessarily some random person that she never heard of who submits code out of the blue, although that does happen from time to time. But there are a lot of full-time paid employees who aren't employees of UW Madison or the Condor project who contribute daily back to the code base. So a follow-up on that. This is a question I ask all open source people just because at a genuine curiosity. What do you guys use for source code management and why? So about a year and a half ago, we switched from CVS to Git. So we use Git exclusively now. And we kind of have a love-hate relationship with Git. What do you guys use? So we use Subversion, but we combine it with Mercurial as well. So Subversion is kind of our main back-end one, but we basically do all branching in Mercurial. So I would say at least 75% of open MPI development is done in Mercurial and then eventually pushed up to Subversion. But boy, that's quite a jump from CVS to Git. Yeah. So like I said, we have a love-hate relationship with Git. There's certain things it makes really easy to do and then other things are kind of hard to do. So I think a good benchmark of your version control system is how often you revert a patch. You know, it works a lot of places where people are sort of terrified to revert patches and they never do it. And I think if you're in that mode, you're not really using your version control. You're really just using kind of a fancy backup system almost. But we do revert patches from time to time and that's not a big scary operation like it might be in other version control systems. So I think all in all we like Git. It's not perfect though. Okay, guys. Well, let's get some contact information for Condor. Is there a mailing list website, download packages, cycle computing's website? Sure. So probably the easiest rule to remember is condorproject.org. So that'll redirect into a complicated University of Wisconsin site. And then there's the Condor user's email list, which you can find on condorproject.org. We also have a ticketing system. So if you on the off chance that you find a bug, you can send us a bug report and we will fix that on a best effort basis. If you need more contract and support, there are several organizations, including Cycle, that can provide paid support for Condor. Maybe Jason can speak to those. So yeah, Cycle definitely provides support for corporations. We can be reached at cyclicomputing.com. We also have management tools that we're actually going to be making downloadable in the coming weeks leading up to Condor Week itself. They make it very easy to have a business analytics for HPC environments style view into what's going on to very large Condor pools. We work with Purdue on their 30,000 plus core environment to help make that manageable as well as a lot of the various companies I've mentioned earlier in the broadcast. Hey, so what's this Condor Week thing? When is it and where is it? So Condor Week is an annual event. It's been held every year for at least the last 10 years. I don't know, Greg, if there's a longer duration than that. Yeah, I forget when the first one was, but it goes back at least 10 years. And generally there's a confluence of government research, university research, and corporations that come and talk about various use cases around Condor. What's new, what's happening in the next week talks. And every year for the past, I think four or five years that Cycles been around, we've also sponsored a meet and greet generally on a Tuesday evening of every Condor Week for people to get together and share stories and have a few drinks together. Cool. Okay, guys. Well, thank you. I think we're all set here. This will go out this weekend and thanks a lot for taking some time to talk with us. We appreciate your time. Thanks guys. Yeah, thank you guys for having us out. Yeah. Thank you.