 Hello, good evening excuse us for the technical difficulties now. We are gonna have a talk about HPC in deviant the distribution for Clustering and Fabrizio Kanini is with us. So Fabrizio Hello everybody Since I came here in last Saturday, I was thinking of what I'm going to tell you I Not I'm not sure about it right now yet but What if what I know is that I've seen some problems when I'm was first when I'm beginning researching to change my the rocks Linux Installation of my company's cluster I noticed that there was some that it was pretty much documentation available but What I noticed that most of the times that the things that just don't work Like you had a lot of things that manually do You had to do you had to edit configuration files. You had to make templates or whatever sorts but There's a lot of manual work to do Things that I You didn't need to do like when using rocks for example And also it seems that people Doesn't seem to be aware people what I'm what I mean people is that Most common of people that use clusters like scientists engineers Especially for numerical simulation problems in weather forecast or Physics or chemistry or Geoscience is also so whatever whatever you need a huge amount of computing power Most of the people they don't they don't seem to to be aware that Debian's also a great distro for making this kind of For this kind of task so What I'm actually what I'm actually proposing right now We have the tools Have the tools you have a lot of Tools by this by this In this area MPI implementations libraries Task managers and You go on Compilers we have G for trying we have GCC. We have G Plus plus that now on G's now on version for three day F Open MP implementations which make for Which makes which makes them on the same level that commercial compilers let path scale like Intel like PGI I Think that we should Reach out these people see hey What do you think about using Debian this in the cluster situation or? Especially people there is already using be already using Debian on the situations Hey, what do you think about? What do you think about using Debian in the cluster situation? Was it hard to set up? Where where you hate to find any difficulties What could go what could be automatized what is the the boring tasks that could be automatized Also Yes, of course reaching out people that don't use that don't use of that have not have to choose to not use Debian for This kind of environment why? Why do we choose another another solution of? Why have you choose to go with Susie or redhead or rocks or? Sentos or Whatever see where you can where you can? Improve I think that You know most of the cases that Debian is successfully used in In a cluster environment. It usually takes what I call an inside champ For doing such a things something that somebody that is already on the organization and that Already knows Debian that is has been using it for quite some time and This guy knows knows how to do it and he wants to he wants you to do it But he wants it needs most of the times that he needs to convince his his workmates or his managers sorry His managers that Debian is indeed the right to the right thing to do the right district to use I think that we We had to Pick people from like Debian science mailing list which has a lot of HPC users I Think we had to like Like I said before we have to we have to reach out. We have to know what these people need so that we can so that we can Improve Debian so that we can Go on one step further to avoid domination I Think it also would be it Initially it would be great to have to to make a main list to discuss what is exactly this this this people need so What I'm proposing to do I'm going to well I'm going to to reach out these people and see what they want in a research like like of what Paul has done peps To But more specifically oriented to to this kind of situations I'm also I'm also going to Start working on Debian cluster components if you know it is It's a very nice Project it is You know have my lesson right do we have do we have oh right? Well links will be backed on on the on the lecture on the The session links and contents This guys I've been talking with I've been talking with the developers of Debian cluster components and They seem to they seem to really like Debian They really want to know they really want you to make this work. They I Think they share the same the same wheel that I am that you make Debian the distro for Cluster environments, so now I'd like to see I'd like to to know I'd like to know from you Who from you are using Debian in such environments? What to think about what it could be what is good what is bad and what is ugly and What can we do to improve it? Which packages would you like to see in Debian? Which packages do you use? One of the nice things about Distributions like Oscar and rocks is that typically you can have a single CD that you put in the system You boot it and then you can boot your different client nodes off of the supercomputer That type of a distribution would go a long way to helping people utilize Debian in say a high school or situation to Allow the students to practice using and programming supercomputers Yeah, there there are some there are some There are some projects like these there are some like Pelican like Pelican HPC That there's a it's a bit. It's a bootable CD There's also parallel gnomics that it does like it does something like this Debian cluster components. They also do They also have a city like this that you boot on a CD and that you They have already You have a host and three virtual Notes so that you can so that you can use it for Learning how to manage a cluster how to use MPI how to program with MPI and how to Not even this but you can do like learn wrapped in your Anything you like and it has a lot of and has a lot of The standard tools that you use it like LAM MPI on pay MPI Compilers Python and Java I Think it's what this is nice for this is nice for educational purposes Should it also it's something that we should take a look but What I'm what I'm What I'm trying to do right now is just to Choose to see to know what exactly what the point is at which point is we are right now and we have a lot of users and we have a lot of tools but What more do we need? to to to make it To go up and to to make things better to to the users which is which are Our final target and to enjoy answered and what's your personal background about clusters I Work in a weather forecast company in Porto Alegre Brazil So I've been in there in more than more than a year working working there. We have a six node clusters Six node cluster which he does The weather forecast for the whole Brazil twice twice a day Yeah It does the weather forecast for the whole Brazil twice a day it takes some Three and a half hours for the simulation to to to complete It takes also some 10 gigabytes per node of Generated data Yeah, that's what I'm trying to that's what I'm trying to have I've been researching and that's what I've been researching that right now. We're using a rocks Linux Installation that's what I'm trying to to put them out and to to put debium in And while researching this that's what I've I've come to to realize that What people were doing to What people were doing to to have Linux to have Debian running in a cluster environment So since since last year have been What can I do to make it To make it better. Why can I do to make it? things goes mother or Go just works so for people In the cluster environment Another question that you would have a marketing standpoint is What is the penetration of Debian in clusters as opposed to the other competitors? How many people are using the different versions of Linux in In that and you know if Debian is 2% well, then you might say that there's a lot to be done If Debian is 90% then you say well, you know, the other is just noise And so the question is how many you know what percentage of sites use Debian? I Guess you could say that Debian is not really the number one in the number one in In clusters usage At least if you if this looking by the top 500 Clusters of well top 500 dot org if you want to know There's a lot of there's a lot of Clusters that they just don't they just don't they just don't Don't release which version or which distro they are using They just don't they just say Linux But the most popular distros for cluster cluster environments, they are Susan 9 and 10 Redhead and the price three and four after that they come sent sent OS and Rocks and after that Debian There are some there are a few clusters running free BSD One or two running HP UX and a dozen running Half a dozen one and AI X and Solaris but Linux Linux as a general is is is the top dog in Cluster environments, but Debian specifically is not I think we should I think we have a long road to go to To go. I think we have a Lot of nodes to to get into if we wanted to be really representative in Cluster community to what answer Well again my only Comment on this is if you contact, you know So you've talked to some of the companies that produce cluster hardware and some of them Have Debian and do use Debian So the question I would ask them is why did you choose Debian over a different distribution and then to contact the people in the In the places that don't use Debian and say why did you choose this other one? I remember when red hat first started charging for their enterprise edition and Their charge was something like a thousand dollars a node and they came to the super computing conference in I forget where it was that year and I went up to them and asked them how much are you going to charge the cluster people for their clusters? And they said a thousand dollars a node and I said good luck So I mean obviously they've changed their licensing somewhat since then But I would imagine that people who run clusters who run red hat enterprise are still paying some amount of money To red hat and for something I hope So you know there must be a reason why they're doing that Again my understanding of rocks and an Oscar are basically that they are more or less You know they use a distribution as a basis But then they add their management software around it and it isn't it isn't a distribution like red hat is a distribution Or Debian is a distribution It's more of a wrapping around a some basic distribution Yeah rocks is based on on sent OS The version that we use that at at my company is rocks for to what One I think it's based on Santos and sent OS for It's a ghost At Sarge time I think So one thing one thing that might help convince them to go with Debian would be more of a liaison between Debian and themselves so that every time there was a new release of Debian that came out It would be easier for them to upgrade Then you know that it might be with sent OS if sent OS wasn't paying as much attention because sent OS is based off of red hat So there may be a Disconnect there that makes it difficult for them to keep up with releases of sent OS That Debian could help to solve with a little bit of communication back and forth I guess that because sent OS it it follows more or less the same releases schedule of red hat enterprise is just Cleaned up a red hat Cleaned up and recompiled red hat enterprise so they have the same reschedule and and Maintainance dates maintenance Periods like red hats enterprise I think one of the reasons that people People don't use Debian is sometimes is the lack of the inside champ Which I've which I've spoken That without this guy People they don't they don't really go after Quite a lot they just They just Take something that they know that somewhere else Did it and it just work it they they just installed it like Real scientists and engineers they don't Actually, they think about the same questions that we as developers think they don't they don't actually think about this Freedom and Open source they that we do praise to that we do praise very much so that's why I think that Debian has a long time to That has a long road to to walk to to it is a A big player in the cluster market We have we don't have Mindshare if we want to say so and Because people they don't act because especially because When installing Debian in a cluster some things that they just don't work. They don't just works people who gets afraid of using Debian because oh because Debian is hard to install that same old phantom that Debian is hard to install Well, I Work at SGI and I have had access to an ice box Which is what we sell as a cluster and it was my bounce box So I tried to install Debian on it and The Lenin installer at that point wouldn't boot on it So like I could debug it and spend time on it But at that point I just really need to work and I believe that when you get to deploy a cluster You just don't want to debug an installer. You just want it to work Actually, it looks like a few years ago as I was using Debian as the base OS for the operating system that was shipped on the clusters Again, that's I haven't been in the company for a long time. That's more cordial noises than anything else And I've been trying to look why we moved away to go to Susie Part of the things could be that Susie slays 10 is shipping a 2616 kernel With a lot of work that has been made to backport all the fixes to the kernel while maintaining a lot of ABI compatibility That in those machines you sometimes have a hardware for which the vendor will ship you a binary module that will work only on one Kernel and and that's really important to make it work and the kernel team is Not doing that in Debian because we don't have the manpower at the time to do it But that's things that you require to make it work properly I've been told about a network driver which had a free software implementation that was really really slow Compared to the binary one and absolutely unusable for people in that environment. So that's Some of the things yes, so Spedial there there really are sort of I think three categories of things that get in the way of Debian having a larger cluster market share right now And if this is something we want to work on we have to figure out How much of this we can address and then come up with a good plan for addressing it? The first one is that lots of clusters do run with interesting interconnect fabrics and things like this for which the Device drivers may not be fully integrated into a kernel or a kernel and therefore may not be in our default kernel packages that we're shipping and You know there's there's always device driver issues with different kinds of hardware But there's plenty of cluster hardware today available today that Debian runs very well on so There's certainly that class of cases that we can think about But I don't think we should get bogged down on that because there are two other classes of Things that we ought to think about one is that a lot of people Running clusters are running, you know commercial ISV provided applications on those clusters and when they do that they have the same Problem of the choice of underlying operating system that anybody with a dependency on a commercial Application does and that is that that application provider is going to choose the set of operating systems that they choose To certify that application for use on if you want to have support for it and today if that's Linux It's probably one of the commercial distributions for lots of you know good business reasons And there is a chicken and egg situation there where if there were lots of demand for those applications on Debian for Use on clusters they would probably port them, but you know How do you get there if the applications aren't available? How do you break that cycle? but the third thing is that for whatever reason the there are a number of Cluster oriented distributions that are out there. I mean rocks is one example. There are others And many of those seem to have been built on you know a de-branded Rebent branded red hat package base it may well be because they See some opportunity to be technically compatible with Some of these binary applications that are built and certified on red hat or Suzer or something I don't really know what the driving force or the history behind that is but it seems to me that one Opportunity would be to create some custom Debian distribution or other Lightly derived version of the distribution that specifically targeting the cluster community and includes You know some interesting management tools and related things that would make it attractive to them I don't know of such a Debian based distribution exists right now or not But if not then when someone goes and starts doing you know Google or whatever search is looking for Linux and clusters They may not see us as being an immediately obvious choice Yeah, one of the one of the other barriers that have seen to to Debian Adoption in clusters is exactly of proprietary applications like Floyd dynamics applications that are I don't remember them the application but It's lots of engineer and lots of engineer applications like wind tunnels wind tunnel simulations and that use a lot of dynamic fluids computations and They are very very expensive applications and they just and when you see when you go to the to the to the the software house web page it it only it only is It only says Red Hat Enterprise 3 4 Susie Susie 9 Susie 10 and that's it about Linux Another another operating systems may have HP UX or Solaris or AI X from IBM But that's that's it I think that's Another another thing is of as of drivers that like they said like mirrored There's no there's no free Mirinette drivers Available today only binary blobs That you just dump in your kernel edit. They they are made only for Susie and for Red Hat So this there's another barrier of Inclusion why on the other side? Most infinite band providers that I know they they have free drivers and Most of them are already on the main mainline kernel And that's a good thing Well actually but Dale says he doesn't know the history of it But I do and the thing is that back in the early days of the supercomputers Red Hat gave a lot of attention to it And they actually put out as a very early CD called the rocket science CD And it just had all of the software for clustering of that time on that CD And it got a lot of the people in the national labs who did a lot of the work on things like rock and oscar using Red Hat Which is why they were so upset when Red Hat changed their licensing of course sent OS got around that But when you say that yes, you know You know infinite band is available with some open Drivers but mirror that is not well, but that's the thing with Red Hat or Susie you can get both and you're not tied to just one See, I mean it's the whole ecology of it It's also the ecology that that a lot of the people that work on supercomputers are not computer scientists They're scientists or engineers and they're still programming with programming with Fortran and stuff like that Not to do anything wrong with Fortran Fortran 77 Was some sometime 90 but in any case in a case what they want is the same thing Running on their notebook so they can do their preliminary programming and everything and then take it over to you know the cluster and And these are all a group of things to do it my greatest my greatest suggestion would be to aim towards putting together a package that as before which would get in students Using Debian so as they went up and came out they would say hey Debian's a great distribution for doing this You know and that would be your inside person that you say you're champion To getting it spread out across a lot more of these areas Would have picked were to pick a side of suit of high-performance computing that a lot of the national labs For example, don't even think about for example financial Computing data mining, you know that type of supercomputing that you know, it would be a whole new marketplace That a lot of the Debian packages could actually utilize that Would it concentrate on things like the real multi-core type of programming so open? You know open MP To get that going and pardon me for saying so but from my experience the G Ccc compilers do not produce the same type of performance That you would see out of a really commercial compiler that is tuned to particular architecture You know you're gonna see a five or ten percent performance that over two thousand processors is like 200 processors that you got If you look at I 64 one of the architectures still used in HPC You see that last time I checked on half of the spec benchmarks actually the pro 64 compiler Which is the open source all this GI compiler is still the fast compiler much faster than a vendor compiler Well, I wouldn't use the itanic anyway, so But but as I said pro 64 is picking up a little bit of stuff Intel has some people working on it also for other platforms There is LLVM So it's not like easy sees the only compiler we still have Okay, but the thing is when you're seeing there with Your stopwatch trying to figure out how fast the compiler is because all you invested in is CPU As opposed to the portability I guess that I Guess that's not just there GCC is not that bad. I've seen tests made for by Brazilian University showing that with With the same level of optimization GCC for two has less than one percent of It was less than one percent slower than Intel nine compiler in C and C++ But it was it was much lower without optimizations But I think that I Think that GCC Why it has it's it's bad things and the law but GCC is Going Pretty strong on and it's getting better each release If we if we do the same the same test again, which is we we just see for three I'm sure that the difference between GCC and Intel compiler nine would be even even smaller and One of the things that you should be pushing with this is in the universities They're trying to teach computer science and trying to teach high-performance computing is the fact that you can see the source code For these compilers where you can't see the source code. That's a huge advantage for in an educational space Or if that you know, you should be leveraging the openness of the operating system and the openness of the compilers to the educational marketplace and then you'll build that that you know support as As the students come out of the college and go to work for different companies Yeah, I had a few words After you talk I personally work in a French aerospace lab We are big users of high firms performance computing supercomputers and alike. This is not really my field of expertise I'm very sorry for that. Oh Supercomputer supercomputers are as far as I know running commercial Linux distributions For only one reason because because this is why what we were supplied with and only for that The other reason probably is because as you said and I wanted to confirm what we said the the showstopper also has been probably the commercial libraries the commercial Fortran compiler. Yes, we are using Fortran. I confirmed that sometimes Fortran 90 and Actually, I think that the people were using our supercomputers and our cluster Don't care a lot about the C compiler It won't care at all, but they care a lot about the Fortran compiler That's on super computer And if this Fortran compiler is for instance provided by Intel and only works on one reddit Derived distribution there will be no choice. We will have to go with them Just to have an idea all the all the the weather forecast models that I use in my company They just they it is Almost How they've written in Fortran 90 Yeah, we actually use just the MPI part is written in C We use the very same kind of supercomputer than French Meteor Center And this is exactly the same culture and this culture is a very old culture And it takes a lot of time to change it and not only From through the students, but the through the infrastructure people and in the infrastructure people those will run these supercomputers to also get this idea that there are some Free distributions around Anybody else? Well, so I'd like to thank you everybody to come this come see me bebo this this stuff I Hope we can we can talk a lot more by mailing lists or by private mail or ear or whatever And I'm I'm personally going to pursue it. What I'm what I what I say here today And if if anybody wants to Anybody want have any idea or wants to talk me with me after that I'll be at the hack lab So thank you a lot for coming