 Cuing systems, we're going to start general and go back on specifics for what BayoCAD actually does here. A queuing system, just like a queue in a theme park or whatever, is jobs are submitted and then processed according to the scheduler. So you take your place in line, essentially. It's more like a mainframe, the old batch computing systems of the 60s and 70s than it is like a desktop or single server. You can't preempt what other people are already doing. Yes. Is that a question? You use those? Me too. I'm old. We do have preemptive scheduling, so we have some nodes that are actually owned by particular groups. So that that means that they have priority access on that machine and if your job happens running on their machine then they kick you off because they bought it. So that's the way life works. But in the meantime, you get to use it for free in the mean and so you're not really at anything. So the utilizing resources is a gameless BayoCAD. The biggest users we have of BayoCAD are people who have set up their own clusters and have realized within a couple of years that the really expensive part about setting up a cluster is not the hardware that's in it. There's a lot of hardware there, but that's not the expensive part. This expensive part is keeping it going and good system administration, keeping up on the latest security patches, all that kind of stuff. That's the hard part of a system and security and all that kind of stuff. So we've had places that have gone through and bought their own clusters and run them for two or three years and said mind if we just run on your stuff from now on and we'll just give you money. So we started this on a budget of zero using used equipment and a few grants, that kind of thing. It's how it all got started. It's built up from there. We also have researchers that are buying their own resources on others. Some in this room that are in that group just from the names I saw that were registered for the class that they bought some of those particular machines with their money and so now they have priority access either to general to the whole machine or to their particular nodes. Not saying that there's no disadvantage. The disadvantage is that you don't always have immediate access to everything. You do have to take your place in line. This is kind of a schematic of Baocat and it's kind of hard to see here. What we've got here is we've got the campus network. We have our Nichols Hall router comes into our firewall and our firewall actually has four networks sitting off of it. Actually, we just added a fifth. So we have our management network, which is how we nobody else has access to that except for actually Adam and I. That's how we turn machines off and on and do our monitoring that kind of thing. OpenStack, which is kind of shaky right now. The future plan is for OpenStack to be a place where you can spin up your own VM kind of like your own Amazon cloud sort of service. We're not we're not ready to go real live with that yet, but we're getting there. We have a production network, which is our main what we consider Baocat and then we have a hosted virtual hosted network, which where we we have a few researchers that have a particular Windows machine or some licensed software. They can't run on a cluster without paying ridiculous amounts of money. So we set them up on a virtual machine so they can run that kind of in their own space. We have several different types of connectivity here. OpenStack, try to think. The orange is our fiber channel. That's very fast disk access. That's what that orange is. Infiniband. Normally orange is fiber channel. Yeah, this is our Infiniband. Infiniband is a very, very fast network mesh capability. It talks at 40 gigabits per second, which is quite a bit faster. Most of our other machines are down getting up to 10 gigabits per second, which is pretty fast, but it's also very low latency. So if you're talking MPI stuff, MPI stuff runs very well over Infiniband. We have our regular network, which is in the blue. We have some other strange things going throughout, like Hadoop, and we have a Ceph network for our OpenStack. But it's not just like you'd see in a lab, something like this, where you got to switch and you got a bunch of machines out there. It's a little more complicated than that. I wasn't able to get more recent numbers, or at least not in the short time frame that I had to update these slides. But I think we have somewhere near 600 people on Beocat now. So you can see that it's gone up through and we're over 2,500 cores now. I probably should have been able to get those numbers, but like I said, I just I grabbed the slides from our last class and it was a couple years old and I forgot to update them before now. So getting new numbers is fast, but so we're still growing. We're always growing. We usually buy more resources at least twice a year, sometimes three times to add into Beocat. There are times we take the oldest stuff when we kick it out and we give it to other parts on campus. We have, in our main network, we have several different classes of machines that serve different purposes. Our scouts are the oldest ones that we have. We bought 76 of them. I was guessing we have about 50 of them in operation now. There's eight cores, two 4-core optorons, grabbed this off the website. Eight gigs of RAM. There's some of those. We stole eight gigs out of some of them. We kicked out and put them in another one. So yeah, we should have some with 16. Kind of a strange lot there. There are the paladins. The paladins have 24 cores, two 6-core, 12 cores, sorry, 12 cores, 24 gigs of RAM. They're the ones that are GPUs on them. If you wouldn't do any CUDA type programming with the high-end GPUs, that's what we have out there. Please do if you have anything that could take advantage of it. They sit mostly not doing anything most of the time. So if you can take advantage of those GPUs, please have at it. We have the mages. We have six of those. These are our big monsters. They've got 80 cores, eight 10-core CPUs in it, and a terabyte of RAM. I tell this to other play other schools and they all start drooling because very few places have machines with that much RAM in them. They've got InfiniBand and 15 gig yet. Yes, they have 10 gig up to those. I don't think I'm going to worry. The old ones, I think we have 80 and 79 or 80. Something like that. 80's squeeze has got the new ones. Yeah. So they are from 16 to 20 cores. These are the fastest ones we have available. Most time we can have people specify something that went around the elves because they run really fast on there. We have at least 64 gigs of RAM on the 16 core machines. On the 24 core machines, we have 96. Sometimes we even have 384. InfiniBand, Andor, 10 gig Ethernet on them. A bunch of nerd talk here. Sorry. Baocat itself. How to get an account. Baocat is available to any researcher in the state of Kansas or collaborating with a researcher in the state of Kansas. Well over 90 percent, probably 95 percent or more of our users are on the K-State University campus. We do have some from other universities. We have a couple from Washburn, Pittsburgh State. I think we even have one from Benedictine, if I remember right, and people that collaborate with them. So we have some people from Michigan, for instance, that are dealing with some researchers on campus. All you have to do is get an EID and then fill out the account request page. And if you hold it out correctly and you get approved, which is pretty much a given if you're a researcher, then you're getting an account. Logging in. Everybody here is probably a user already. User name, we use EID credentials. So we do have a couple times a year, somebody say, hey, I forgot my Baocat password. I said, I don't know what it is either. You have to go talk to the EID people. And usually we get a cheapest response like, oh, yeah, I forgot that. I just changed my EID password and didn't work. Um, creating programs. We talked a lot about the programming, that type of thing. Running your own toolkits. I'm not going to go into that in any great detail. But we do have all the development resources out there. So you can do your own programming. If you want to, you can download your own toolkits. We've gone over how to install things in your home directory, that type of thing. We do have, whenever you log in, you'll either see Athena or Minerva on the command line. That's the machine you're logged in to. Those are our two head nodes. There's maybe a few of you that get a Pade. If you're in one particular working group they bought in a head node because they're doing some special things on there. I'm not thinking if this group was on there or not. You'd probably nodding at me now if you were because you do that on purpose. But running jobs on the head nodes. We try to limit to one hour CPU time. We don't enforce that. Um, right? Rules change all the time. So these are meant for like light prep work. A lot of your pipelining stuff that you use in genomics where it submits the jobs for you. That serial part until you go serial and parallel, serial and parallel, serial. A lot of times that serial part is not real heavy duty workload stuff. So we'll run those on the head nodes and let it submit the jobs. That's fine. And used mostly for testing. I mean if you've got a program that's going to run for 30 seconds and even using the entire thing, we're not really going to care if it's taking 30 seconds. The idea of it is just used for testing. Get it tested out there. Make sure it's working right. On our system before you submit a job. Hey, Catur. Hey, we already did that. Okay. Submitting a job. Yes it is. Dang it. I'll get that corrected before I put these out there too. We'll go out into the... There is a page here called SGE Basics that talks about how to do all this stuff that we're talking about. This is what I was talking about. I try to lead you through step by step on this because I know that when I first got started personally I had lots of Linux experience and that part wasn't the assembling block for me. But when I first came here I'd never used a queuing system or anything like that. So this was kind of the hurdle for me. So I just kind of wrote up some documentation on how to make this happen. We use the queue sub-command to submit to our schedule. Our schedule is called grid engine. Sometimes they call SGE for sun grid engine, which is where it came from. Although technically we got the fork of it, which is now sun of grid engine. So we call it SGE or grid engine. That's our queuing system. There are several different queuing systems out there that all do essentially the same thing. If you come from other universities you might have seen Torque and Maui or PBS, Slurm. Yeah, there's several of them out there. They all essentially operate in the same way. They're ways of scheduling resources mapping that onto the system. Basically if you can think of having a giant spreadsheet of all your resources out here along the top and all the people who want resources out here along the side and trying to map those up. That's what it does. It tries to say US for eight quarters and 16 gigs of RAM. I can fit that in over here. That's what it does. Or it might say I can get you in anywhere for a while, but I'll be able to in... We're here and so kind of reserve your spot there. That's what the idea behind a scheduler is. So I'm not going to read to you. I try to do that very limitedly. I doubt I did a little bit there, but here's a Q sub command that might, for an example here, I created one called myhost by SH which just echoes the host name. Q sub it with a time limit of 60 seconds because I know this isn't going to run very long. I use the move amount of memory possible, one processor, and I ran it. All that does is it submits it to the Q and that goes and brings the host name back out. Your output, whereas your... Yes, between a Q and... Which Q you submit to. You can. However, if you give it the correct resources, it will do a better job scheduling it for you than you probably can yourself because it's going to try to optimize that for wherever it'll fit. So if you tell it what resources you need, you tell it how much RAM you need, you tell it how many cores you need, how long you need it for, it will schedule that for you. That is its job, is a schedule it for you. We have people that try to use the... As a matter of fact, we had somebody just today try to use a specific host. Well, that host was busy, so the job is going to sit there forever and it could have already been finished somewhere else. So that's the reason why you tell it what resources you need as opposed to which machines that you want, by and large. We do that part automatically. So here's what I did. I created the file, I submitted my job and it popped back up because it can schedule that just about anywhere. When I did that, this is what I started off to begin with. I started off with just... When I listed, I could see my host.sh. Now when I do the listing, afterwards, I see all these other files that it creates. It creates O files, E files, PE files, and PO files. Those are for output, error, parallel error, parallel output. And if you look at them, usually, when you run something on the command line, it'll just send text back to you. Well, these are happening on the machine, so that's not going to work out so well. That sends it to these files when you queue sub. And if you take a look at them, and I didn't show it on here, yeah, we kind of go with that. I said, cat, this file name. I did it in the order I wanted to on purpose. And it shows nothing. So that file was empty. That file empty. That file was empty. But now, when I did the last one, my output file, you'll see that it has, that's the output that would have come on the screen if I was running it interactively. When you put it in batch mode, your errors are going to go into one file, your output's going to go in the other. Sometimes we'll have people come in and they'll say, I don't have any output. Well, did you check your error file? It'll tell you when the error file or something went wrong. So that's where you really need to look for these. You also have some ways of monitoring jobs. The status command will tell you that things are running and what's waiting. I actually created jobs in particular so that one would be running and one would not be running so that I could make this example up. But the status command will tell you what's running and what's not what's waiting and how and when it was submitted, when it was running up here and status dash r will give you a relative time. So it's saying it started 27 seconds ago in this case. The question people want to know is when my job is going to start, which I think is really a silly question to ask because the real question you want to know is when will my job end? You don't care when it starts, you want to know how soon is my job going to run, but they are related so I can understand that. So I just took a, we have a command out there called QStat and QStat will tell you just about everything you want to know about what's going on. So I took at this point in time before I was writing this article and I just ran a QStat command and I edited stuff out. I can put some things in here, kind of some boring parts just so we can kind of see some things that were running at the time. So anything with the lowercase r in this column, in the state column means it's running. The capital r over here means it was resubmitted. Right, I said they're rerunning, yeah. So something went wrong with it the first time and automatically restarted the job. So the capital r means it was resubmitted. So something went wrong and it's doing it again. The lowercase r means it's actually running. So it's running basically for the second time is what it's telling you here. It's what? Still running, yes. We get down here to the QW. This is where your job will be when they're waiting. That means it's queued and waiting. The Q and W means queued and waiting. Down here you'll see another strange one. HRQ, that means it's queued. It's been resubmitted and it's on hold. I don't know if they did something themselves or if we did something for that person right at that point. Yeah, there are times we'll put jobs on hold. We'll suspend jobs saying, hey you can't do that. We just kind of keep an eye on things and make sure that you're asking for roughly what you are. We don't have any, there are queuing systems out there as a matter of fact we can enforce this on ours that you can say, I want eight gigs of RAM. And if you use eight gigs in one byte it'll shut your job off and say you're done. You ask for more than what you want to do. We don't do that here. We kind of go on the honor system of you said this is not what you're going to use. We're going to trust that you're using about that. But if you don't then we're going to shut you off. And another one, this one is also put on hold for some reason or other. But most everything you see is going to be either QW or R. That's queuing and waiting or running. Yes, you have to make good estimates. I didn't have, we could. I don't have that on my thing but we ask us here, we can go through it. Yeah. I've been working to ask for way too much memory. Give another question? More or less. Yeah. Why in large that's the order they're going to be in. So the closer you are to the top of the QW the closer you are to getting your job done. Get the job ID what? Get the command line. When you submit your job it'll tell you the job ID. You can get it from here. This is, that's what this column is with the job ID. If you want to see what jobs are, what jobs of yours are submitted, so what you're saying. Yes, if you do the QStat command, again using GRIP, which is just saying just show me the output. Let's look at the current queue right now. Okay, I'm just looking here. Jeff Cumber has a bunch of stuff here. So we're going to look at just his job, right? Let's say QStat when you use the pipe then take the output of QStat and run it through the next program. Just GRIP, that'll show me just his jobs. Does that make sense? Are you Sophie? I guess from the accent. I knew that Sophie was in the class. Do you have any jobs in the queue right now? You can grab it from here. See what I did out here? So I just used QStat and then I grabbed your username. And those are your two jobs. There'll be one line is 185.11.80. There's a little 185.12.21. You can use Auk or some other command line utility to say grab that as of right now. Monitor node. Here's something nice too. But it won't let me do it even though I'm admin. We'll see if it works or not. Nope, I don't have any jobs running. This is a nice thing. This will let you log into a machine that you already have a job running on. So let me just second here. I think I still have that in my... I'm basically making a job here that's just going to sit there and do nothing for an hour. Should start running pretty quick, I would think. Oh, I see what you're saying. Make it as wide as I can. If the small's it'll let me go. Oh, come on. You should be running by now. I'm not using much memory. The three things you're going to need to know when you submit a job are how many cores you need. It defaults to one. How much time you need. It defaults to one hour. And how much RAM you need, which it defaults to one gig per core. The biggest thing we get is people who don't... who submit jobs with how much RAM they need total. Is it how much they need per core on multi-core jobs? So let's say I need 16 cores and 80 gigs of RAM. That's five gigs per core. So when you do your QSub, you're going to tell it you need five gigs per core. Because if I tell it 16 cores and 80 gigs per core, that's going to be more than we have available. So that's the biggest trick is, especially for big jobs, is you have to specify the amount of RAM per core. Thank you. The limit for what? The RAM and any core? Well, there's not a limit per core. You could ask, if you wanted to mage older yourself, you could have more than a terabyte of RAM. It would do that. So, but I'm saying, when you specify what your memory requirements are within QSub, you're telling it how much RAM per core. So you take your total RAM divided by the number of cores that you're using. Must be busy. Well, I'm waiting for this. I'm going to show you. We showed you how to do a QSub on the command line from this page here. But the better way of doing that is create actually submit script. And I have an example. That's true. But I don't now. That works too. There we are. It's already running. It's running in L51. Okay. This is your question now. How to get on there? Now that I have a job running on L51, it's capital M for monitor node. L51 should come on. Come on. It's waiting for my job to get past the sleep command before it lets me in. I said it was going to be answered your question. Apparently your answer takes a while. I don't think so. I'm proud to put that in there. Somebody could add that to the wiki. Just saying. It is a wiki. So we can turn it back if you screw it up. But this is supposed to be the fast part. The maximum CPUs. If you have a well-written MPI job that will do it, the whole thing is about 2,500. I guess we do have a limit of about 500? 500, of course. Yes. We have MPI spread available. So I've done some work with it on the pelleters. And the infinite air on the pelleters is much more stable. We still have to fix a few issues with the pinnacle. It should take a couple weeks. If you had it fresh on you, we had issues with the few logic and pinnacle on somebody else's node. But again, for me, there are people with MPI code, both in college and online. So I would, if you're doing MPI code, I would try to come in college and do that work. But again, it's not going to work very well for some of you. It's only giving us about one or two to give you this for a second for one, but two, seven. Yeah. And if you do MPI within a node, that will work fine. It's just between nodes. Which is the MPI spread, which is what she was asking was between them. Single connection, please. We'll send out another video. So are you working with the Rue or the Crystal? Do you have one? Okay. So I'm working with them to get it optimized. And so as soon as we get the MPI stuff worked out, then I'll do more testing. Okay. Now the monitor node is actually working. I eventually got me logged in to L51. So now I can use htop and see how things are going, or top, or PS, or that kind of thing. That's the purpose of the monitor node command is to get you into a node that you're currently running on to see how it's doing. Running, John. So here I am. I have my job. And I'm done with it. Make sure I got my numbers right so I don't kill somebody else's job. There. I just deleted my job with QDale. You can also use the QUltra command to change your, if you, before your job starts, if you said, oh shoot, I submitted that with six hours and I really needed 60, I did that wrong. You can go in and change that from before it starts. The very good reason why we don't let you do that after it starts, that's because it's really easy to game the system. If we let you say, oh, I need an hour. Oh, sure, we can schedule it anywhere. No, I was just kidding. I need 400 hours. No, we don't let you do that. So that's why we let you do that before the job starts is the QUltra command. Anything, oh, QSub directives. I have in my BioCAD intro and you'll have this on the slides that I'm going to send to you. Everything I have in here is in this BioCAD intro directly on my folder, which is publicly accessible. Anybody can get to it. But I have some QSub scripts. By putting things in a QSub script, as opposed to typing out the QSub command every time, that keeps you from having to A, remembering all the parameters that you used, and B, it's a whole lot easier to go back and change things if you need to change anything. So here I have it. It's a simple QSub script, blah, blah, blah. Because the directives that QSub sees start with dollar sign or hashtag dollar sign, then I use the double line here. So all these are comments that start with double hashtag. And so if I wanted to change this, all I have to do is delete one of those and then it starts with hashtag dollar sign. And that's what anything that starts with that is a directive toward QSub. So this is how I tell it. I tell it I want how much memory. This is all well documented. Feel free to copy this over to your own home directory and edit it, use it however you want. That's the whole reason why I wrote it is so we wouldn't have to reinvent the wheel. Runtime, this is hours, minutes, seconds. Or you can put the whole thing in seconds if you want. Most people don't want to have mess with thinking about how many seconds is in 300 hours or something. Infiniband, I did put a note in there that if you don't know if you need it, you don't need it most likely. We had a lot of people for a while, so we're looking at our examples and saying, Infiniband, that sounds fast. And they were asking for Infiniband and that was actually keeping their jobs from running on somewhere the way it would run just fine. Same thing with CUDA. CUDA is the GPU processing stuff. So if you don't need it, don't ask for it. Parallel environment. And here's a bunch of things you'll likely see. Single one is the default. It means you're using one core. A single, you're using 12 cores on the same machine. MPI, there's several different MPI options. Again, you only do this if you're going to be using MPI code, otherwise it's going to blow up horribly on you. MPI spread means to spread it out over as many machines as you can. MPI fill means I'm trying to put it on as few machines as I can. And MPI, we have one, two, four, eight, 80, 16, something like that. We have several of these documented. But that means you're going to allocate this many at a time. So if I had MPI four, 16, that means I'm going to try to run... It's going to allocate four at a time. So four on this machine, four on this machine, four on this machine, and maybe four back on the first machine again. That's going to allocate them in groups of four. Checkpointing, I'm going to skip because we're probably not going to do much of that. Some handy things to put in here. Again, you can put these all in the Q sub command line, but if you're doing a lot of these, put in a sub-script. It's a whole lot easier. CW means change current working directory. By default, all your output files and everything it looks for is going to be looking in your... Is right in your home directory, where you are in your first log in. Most times, if you've been using BioCat for a while, you're going to have your projects in different directories, that type of thing. So this lets you run it all from the directory that you're in. At the time you submit it, that's a handy option to have. So all your output files and everywhere it looks for data files are the same. Merge output and error text streams into a single stream. Sometimes you have programs that are expecting, for some strange reason, they try to put their output on error lines. That kind of merges those together. Sometimes it's handy, sometimes it's not, but so that is an option. Name your job. That's a really handy thing to have, especially if you're running lots of different jobs. I know some people submit several different jobs and they'll be N47A2, N47A3, N47A4, you know, on so on and so forth. By doing this, you don't just see, run my job being executable. So that way you can see when you do a QStat, you can see what the job title is. It'll also change the name of your output files, so that it'll be easier to read and easier to tell what's what. And then finally, when you run the program that we're meant to do, especially if you're running executable, so submit script is really a good way of going about things about running your programs. It's almost, is it required? Probably. For R, for Perl, for any scripting language, Python, that kind of thing, you just about have to put it into a scripting, into a submit script, and then you submit this. So here's my sample that I have here, and then if I was wanting to submit that, I'd just say qsub sample.qsub. If you look at our advanced SGE page, it details all this too. So this is all documented. That was what I had for SGE. Again, if you go to our web pages, you can find a lot more complex examples, array jobs, things like that. I think you're going to talk about that in the next... Okay, that type of thing. Some strange things, variable number of quarters. You can do all sorts of strange things. Most everything is documented on our Wiki under the advanced page. I tried to make the beginning page to be really as simple as possible, just trying to get your feet wet and figure out how this thing works with the advanced page being now that you've figured out how it works, here's how to get the most out of it. Questions, comments, or snide your marks? None of the latter even. Okay.