 All right, we're gonna go ahead and get started then Hi everybody My name is Kyle. I'm one of the system administrators when you send up emails you likely get emails from me or one of these other guys here Just a quick introductions here of who we are and and what we do here. I'm Kyle I'm a system administrator Seniors that are looking for projects So I will say that I'm Dan Andreessen for those of you listening over there I'm teaching a class in a called CIS 625 which has about 60 computer science seniors looking for team projects to parallelize And otherwise make code run fast. So if you get into this and realize hey, I would mind some fairly experienced help Shoot me an email. We'll see if we've got a project or your see if your project would work for them But if you've got something where you've got our code Matlab code Other analysis is just taking too long. Let me know Okay So We're gonna start just kind of at the beginning We just talk do a little talking first before we actually get into any demonstrations and again anybody in in in zoom Don't talk because we can't hear you. Yeah, I put it in chat and we'll see you up on the screen Matter of fact, let me go ahead and join You got your join there so you can look at chat very good just in case all right, so We talk about bao cat bao cat is a what we call a high performance computing cluster or HPC What are we talking about when we're talking HPC? What does that mean? This is where I ask the question you get to answer. It's it's kind of cool that way What is it what what is it? What is it? Well, how is it as an HPC cluster different than laptops that much you guys have in front of you More processing power exactly what about How what kind of limitations do you have on your laptop, okay? How many cores do you have in your laptop? Okay Okay, six quarters What's our smallest one we have in bao cat Dave what's our smallest smallest number of cores we have on bao cat I Believe it's 16. Yeah, I was gonna think the old elves. I don't think we had anything smaller than that and But we also have multiple nodes in there, too so you can run jobs with multiple nodes and That makes it obviously if you if your if your machines are made to use those kinds of Those those programs to use multiple cores. It's gonna make it run a lot faster We have over 8,000 cores available on bao cat So quite a bit different scale than the six some of the bigger laptops might have eight cores on it You know 80 on a single machine or 8,000 in all of bao cat now a High-performance computing generally speaking is a lot of computers working together for parallelism that being said Well over the half the code that we run on bao cat is not parallel code It is running the same kind of things you'll run on a laptop It's just that instead of doing one of those processes people actually there are people even do one of those processes just because they Don't want to run on their laptop and leave their laptop on for three days straight So they'll kick it out to us to bao cat and let us run it for three days straight because then they don't have to worry about it That's fine, too But probably most of what we get over half I won't say the majority won't say you know the vast majority, but probably somewhere over half of what we run on bao cat is Actually a single core Running and it's usually doing people were doing parameter sweeps those kinds of things where yes I'm going I'm doing this one thing, but I'm doing it a hundred times in different ways You know changing my input parameters just a little bit run it again change my input parameters a little bit do it again they batch those together and You know they can run a couple hundred thousand of these and we can run them, you know several iterations of that at once Probably nearly the other 50% of our cluster is multiple cores on a single node So that's what we see the people buying into bao cat people doing a lot of work on bao cat tend to be a lot of this is It's stuff that could run on your laptop Using it six cores or sometimes two cores in my case that kind of thing. It's it's not using a whole lot of cores Some of these might scale up a lot bigger might be able to use 32 cores at once 16 cores at once So they'll even run it faster than your than your normal desktop or laptop could And we have a very small percentage as well under 5% that are running multiple node Multiple core jobs bao cat is meant to do that But just in practical usage. That's what we see is that is a very small percentage And that is the ones that are you know branching out running several hundred cores at a time with machines talking to each other And that kind of thing those are the kind of things they're working on in that class that Dan was talking about the CS6 25 So if you're wanting to scale up and out and that kind of thing That's really a good place to go if you're if you're writing your own code to do that now if you're running somebody else's code Which probably 90% of people don't write their own code on bao cat most people are running a a Package that's already exists and they're just running running with their own data on there. That's fine, too We're we're not picky. We're we're happy to help you out regardless where you're at so Try to get you going has started to begin with though and of course my laptop went to sleep over here give my notes on it Parallel and serial calculations of parallels when you do obviously do things more than one at one time serial is Running one process just running straight through Running your computer running a program on the high-performance computing cluster is not going to inherently make your stuff faster If you're running for instance are Most Python most Pearl unless unless they specifically gone out of their way to make it this way Most programs that you run will only run on one core. So don't log into bao cat and we can help you out with this If you if you want if you don't know but a lot of those programs it doesn't matter you you ask for 30 cores and It'll allocate 30 cores to you and all your program will be running on one core because you haven't told it how to You have to specifically tell it how to do that That's beyond the scope of what we're doing here today We've gone through some of those trainings if you want to look at our videos some of Dave's videos if you look at like hour For other previous ones Dave kind of goes into some of those details, but that's even that just barely scratches the surface So if you're wanting that level of help Probably talk to Dave or talk to Dan with his class I mean that he that he's working on because they're that's beyond what we're going to get to here today. So Just be matter of fact a lot of laptops, especially like the brand new laptops Their cores on that will be faster than our cores on bao cat So if you're just looking at one single thing to run through and it might actually go through faster You know being on your own laptop that will be here now The advantage of that by running it with bao cat is obviously then you can take your laptop home with you Shut it down those kinds of things You you can use your use your laptop or desktop for other things at the same time. So it's not chewing up those those cycles So I'm not saying that there's no value, but just so you know We have people come to us really frequently and say hey this thing takes four hours on my laptop But it's taking five six hours on bao cat Yeah, that that that very well could be and that might just be the nature of the beast and That's the way it works now if you run it on some of our faster machines You might get so might get might run faster But it might have a longer waiting time to get onto those machines, too So six and one half dozen together so Even though we have you say about 8,000 cores or or so There's still always more demand than supply so unlike when you're on your Your own laptop your own desktop When you are on your laptop right now, and you say go run this program it starts running it right away That's what we're used to However, because we're a limited resource and even say even though we have a whole lot What you have to do through bao cat is we have to run through what's called a scheduler and a scheduler is a You can think of it. It's a little more complicated than this in real life But you can think of it as like a giant spreadsheet in in in memory where it says oh You need four cores for four days and 16 gigs of RAM I can put I can fit you in there in that Tuesday at 4 p.m. And then This guy's the other person whose job was running on the cluster It quits it didn't use all of it. They asked for a week You know they ran for four days and then it was happy so it might back moved it back and says Oh, I don't no longer Tuesday at 4 p.m. Now I can move you back to Sunday at midnight and Then I know dies so it has to take all the jobs that were on there And now now yours doesn't get a Sunday at midnight now pushed you back even further to Thursday at noon It's doing this juggling all the time trying to figure out how to fit things the best that it can now So we we run that through the scheduler and that's how you submit jobs To be a cat is through the schedule. I'm just keeping my laptop alive if you see me walking back and forth there Just so I don't have to log back into it again And that's what the schedule does we use a schedule called slurm and we're gonna go through some slurm stuff here today Of what to do to get your jobs running on bio cat Questions at this point Anybody in zoom questions. No, okay good So first thing is Everybody here has an account, right? I don't need to worry about getting in accounts Everything we have we do pretty much. You don't have one. Okay, you don't have one either Okay, our support pages will tell you this support.baokat.ksu.edu Is where we keep all our documentation for everything bookmark that page and it's a wiki So if you see something there don't I think the only thing that we don't let people change or like our policies for somewhat reason We don't want you guys going and saying that we're gonna do, you know, more than we're we need to and hey We you we said we just changed it. So you said we'll we'll take fruit of data now But no, we're not doing that. So our policies aren't changing everything else on there is a wiki You guys can log in if you see something that we need to document better feel free log in change it There's a log in button up in the corner and that just uses your EID credentials. So Log in change stuff if you see something we could do better Be happy to we're happy to have those kind of contributions now We do watch it so make people don't be face it but that's has never been an issue. So our people are good We like we like that so To get an account and it's listed on that page It's like the first thing on there is how to get an account and that is to go to account.baocat.ksu.edu We have several web pages set up and let's It'll look like this and let me go ahead and share my screen here through zoom Ah somebody else even posted the link nice job. Thank you guys. Oh, that was you. All right Still good job, Adam. Where's my little thing? There we are Screen At least you guys can see what I'm doing here, too. So I go to this It's going to Ask for a bunch of information what you want and Actually, two of these things don't even no longer even exist. I don't want to even matters right now as a baocat So whatever you select there you always get baocat. That's all We haven't bothered changing those pages ask some This it already knows about for me your institution your Department that kind of thing and some demographic information that helps Helps us to keep track of things for grants and that kind of thing So then we're going to talk about how to connect to baocat now baocat uses What's called SSH and it throws people off a lot at first because People are used to typing on their nice pretty screens and all this and SSH is all just plain old text remember the old We have some from front ends that'll help with that the one we'd recommend for Windows is called MOBA X term you can see it there on this screen If you just go to just Google MOBA MOBA X term it'll it'll take you right there to the download page downloaded It's it's a free download There are some other ones that work out alright if you're on a Mac you already have a Terminal built-in A lot of people use putty if you use that in the office that'll get you there the same way MOBA X term has some nice drag-and-drop features that putty doesn't have so that's why we prefer using that So SSH is Wow some I must have already logged into the screen before because already tells where Already has my stuff on the screen. I'm kind of surprised at this So But we have to tell it how to connect to there so what I'm going to do on MOBA X term I'm going to delete this one just so that we can see the process from the beginning Delete Yes, okay, so now To do this in MOBA X term. It's gonna be a very similar process in whatever else you do if you're on a Mac It's like you only want to see over the Mac except for me see I like my Mac too, but You'd use to SSH space username at and Dave to thank you Others another yeah, they just got there covered up with stickers. I'm just seeing I'm like I'm my vision is apparently really short in front of me SSH space user name at head node.baocat.ksu.edu Little bit different process here on on the windows side of things. We're going to create a new session I'm going to show you get MOBA X term in particular because it doesn't like that doesn't cool stuff Does SSH For a moat host this is we're going to do the same thing here And if I mistype anything Adam tell me because I'm facing away from the computer Did I really you know, I'm messing with me. I really don't really did There you are head node.baocat.ksu.edu probably be a guy did a good specify your username here And I'm going to use my EID here and we're done. It's pretty easy to this point And I say okay, and now it's going to ask for my password now Unlike most Windows programs. It doesn't show anything when When you type your password in here and I need to look up my password Because I normally use a password manager and I don't have it on this computer come on I put in my password and No indication that anything is on the screen Asked if I want to save it now if this is your own laptop and you've got pretty firm control on it You might do that. I'm not going to here and we get a screen now That has the baocat logo a lot of our policies what what we can do here what we can't and We have files and folders over here on the side. So I'm going to make this big and Adam what's a magic keystroke since you always seem to remember these things on everything I was gonna say you I always ask him because it's easier than looking up myself. I'll eventually get there So you'll see a screen here that just has username and So this says at Celine Celine is one of our head nodes. We have two head nodes. So yours might say Celine yours might say EOS, right? Anybody get EOS on theirs? Or does anybody logged in? Not even logged in you got you got Celine Yeah, oh Yes, the first time you go log into a machine it'll ask if you want to if you want to continue connecting You just say yes Sorry about that. I forget that because I've done this on most of the things I've done. I'm already The process is very similar if you're using Putty on Windows That's your SSH connection Again open SSH if you're using Linux or Mac is all command line driven. Yes, SSH Should ball be lowercase that might be one issue Yeah, you everything should be lowercase for your and you can't click on it You got a back back over it like say that there's no mouse. I get you there Okay, good deal. Oh and so okay I see something else somebody else's screen. I forget what I've special what I've done with mine and what What are the people what the defaults are? So I'm assuming you haven't logged in before And and his screen shows I'll put I'll type it up here. It says like bash dash for 4.25 4.2 Anyway, anytime you see that the dollar sign at the end That means it's ready for you to type something there So that means it's already everything's doing what it's supposed to be doing by having the dollar sign That's your prompt that you can type what you want on there now Doesn't do us much good having this thing out here if we can't Don't have any files Right. So you want to be able to transfer files back and forth The way we do this on a Mac is one of the there's a program actually works for Mac and and Windows 2 called file zilla if you're if you're running Mobile X term you don't need this you don't need that for Windows because what I can do then is I can grab files here Let's see if I have files out in the Network anywhere now What do I have? Have my you drive. I think I have files out there dying. Oh, yeah, I got stuff here Here's a an old version the bit intro to be a cat 2014 edition Slides that made up for that. So I can just take that and I can drag it over here and now We look down here that file has been transferred over to be a cat file zilla works much the same way That's probably the recommended graphic version on On Mac and it actually even has a has a Linux client to I Like mobile X term for Windows because it does all this in one program So you don't have to flip back and forth between putty and file zilla and different programs to make things all work It's all just right there. I Can do the same thing other way other way to Pull up this here this file 10 dot text. I can take and Download that Onto the desktop sure There it is. There's my file 10 showed back up here So that's file getting files in and out into and out of bio cat now Editing files on yes, go ahead on Mac Yeah, yeah, you drag and drop same thing. Oh You have to you have to use a program to copy things back and forth So you have to go go to the internet and download file zilla Right on on a Mac. Yeah mobile X term doesn't work on Mac. So we got to use file zilla on a Mac Yeah, did you file zilla download and it'll ask for the the client yeah, so Let's say I want to edit some files on bio cat a couple ways we can do this I say everything I'm doing right now is all text-based things Um, how many of you guys do any have you guys done any programming at all? Some okay, how many of you guys who did programming would write a program in notepad? nobody a full program in notepad You're a brave guy So I'm gonna show you the first version the first thing I'm going to show you here is Call is a very simple file editor. It's called nano and that's kind of the window the Linux Correspondent to notepad If you're going to do anything serious if you want to edit, you know a quick file It's fine. If you want to do anything serious. There are other tools for that. We'll we'll we'll talk about that here in a bit So I'm going to use nano Nano is the name of our is the name of a file editor and then I just had a file that file 10 dot text So you notice what I did there. I started typing fi and Then I hit tab and there's no other program know that no other files in that directory By the way, I say I say this a lot. I say the word directories everybody know what a directory is a Directory is Linux's name for a folder They they used to call on that on back in the old DOS days and went to see the folders are a lot more More easy to think of people don't so I used I still use the word directory because I've been doing this forever But it's so if I say directory folder really the same thing You can use those words kind of interchangeably So I have no other files in this folder called the start with Fi So I hit Fi Fi and then I hit tab and it went ahead and completed the rest of that for me So I didn't have to type out the whole the whole name of the file so I do that and I hit enter and Red zero lines must be an empty file. Okay, so I'm gonna put something here Blah blah blah and I can't even do that right Okay, here's my story and I was taken to it I Wacked off the very end of my fingernail the other couple days ago And I've got a big old band-aid on the end of my fingers. So that's my typing excuse for today By about two weeks from now. I'll have to find a whole new excuse. So So I just put some text in the file Like I said, if I'm just if I'm just writing something real quick like some of our submit scripts We could do it this way. That wouldn't be so bad All these down at the bottom Those little carrot things that's saying control. So I'm going to do control X To quit to exit It asked if I want to save and I say yes Ask the file name. They already had file 10 dot text. So there we are now I want to look at what's inside that file Look what's inside of a file. There's a couple ways of doing that. We're going to use the command called less They're back in the early days. There was a program called more and More was just you know show one page of text and then to say more at the bottom And you hit space and it'd go to the next one and hit space and go to the next one Well, people got to the point where that was all well and good, but sometimes I want to go up the screen So they came out the new command is less because less is more. I didn't make that up. That's That's the way that's the history of the command computer guys have odd sense of humor so Less file 10 dot text now. We know this one's going to be really small So it's only got the one line of text on there. It just says blah blah blah and At the end so that's all that's how we can look at what's inside that file If I'm going to do any real editing I'm going to use a program called vi or VIM VIM. They're both kind of relatives of each other It has a learning curve somewhat akin to a brick wall Like the most commonly asked question on Stack Overflow is okay I got into them. How do I get back out because it is not at all intuitive Once you get used to it you will hate going back it it drives me nuts even doing nano for this little file because I'm used to using that using VIM but Like I said for for short little things it works out pretty well You know you see the little little prompts at the bottom. What I was like control X for exit. What's up? You're going to go to a new session up here SSH Head node Bayo cat dot ksu dot edu and then specify username as your eid Yes, sir. Oh just now, okay Yes, sir. It's gonna be in bin Yeah, should be rentable. Did you bring in Linux here? Hmm. I've never I've never run files on Linux. I know that it exists, but I've never actually tried it there We'll play with that He's got a file zilla for Linux. Yeah, which I've never messed with before and it's not running. It's like so using Linux itself Again, we're gonna go back to our support pages here and I we have a page on here called Linux basics again Apparently, we're not doing a real good job With our wiki because I'm even knowing what I'm looking for This this is kind of a cheat sheet on how to file transfer these files in and out and and some Points just some tutorials here too. So Basic Linux commands. We're gonna kind of go through some of these things If you're wanting to transfer files from Linux or from a Mac through the through the command line interface Right above here. It talks about using SFTP. That's the way to do that I'm not gonna go through this for go through that for everybody because most people are using the GUI But that that that'll show you how to get files in and out over there So when we talk about things here a directory already said that that is a folder a Shell that is that's the prompt that we saw on there. Well, it has a dollar sign at the end that that's how we That is an Environment under which you can run commands. I know that sounds kind of strange Every every machine has a shell just most of them kind of hide it away from you so That's This the shell for most everybody here probably will be bash and Bash is bash is kind of what we assume for pretty much everything we're doing here SSH SSH is both a command and a protocol so We talk about getting into something remotely we're using SSH note that it does not have Any graphical capabilities built into SSH and that's what we're gonna do most of our stuff on and be a cat So it's something you gotta gotta get used to it's a little bit different than then most people day-to-day work process SCP that's a secure copy. That's a it was very related to SSH. It's how we get files in and out and matter of fact Mobile X-Term file Zilla all those programs. They're actually using SSH the SCP protocol To get things in and out of they're just kind of hiding it away from you. So you don't have to remember the specifics a path now Let's go back to Going back to mobile X-Term right now When I first log in it puts me in what's called my home directory. That is my spot going way back to the olden days when Unix was first created back in the 1960s Space was a little premium. So we see a lot of things that are abbreviated very small. So To look at what our my path is right now, I'm going to use the command print working directory, which is PWD Again very much abbreviated and if we look at that, we'll see right now. My directory that I'm in is homes Kyle Hudson everything we start Everybody's directory will be homes and then your EID Now, let's say I want to go somewhere else. I have a Folder in here and I'm going to do a file listing listing of all the files here Would help if I type it on the right keyboard I know notice there is that file 10 that I just changed. There is the intro to be a cat that I Just uploaded. I'm going to go ahead and remove that and I'm going to do that with the RM command again Old commands they made in the smallest possible that minimize the amount of type of people had to do or sometimes even punch cards so RM I'm going to start typing intro actually. I only need to get one character in there's not no other file to start with I I'm going to hit the tab You notice how it puts the little slashes before spaces That's because Linux and spaces don't really get along they'll work But if you're going to use them in a command line interface, they can kind of be a pain in the butt So I suggest you avoid them if you can Sometimes you'll get programs in that already have it if you do don't worry about it. It'll be fine But I'm going to move this command I hit I I hit tab and now that files there I hit enter And that's what I did is I removed the command Remove that file. So now if I do my file listing LS That one's not there anymore. Now. I have my file 10 text My saved stuff folder and my shared files folder Um, how do I know those are folders? Instead of knowing their files What's that? The color of the text is being hit somebody just put something to chat Adam. You want to check and see what it was? If I there if you happen to be on a terminal that's not color-enabled I can do LS dash L That's a long listing How do I move multiple files that are similar like slurred out gotcha And I have a few of those I will I will demonstrate that So I just did an LS dash L and that gave me what's called a long listing now if you look Right over here on the left side of the name. This is my Sorry I got caught up on here, too So this is telling me all sorts of information about the file This is the permissions This is the owner This is the group The size which is getting truncated over here and the name. So that's the information is telling me Yeah, that works. Oh And the date that's what I'm sorry. Yeah, that's the size. This is the date and the time of that file When I create files it goes in my default My default owner is going to be me. So if I was great a file from scratch It's gonna be me because I just copied this file over the file 10 and my own group, which is Kyle heads and users There's nobody else in your default in your group besides you If you have group projects There's a whole section on website about about using group projects and getting permissions right on that But what you'll see here then is this These three these three these three Are all Telling your permissions of the file the first one's going to tell me the first this first dash here That just means it's a regular file down here. This is a d that means it's a directory So if I if I didn't have the color on there, I could still tell That this is a directory So you don't have to memorize what colors or what I you know Which I generally don't I will generally I know what stuff I've gotten my folder So I don't worry about a whole lot, but if I don't know This is what I'll usually do is I'll just do a l s dash l and I'll look it for that d that d means as a directory So this first three characters, this would be either an r or a dash a w or dash or an x or a dash That stands for read write and execute The next three will be the same thing the next three be the same thing This is going to tell me for the user so me Kyle Hudson I've got read and write access to that file I don't have execute file access to that file That's okay. That's not an executable file. Anyway, so it doesn't matter People in my group the Kyle Hudson's users group or for those of you Are in some lab groups that already have these those will be when you go to when you go into your professor's folders to the right spot This won't be somebody users this might be I know Jesse Poland has some students here so KSU plant path j j poland Will be in this will be this with the be this group and that will have some other different permissions So anybody who haven't been in that group would have read but not write and not execute And then the last one is everybody else in the world So everybody in the world has read no write And no execute That's probably good. I probably don't want the entire world to have right permission to my stuff, right? by default everything on bio cat has a Read permission for the world enabled. Yes It'll be an x Yeah, yeah, this is this one execute mean in a directory means that I can I can see what's inside there So read write and execute That's going to be a little bit more important when we have when we're actually starting to run programs here so that gets into We're talking about permissions there. Yes la la is is all so if you noticed On this side over here. These are this is also a directory listing and I've got lots of files That start with a dot by default those are hidden Generally, there's generally speaking any file that starts the dot is Something used by a particular program like for instance the first one I have there is dot ansible so that's an Information about ansible is a program that I run. It doesn't need to know. I don't care what's inside there It's kind of ansible taken care of its own stuff I'm not going to want to see that Generally speaking when I'm doing this it shows it to me on mobile x term But if I do ls space dash la And I'm going to pipe that through less Because it's going to be a huge listing But the first thing up there now you'll see that Matter of fact, it didn't even my screen's too narrow. It didn't even like that But you can see all those dot files. That's what the difference is between the l and the dash la It's showing you all the files including the ones That are hidden And you see I have a lot of those good question now Let's say That I have a file here Let's say I wanted to make that file 10 Uh executable It's not I can't actually really do anything to make it executable. Let's say I wanted to make it executable We use that we use the ch mod command again. These are all on that quick tutorials that I have on on the the bio cap page So ch m o d I'm going to add execute something plus x file 10 dot text And it just it doesn't even give me any feedback. It didn't tell me it did anything, but if I do the l s dash l now First of all turn to green telling me that it's executable And secondly you can see that r w x as those first three letters on there So that means it's now executable now. Let's say this file has something Important in there that I don't want the rest of the world to see this is probably going to be a more common thing So I can say ch mod again This is change mode the mode of the file is what is what we're looking at here And I can change this for u for user g for group o for other So I only want myself to have access to this file So I'm going to say User I don't want to change my permissions Group and other I do want to change someone say go for group and other I don't want them to have read access and I don't want them to have executable access so I'm going to say minus r x file 10 dot text Again, this is this is on the website. I'm giving you kind of a brief demo because there's no way you're all going to wrap your heads around this in a short period of time It does get to be useful in one particular instance and I'm going to talk about that here in just a minute But now you can see my permissions on that file Are I have read write execute? My group has no access And the rest of the world has no access. So now you guys could not go in and copy that file from me Before I did that you guys could That is an intentional design thing for for linux systems for high performance computer systems in particular in In general Is having Default is people want to share their data. We want to be able to copy data from person to person So make sure you turn that Read access off for others. If you have something private that you don't want anybody else getting into What else we got to go here? Oh, yeah, yeah So i'm going to create A bunch of file a matter of fact I have in this shared folder I did this for a different purpose, but it's going to work out really well So here I have ls And you can see I have these files. I created these just for this demonstration. I file one dot text file two dot text Just plain out there um So let's say I want to delete all those We can use what's called a wild card, which is a star so I can say r m space File star dot text And what that's going to do is that's going to say if there is Anything that starts the word file and ends with dot text. I'm going to delete it If I did rm file star, what would that take care? What would that delete? equity Everything starts with the word file, which would be everything that also ends in file dot text So that's actually dot text is actually a subset of file in this particular case. That's all my files Anyway, I normally try to think of things and try to limit things in my mind if I especially when I'm removing files Think of all the ramifications Could I do rm Could I do that actually where I am right now that'd be okay Generally kind of a scary thing to do. I would advise against trying that Yeah, so I try to think of it was like I I know all these files start with the word file and end with dot text I know I'm completely safe doing that. So now when I hit rm file star dot text now If I do my ls There's nothing left in there in that directory because they all started with file Yes, that's true. I've turned I've turned mine off But but yes by default if I do rm file star dot text It'll ask me do you want to leave file one dot text you want to leave file two dot text You want to leave file three and it'll ask you for everyone that goes through there um There are ways to force that like so and I just turned mine all the way off. So mine doesn't do that but I've been doing this a while. So you didn't see me doing the file star even though that would have worked Any other questions Do you want to go ahead and go on the tour now since I have to leave a little early at the end? Does that work? Has anybody not seen baocat yet? Let's go ahead and get and let's just go on a tour down there We can show you what kind of what the stuff we're talking about here is and then we'll come back if you're not going to go on that Take a break here Get back about three o'clock or so would that work? Dave would be the one to take all your stuff Sorry Only the apples okay good We're actually going straight down about two flights of stairs So You're making control You need to join those in the session Yeah Yep There's a button We assume you can choose which screens to share which which applications or just to share the entire desktop Your microphone there Very good To this one You You need to mute your system entire Okay, I turned my I unmuted mine Can you flip it over to mine my video and audio and see What Yeah, I think yours is what's yeah yours is what's showing up on the screen now and if you talk Test yep, okay, and it even pulls up your camera there. So And so is that what's going on is zoom showing mine then Okay, I would have yeah, I have to share a screen And I can't because Kyle is But that's fine though So I think we got it taken care of whatever you did You have a couple people have to take a bathroom break for you people out in zoom, but we'll be back here in just a couple minutes It's I Think most everybody's back here. So let's go ahead and get started and anybody's not here is gonna have to catch up so Talked about using the scheduler and how we're gonna make make things work on on the scheduler So first thing we have to do here. I'm gonna change back to my Normal directory and I didn't even I kind of skipped over this when I was talking about what directory you're in I said pwd for working directory cd has changed directory You can see I did that about five lines up on this 455 lines up on the screen says cd space shared files That's changing the directory to my shared direct shared files directory Now I want to go back to my main directory So I'm going to say cd Enter and my keyboard is apparently not My mouse doing anything Yes, my mouse is my keyboard is not hold on Touch the keyboard. It's out of battery or what? I brought up the calculator. That's great It's what I was wanting Not press alt No, that's not what I want to do Almost like a key guts There we are Had a key got stuck a control key got stuck. Okay So I'm going to type cd enter. That's change directory um here I can go Nice thing to know is dot dot means up a directory. So now pwd I'm back there if I was back in my shared files again Another thing I can always do is I can do cd and nothing behind it That always takes you back to my home directory In this case those both happen to be the same thing But I can be in my bulk directory. Everybody by default has a homes directory. I'm right now in homes. Kyle Hudson there's also a bulk Kyle Hudson and I have files sitting out there too So from here if I do cd dot dot it's going to bring me up one Now I'm just in bulk Or if I do cd enter that's going to take me back to my home directory and the home directory is also I denote it by the the tilde which is a little squiggle you see over there on on the screen So is that to Adam is that the default to show the directory you're in or is that something I did? I don't remember That's default. Okay so First thing I'm going to do here then I'm going to turn myself around here so I can see the screen and still hope have the keyboard work. I hope um I'm going to run a simple very simple command and We're going to see how it behaves and then we're going to run it through the scheduler and see how that changes things so I'm going to run the hostname command When I run the hostname command, that's a built-in system command On linux system and all it does is it tells you what the name of the host is you're on I can I knew what the answer was going to be because it says Kyle Hudson at saline And so therefore I knew my machine name was already saline. That's that's the head node. I'm logged into right now so The first thing I want to do Because a scheduler can't run a program directly A program a scheduler has to run a script A script is a set of commands that Just runs one right after the other after the other So I'm going to create a script And I showed you I was going to use a nano earlier. I'm going to use vi because that's the one I like to use hostname.sh Now in a script The first thing first line always has to be We call this shebang a hash An exclamation point And then the path to the program been bash in my case You can think of that as just magic that you have to put there. There's there's no That that that is something that's been there since the beginning and I I don't even want to try to explain what the whole history is behind it Just know if you're going to start a script start it with the shebang and then and then the script the the The shell you're running on been bash is going to be 99% of what anybody does probably a hundred percent of what everybody in this room does So do been bash now. I want to run that hostname command. So I do Would help if I hostname Now this is an executable within my script. So this is a script that will run now, right? So i'm going to save and quit this And now I have hostname.sh In nano, it's control x and then yes in vi you're going to hit press escape Out of your insert mode now you press colon get to command mode and I hit type in w q right quit Again, like I said, vim is not user friendly to begin with I really like it, but it is not user friendly There are cheat sheets for it. They help a lot, but just getting getting used to it helps a lot too. Yeah So now I have this hostname.sh Can I run this right now? Why not? I saw a head shake. Why not? Because I need to make it make it executable, right? So i'm going to use my chmod plus x Hostname.sh Now let's look at it. Ah now it's green So now I can run it If I do hostname.sh Right now will that run it? answer No, it doesn't know about it and that is because my path The path is what tells you what? Where where it's going to look for executable files? So if I do echo path That's going to tell me I'm going to look in these locations I set mine up to have my own bin directory Uh user local bin user bin all the system defaults at that point My current home directory is not in there So it doesn't know that I can it's even executable. So it's not even a search to try to go there I can force that by saying run this from the current directory So my full path to this Is there's a few ways I can do this. Let's start with the The full path I can say homes Kyle Hudson Hostname.sh Enter it ran it did what it was supposed to do The other thing I can do I told you that the tilde is a shortcut for the home directory. So I can say tilde slash Hostname.sh and that works The other trick is to use what's in my current directory right now So if you happen to be in a sub directory, this will work this this without adding anything on Whatever directory I'm in if I'm in that directory right now, I can do dot slash dot means current directory slash Hostname.sh and that also works Okay, pretty simple command just does one thing is just running it through a script as opposed to running it through the scheduler Now I want to schedule that command. I want to run run this through the scheduler through slurm And the way I do that is with the s batch command so our an s batch dot slash hostname.sh And the s batch command will submit that to slurm And it tells me It's submitted batch job seven three eight two three eight one that happens to be the next job number They started at one and were up to what seven point three million now Jobs that would have been run on baocat in the last since we started in january of 2018 right Now It submitted that job Let's look what's in my directory now There's a file there that says slurm dash seven three eight two three eight one dot out That wasn't there before That job is created when my job starts running Now this job already ran and was done because It could fit it in really quickly Running one core a single core job Can pretty much happen now If as long as a short job and that kind of thing says oh, yeah, I can schedule that So if I look at what's in there, I didn't see any output on my screen there What happened is that went into the file. So when I look at what's in that file less slurm Dash and then the job number. You notice that was the same job number. It said it was submitted door 52 So door 52 is one of the machines that we went and saw in baocat just a little bit ago That's a place that had open space available It submitted the job slurm got on to Door 52 said slurm has a job for me. It picked it up. It ran the script put the output of the file Into that file name said i'm done slurm says, okay, you're finished So it's no longer out there I have The output of my file there. So it actually didn't run on the head node on selene It actually ran on that particular machine door 52 Now you notice what I didn't do was I didn't tell it how many resources I needed And that's something that we need to tell it the system defaults if you don't tell it anything else are one core One gig of ram and one hour It's not as common as it used to be used to be our most commonly asked question on baocat is My job ran for an hour and then quit That's because you didn't tell it how long you needed and if it goes over your time limit. It says yeah, you're done So my point here is don't underestimate how much stuff you use now. How much ram did hostname use? Yeah, yeah, so so small doesn't even matter. I mean hardly any maybe a couple maybe a k of ram. Maybe yes Yes, yeah, we're going to get there That's that's exactly what we're going to talk about is how we change those things and the easy way to do that so Um Use it did it use more than what do you have used more than one core? Even if I told it to use more than one core Would the hostname command have done that? No, it's a really small command It doesn't take long enough It wouldn't it take long to fire up two cores and it would be to just run the thing. So it's done Okay, and it is one hour plenty of time to run the hostname command. Oh, yeah, it took less than a second. I'm sure so Let's say that I do want to try using more parameters If we look at our help pages here There i'm on linux basics If I go back Your mouse Okay, I really hate this We're going to go through slurm basics here submitting your job Hey, there's the my host. That looks really familiar, doesn't it? I'm not very inventive. I use the same thing over and over again We're going to skip down here a bit detailed information about a single job Oh, come on. Where's my list of commands? Ah, there we are you notice this line Right in the middle of the page here This tells me a lot of things I can do So here Let's run up with kind of these same kind of things We're going to say s batch um, let's see dash dash time equals n by default it's in seconds No, no slurm is in minutes slurm is in minutes Dang it What's wrong? I want to run in 10 seconds. I'm gonna say zero hours zero minutes 10 seconds. That works, right? Now what if I wanted to run it in five days? I could say five minutes and zero hours 10 seconds These commands are all documented on our pages there You can tell I don't use this a whole lot because I'm having to ask questions on how do you or at least not this way anyway so Uh Let's see number of cores Yeah, member cpu 512 megabytes The fault isn't gig so if I said 512 it would put 512 gigs and only one on like a few machines in our cluster That wouldn't be good um CPUs per task just one nodes equals one dot slash hostname And I submit a new batch job And it's already done This time ran on elf 73 So I can specify all those things on the command line That sounds like a pain in the butt to do every time though, doesn't it? We don't have to do that every time So What I can do is I can set up this script since I've already got a script anyway. I can put these parameters in the script so normally When I start any command in a script with a hash That is going to say this is a comment So It won't do anything that if I just run this directly However, Slurms looks for anything that starts with the hash and then s batch I can put those same kind of things that I would put on the command line Inside the script That's really handy if I'm running this thing several times, you know different parameters that kind of thing I can I can run my script And I can change my parameters here in one spot and keep all the rest the same Yes batch, I told you my fingers messed up CPUs per what is that on there? I don't even remember anymore Mem for CPU is what I had when I was saying they get that right CPUs per task Which one now I have a script that has told slurms all these parameters That I want to use on a regular base that I want to use and now I can change this if I need to run the next time Nice thing about that is because those are comments within bash itself I can still run dot slash hostname.sh And it runs just fine From the head node without being submitted But I can also submit it Exactly like it is And it hasn't run yet. I'm going to see what it is in the line Kind of surprised I'm just going to go through the whole thing to the bottom Ah Dave Huh My name is not at the bottom of the list. So it must have finished by now. Yep. There it is Dave wrote case that so I give him a hard time if anything goes where it goes wrong with case that That time ran on l 74. So you can see that it actually did this Let's do case that dash dash me Dash d one right This is everything I run in the last day And you can see that all of these have Uh One node one core the last two have asked for half a gigabyte instead of a full gigabyte Uh, the last one took four seconds to run. That's strange. You must have had Some slowness getting moved over or something instead of just one second But they all show they're completed. They this will show all all everything I've done within the last day So you can see by doing by changing those parameters. I can do lots of things Um and and give it what information I need now. Here's the important things to know Like I said first of all is running on Uh, I'm you'll hear me say this over and over again running on the the cluster doesn't necessarily make it faster If you're if your code is not meant to take to make take advantage of more than one core Asking for more than one core won't help some programs explicitly Tell you how to tell me how many had tell how many cores that you wanted to run on Some try to detect how many are in the whole machine So let's say that I've told slurm I want to run this on four cores But your program it just says I'm going to detect how many cores that are on the box and it happens to be an 80 core box It's going to try to Spin off 80 threads at once But your stuff's only going to run on its four cores. Which means those 80 processes are going to be fighting back and forth each other. So A lot of programs do this You can use the In was in tasks. There's if you look in the slurm documentation There are ways you can tell it to use How many cores you've been allocating? That's one of the parameters that that you can That uh slurm puts in there So that way it'll detect it'll say, oh, you've only asked for four. I want to try to run it on four. That'll help um That doesn't that won't affect anybody's jobs, but your own But it'll slow your jobs down a lot. So if you want your stuff to run well Try to match the number of cores that you're asking for with the number of cores that your program is expecting when you ask for ram we will allocate That much ram for your process and another quarter of that and swap. So if I asked for one gigabyte of ram like I did the first time and It tried to use 1.1 gigs of ram It would let you use that first gig of ram After you use that much it'd make you swap. So slow your job down again when it's going to swap just saving information onto the hard drive And and then it but it would still continue to run If you tried to if your program tried to use two gigabytes of ram It's going to use the first gig. It's going to swap for the next quarter of a gig And then it's going to die saying I can't allocate any more memory So don't underestimate how much ram you need The last thing is the time I said don't underestimate that either because once it gets To the end of your time It's going to kill your job off. It's going to it's going to shoot it in the head there'll be Your job is going to quit processing at that point now We can extend some time limits on there We usually don't We will usually only do that if you're on a node that's owned by a research group Uh and People in that research group are kind of okay with it and you know if you say Oh, I had this to run for an hour and it really needs to run for an hour and a half No, that's not we're not going to mess with that for a half an hour um If you've had I've been running this for 15 days and it's given me progress reports and it looks like it's going to take 16 days Yeah, you're going to waste a lot of processing power That's not going to get finished if you do it that way So that's that's probably something we're going to look a little more deeply into so that's a manual process We don't let you guys change it Uh for the very simple reason that you could very easily go in and say hey, I'm going to use one hour And then your job starts running because oh just kidding. I'm in 100 hours So it'd be really easy to game the system if you did that so we don't allow that So it's only us admins can can make those changes and we make you submit a request for that so Don't ask unless it's legit because that that is manually reviewed Um one other thing on there speaking of of support On our support pages Silly mouse There's this whole section on here to here to get it how to get help There is a frequently asked questions. We have freeing videos that kind of thing Right here is a very very important sentence Mail us at that email address be okay at cs.ksu.edu Give us the job ID number the path in the script for the name of the job in a full description of the problem That will help us as much as anything, you know when I when we get emails that say my program doesn't work What does that mean? Does that mean it won't submit? Does that mean it won't start? Does it mean not giving you the results that you thought you had? Be as descriptive as possible We get a lot of questions from a lot of people and you need to be Cognizant that we don't know exactly what it is that you specifically are doing most of the time so That is probably the best way of getting ahold of us is through that email address That goes to the three of us to dave to adam and to me And even though it goes to all three of us usually we'll divvy that up You know as far as who's the best person to answer that question But it also helps even if you know dave might be the best person to ask that question But he happens to be out on vacation that day ebb and I can answer that pro answer answer that question so We ask you to please Send your questions to the list even if you and if you think it's really if you've been talking with dave about it Send you email to that address and say dave We were talking about this kind of you know refresher and that kind of thing and that way that'll tell us too that You know you've already been working with him. That's probably a dave problem But we'll also know if he's sick or something like that and and can't answer that and maybe get you caught up as quickly as possible Uh, we all rely on filtering mail if you send mail to me directly. I probably won't see it for a while Now we are ready to move on to case that and modules So i'm gonna let dave take it from here Oh, just okay Got to find the right spot to do that first and get my mouse working in the right spot first And then I got to make dave a host, right? I think he should feel the Share since no one else Okay, and I'm gonna give microphone to Oh, yeah, okay Okay, so everyone on zoom on zoom should be able to see my screen uh should say List of the nodes the wizard nodes on the left hand side if that's not coming through on zoom, let me know Uh, if you can't hear me well enough, let me know that as well. I'll increase the font size a little more Okay, what i'm going to start going over is uh, some uh things with case that Case that is a pearl script that I wrote up to help us See what is running on our uh bail cap supercomputer um If you do a case that minus h That'll give you a sorry, that's the wrong one You do a case that minus minus help That'll give you a basic, uh overview of the command itself and how to use it So this is true of a lot of applications if you do minus minus help They'll give you some basic information on how to use them At the top here if you just use case stat with nothing else It'll give you information on the hosts or each compute node It'll also give you information on each job that's running on that host And then at the end it'll give you information on all the jobs that are queued but not yet running So that's one common thing to do if you just want to see what's in the queue then use case stat minus q Case stat minus c will give you a information on What percentage of the machine is being used by each person or that's the core usage? I'll go over examples of these in a moment If you want to see Your jobs that have run already Case stat minus d and then the number of days you want to go back We'll show you all the jobs that have completed in the last say 10 days if you put 10 there And give you a lot more information like how much memory did they actually use As well as how much you requested And some things like that Um Another one i'll go over is a long list of minus case stat minus l It gives you a lot of information on the performance of your jobs that are currently running But maybe the most useful one Is this case stat space minus minus me? That'll give you information on just your jobs that are currently running and in the queue So let's start with that one I have a sample script Uh That i've written up i've named this Sleep because that's all it's really going to do It's going to print out the host time host name The uh length of time that that host has been up And then it's going to sleep for 300 seconds I put that sleep in there because if we just run the host name it'll launch it It'll run it and then it'll be gone I want to let it sleep up there so that I can show you How to Take a look at a running job These other things here are things that kyle went over We're giving it a time of one hour, which is way longer than I need This is the common way of asking for nodes and cores As for one node with minus minus nodes equals one And then tests per node equals one That's in general how you're going to want to Uh submit most jobs is to tell it how many nodes and how many tests on each node Yes Yeah I'm putting it in for a specific purpose. There are people who put it in They think it's for specific purpose. We may not agree with that And I also put in Memory of one gig, which is way overkill You can specify memory as memories memory per cpu or memory Overall is what I did. This is the amount of memory for each node So I'm going to submit this With the simple s batch command again Now I'm going to do kstat minus minus me To look at just my jobs And I can see that it's currently running Uh, it's showing me wizard 22. So it's identified that that's where my job is running And my job is highlighted in green down here And Again, it gives a job ID number. It's running on one core Uh, I've allocated one gigabyte and it's been running for zero minutes. So this will be the Clock and show me how As I if I tracked it it would show me, uh, how much time it's being used Uh, so again, this will just run for five minutes, then it'll die I don't think that Kyle showed you what to do if you wanted to kill a job off So, uh, what I'll do is I'm going to do an s cancel And then I'm going to copy The job ID number and S cancel will just kill off a run job Now if I do I'm using my up arrow to go back. So I don't have to retype this And then kstat minus minus me Should now tell me that I don't have any jobs running And if I do an ls I am on my home directory So if I actually, uh Do a cat or a listing of the job It says that I I did uh the host name and it said wizard 22 which we already knew And then the uptime says that that node's been up 11 days And that it does tell me that I did manually cancel it So kstat minus minus me is good for telling you what your jobs are currently doing Um kstat minus h is another common one to use This will give you information about just your Uh just the hosts not the jobs or jobs in the queue And I'll go back up near the top here We have about 340 nodes at this point So quite a few and this starts with the oldest ones up front The oldest ones are our mages 80 cores each They're about eight years old We keep them around Just because of this They have A thousand gigabytes or one terabyte of main memory each So there are codes chemistry codes quite often and Bioinformatics codes use large amounts amounts of memory And we need to have some of these machines available with very large amounts even if they're old These elf nodes mostly have 16 Cores to them and usually 64 gigabytes They're over five years old. So you'll see some that are down But probably most these are dead Uh and won't be brought back You can also see on the right side These are nodes that are owned by certain groups Down here There's some that are owned by a professor Ben Liu for example That means that people from his group have priority there Anyone can run on that But we restrict People that aren't in his group to run short-term jobs Because if someone from his group comes along they can kill you off And we don't want you to waste too much time So if you're running jobs that are less than 24 hours typically you can run on jobs owned by different groups But if they come along then that'll kill your job off It'll recue it automatically and you'll get running elsewhere Some of these Are not owned at all For example, that means everyone has equal access and if you run something there won't get killed off We have some reserve nodes that we usually use for classes that are being taught Like dan when he teaches his parallel computing class cs6 25 They'll reserve those nodes The hero nodes all have 24 cores 128 gigabytes for most of them A lot of them are owned But again, there's a lot of these that are not owned And so they're open for anyone to use Um one thing I want to point out We do have some nodes owned by The large memory group These are in this case hero nodes Have about half a terabyte or 515 gigabytes of memory They're open to anyone who needs large amounts of memory And again, what is large amounts? Well, a lot of our nodes have 128 gigs If you're going to be running let's say 100 gigs or more Then you have Reason to email us and say hey, I'm running lots of jobs that are large memory 100 gigs or more Can I have access priority access to the large memory nodes? And the answer usually is yes Those are there to give priority to people who need large amounts of memory Dwarves are even newer They're very similar to the heroes other than they have 32 cores They again mostly have 128 gigabytes of memory Um, I do want to point out there aren't very many that are not owned But there are also two owned here Dwarf 38 and 39 that are owned By the gpu group the gpu group is just open to anyone who needs access to A graphical processor to speed up their code So if you have code that's enabled by kuda to run on a gpu and can make use of Graphics processors to speed it up Then that's where you can get in touch with us and we can give you priority access to the gpu nodes The moles 120 of those have 20 cores There we got a great deal on them um They have very low amount of memory 32 gigs is still plenty to run a lot of jobs But it's not as Much as a lot of our other compute nodes A lot of these are owned by groups But again, you can see that there's a lot that are open here Zero of 20 cores being used By mole 53 through 64 So if you had Small memory jobs that ran for 24 hours or less And submit submitted those to the queue they would run right away on those And then our newest ones or some of our newer ones are these wizards 32 cores and you can see they have four gpu's in each of them And they're owned mostly by the chemistry mri group Purchased with a national science foundation mri grant Uh, so they have a lot of codes that can be sped up using these graphical processors So we have four gpu's in each of those The newest ones at the bottom here are warlocks Everything before this is an intel processor the warlocks are amd epic processors 32 cores 128 gigs of ram And these are owned by again One professor Um If you want to see who's Running on the cluster as a whole kstat minus c will tell you How the cores are allocated I colorize everything to identify the heavy users or red use Is something to take note of or it's a warning or something like that In this case anyone who's using more than 10 of the cores Will be highlighted in red. It doesn't mean they're doing anything wrong So this person is using Uh, nearly one quarter of all of baokia Now some of that is because the professor owns a lot of the cores So they get priority access and others. They're just Uh, submitting a lot of jobs and They're more relentless Yellow just means they're using more than 5% Again, here's a user that's over 10% Now if you do kstat without the minus h it's going to show hosts All the jobs on that host and then at the end all the jobs in the queue And this is a big long list. So Again, if we go To some of these wizards you can tell that some of them Are open like wizard 22. There's zero of 32 cores being used Wizard 24 32 of 32 cores are being used This is a one and a half terabyte machine with only 62 gigs currently being used And here you get the breakdown This one user Is running a code Their script name is coral underscore p You can see the job ID number. They're running on 15 cores and they're using 25 Out of their 60 gigabytes that they requested And here you can see that they're running for they've been running for 14 and a half hours So this gives you a fairly complete listing Of what each job on that machine has been doing And one thing you always want to pay attention to is the colorization again red Background means there's something wrong wizard 26 is down. So it's highlighted red background This job here Has a yellow background for the memory request That's just indicating that they've asked for 16 gigs, but they're only using Half of a gig of memory So next time they run they should request less Excuse me Down at the bottom here is everything that's queued And waiting to be run Up at the top there's The time that these have been in the queue Is highlighted in red or even Back highlighted and red meaning these have been waiting these top five jobs have been waiting a long time So that's usually because they're asking for specific resources Like this first one is asking for the four GPUs on the same node and there's just not that available at the moment These other next two jobs This one's asking for 160 gigabytes And there are nodes That they're requesting that are available for that, but there aren't a lot So it has had to wait The fourth and fifth in the list here asking for one GPU but also also some Fairly large memory as well Everything below that you can see four hours max So we have five jobs waiting more than a day But most everything has been waiting four hours or less and a lot of these are requesting GPUs and again This professor by Hadori has a lot of nodes that have GPUs on them. So they'll get through it fairly quickly The ones at the bottom are more generic They have not been waiting long. They're asking for a specific set of nodes the heroes We typically ask Not to do that because if you ask for a specific type of node It'll be it'll take you longer to get access And here if there were dwarf nodes Or something like that that were available It would be better If they didn't request specifically the heroes And in this case, especially they're asking for 240 gigabytes So I think the last thing I want to show you is I'll pick a node wizard 24 I want to show you if you'd use the minus l gives you a long list or more information And then i'm going to just choose wizard 24 now And what this will tell me is more information about the memory usage and the cpu usage for each job So in this case again It's telling me 62 out of the 1500 gigabytes has been used It also tells me how much of a local disk is available if you're running from slash temp It gives you information on the cpu usage. So it's this is good. It's saying 95 94 Of the cpu's are being used. That's really pretty good It gives you some information on the network rates, but then it breaks down by each Job This job the the memory Is currently running 25 gigabytes and the maximum was 25 gigabytes And again, that's out of a requested 60 gigabytes. So everything looks good there And for that job, it's telling you That 97.6 of the cpu is being used So all these look pretty good this last one though What you can see is the cpu usage here is highlighted in red. That means something's a little long there And they're only getting 48.6 Usage and half of the cpu time is idle And i've looked into this Code that's being run And they're running it on multiple cores. It is capable of using multiple cores, but not efficient at all at doing so so they're Submitting jobs on one or on two cores or four cores But they're usually only a little bit more than one core. So in this case, I would contact the person and say, hey Uh If you submit on two cores or four cores and only getting a 30 bump, uh, you should back off and only run on one core But by using this minus l flag, it's a way that you can look more in depth At your own code Okay, any questions on kstat? the best way to Deal with kstat is just to do kstat minus minus hell And try things out and if you have any questions, uh on kstat or any of the rest of this Email me or you can stop by my office in 2019 engineering hall over here So if there's no other questions, uh We didn't get to cover modules and that will do that on thursday Okay, then we'll wrap it up for the day And I shut off Yeah, I don't I don't see where any of that is at the moment Let's do it that way. So sometimes like yesterday last night this morning