 It's a great jobs and we're actually right on time. So should we try the same thing we did yesterday where we really quickly discuss things and go to the exercises? Yeah, I think that's... And then do a post-discussion. Yes, that's a... we can then think about all of the... Like I quickly mentioned, like throughout these coming sessions we have exercise for all of these, but it's very, very hard to create the exercise. Like I used all evening yesterday trying to figure out better exercises. It's very hard to create exercises that throw the... the... the concepts and at the same time they can be run on every kind of a cluster, like they're not only on us, on our cluster, because that would be like not nice for people who are running in different setups. So the exercises, if you think the exercises are... like if you don't feel that they've demonstrated what you would want to... like if you have more stuff you need to... like you would want to do more, then ask in the... ask in the notes about like does this... is this applicable for your program? And maybe we can answer that. But yeah, like it's very hard to write the exercises that are... like these might seem trivial exercises in some sense, but the point is that they're trying to show the... show the concepts. Okay, so we just talked about this, we showed the picture, so I think we're good there. Should we go straight to an exam? Okay, good. So, okay, so what do we do? Should... well, so what's the difference between a rate job and what we did yesterday? So, yes, so we just mentioned the scheme of the embarrassingly parallel. So there was a question in chat wise called embarrassingly parallel. That's because like it's embarrassingly easy, it's easy to do. And this in... like it basically means that you're running... like if you think that you would run like same command or multiple terminals at the same time, that's basically what this is. But it makes it so that each of these terminals or each of these commands gets executed in a different place with its own like... that definitely has the resource to run the stuff. So previously, when we were running stuff in the queue, we were using these serial jobs and these AS batch scripts. So in the script, we had written what sort of requirements we wanted with the comments, the AS batch comments. We were putting their like time required memory code, that sort of thing. And then we were putting whatever code we wanted. And when we submitted that to the queue, that got executed somewhere. So what if we do that, but we do that multiple times at the same time? And this is the idea behind the array job. So with the array job, we have this array syntax or this comment, this AS batch comment. So over here, we have an array that goes from zero to four. So it goes from zero, one, two, three, four. And what Slurm thinks, when it sees this comment in your AS batch script, in your normal script, it will understand, okay, the user wants to run five copies of the same thing. I will book five different jobs. But as a collective, so I will book five jobs that are independent and they are called an array job. And all of these jobs are the same, exactly the same. And the only difference is that Slurm will set this one environment variable called slurm array task ID. And environment variable, I don't think we mentioned it yesterday, but these are basically like stuff that you can then use in your script. So they are like variable set to the terminal. And when Slurm sets this environment variable, you can then do whatever you want with that as long as that does something. So you can choose, let's say a data set, or you can choose a different parameter or something if you, and it melts multiple ways of doing it, but you can use this environment variable to decide what the program wants to do, basically. So if the number is zero, I'll do something. If the number is one, I'll do something. So maybe we should just go to the example. Like last night, I can make an array job from zero to four. The job zero processes the zero video in the list, one processes the first, and so on, or something like that. Okay, let's, can we see an example? Yeah, let's just do this first example here. So Richard is now going to the SPC examples repository. So in his home directory. So at this point, I again say that it's a good idea to log in to your cluster and go to the SPC examples folder so that you can then do the same kind of thing. Okay, so let's create a new file called the array example of this page. Okay, I'm making it with a dash instead. So I'll copy the basic stuff. Yeah, so let's focus quickly before we move forward. Yeah, okay, this is the basic stuff we did yesterday. Yeah, so the basic stuff, so here we see that this is basically like a normal stuff that you would add to any kind of like a slurm script. And now we're adding two lines, actually. Here we have one line that has these weird characters. We'll talk about them after we see the output probably. But let's just use those because we'll see what mysteries they will provide us. Yeah, so I guess this sets the output file somehow. Yes, and then let's add the array thing. So here we have an array from 0 to 15. So we can calculate that, okay, there's 16 different jobs that we expect. And now let's put some commands below there. So what Richard will place here is that he will put the command called echo, which will just bring stuff onto the screen. And yes, and then he will put this environment variable here. So what we expect, like if we look at this code, what do we expect? We expect that when we give this a slurm, slurm we would book us 16 jobs, one with each array ID. And each of these jobs should print something like an array task number and then the number that they've been given. So let's see if this happens. Okay, so I will ctrl x to exit and save. And should I s batch it? Yes. And again, when we're talking about bash and s batch, remember different words, they are very hard. Annoying to pronounce, but s batch. So s batch array example.sh. And then should I slurm q to see how it's going? Yes, it might. Let's hope that it's not finished. Okay, so now we see that there's lots of these jobs now. Like something really happened. So at the start, we see that there's like strangely two numbers now at the start and we noticed that the first number before the underscore is some number and then after the underscore we have another thing. And this is very easy to decipher. So the first number is the slurm job ID. So when you give this array of the slurm, it will make one of these job IDs for the whole submission and then it will, with this underscore, separate the error ID. So each of these second numbers corresponds to the array ID. So let's check what the output is. So if you type ls, now we see, yes. And if you look at the pattern of what's in the name of these files and compare that to the pattern that we put into the s batch script. So in the s batch script, we wrote dash dash output equals array underscore example underscore. And then this percent capital A underscore percent small a. And these are basically wild cards. There are a few of these in slurm. You can find them all in the slurm documentation or s batch documentation. But basically what slurm does is it feels the array job ID and the array task ID to these environmental variables. So then you get this fancy and nicer looking output. And the outputs don't crash. The main point is that if you write all of these, if you would have only one output file, then everybody of these would be right into the same file. And that makes it impossible to see what is the output really. So what you want to do usually is to divert the output into each so that each task gets its own array task, gets its own output file. So let's look at one. OK. So we're looking at the 0th one, I guess. I will use the less program like I do. And I'm array task 0. Should I look at a few others? Yes. And Q2 quitter. Yeah. So I guess like this. Yes. And do you want to show like slurm history? So if we look at the slurm history, do they all end up on the same node? I think you zoomed in. I can make it smaller. Yes. So you see that 31. Yes. So you see that many of these ended up onto like CSL 44, 36, 31. They ended up on different machines. So they are completely independent and they can end up to different places. So the main thing is that they got the memory and the time they needed. And they all get the same request. So there's no multiple gates or anything like that. Like you don't have to divide anything or something. Like it's just like they all get the same resources. They get the same time, same memory and they are completely independent. So each of these gets 200 megabytes of memory and 15 minutes of time as per the request. So let's, I think we could check. So yeah, what else is there before the exercises? This is basically the main idea. Yeah. So maybe, yeah, we can go for the exercises. But yeah, we have some examples here of different ways you can use this array index. So you can basically, let's say you have different input files. You can put it the array task ID to decide which input file to use. You can use this case syntax. If you scroll a bit down. This case syntax to decide on, like in this case, based on like the Slurka-Margaret task ID, each job will get two set of different parameters. You can also use... And this is bash syntax here. Yes, that's... So bash doesn't just let you type stuff, but it's a whole programming language for running things. So depending on what this is, it sets a seed differently and you notice it gets put as an argument here. Yes. And you can also like read full lines from a file. There's an example of that as well there. And even more things you can do. But I think now it's a good time to maybe just jump to the exercises and let's return to these afterwards. So in the exercises, we previously run the pi.pi to run multiple, like we run the simulations of the pi iterations. So in this case, let's do iterations in the array jobs and let's collect them together and get the better estimate for pi. And in the first exercise, I will mention that like part of the output of the pi.pi is thrown into the standard error and part of it is thrown into this output file. So if you just like use these spats, that's this error, you can like get the JSON output and the rest of the like human readable output separated. So that's why it's mentioned there. So, but yeah, let's just try running these exercises and let's see what happens. Let's see. Yeah. And I guess we come back. Yes. After, well, when do we come back? Well, it's almost break time now. So should we give 15 minutes to work on it? So come back at 15 past the hour? Yes, yeah, I think that's a good idea. And ask your questions in the chapter. There's already like a few good questions. We'll try to answer them. Okay, so here we are, exercise info. Yes, so now's the time you may have lots of questions about how to actually use this. So do let us know. Yes. Okay, so see you in about half an hour. See you. Thanks, bye. Hello, and welcome back. So it looked like people didn't quite have, their exercise didn't quite work out. Yeah, they were. So should we go over them? Yeah, maybe we should quickly go through the exercise one. Do you want to show it to us? Yes, I will come back to it. Okay, to, oh, I'm not sharing. Okay, here we go to my screen. Yeah, we'll add up the code in the future so that it has a flag so that you can quiet the rest of the outputs or something like that so that we don't have to worry about the error. Error, that's something that we'll have to modify in the exercise in the future. So, yes, where are we? So exercise one. So we'll have an array job that combines different iterations and seed values and saves them to different files. Should I open a script? Yes, let's call it like that. No, array exercise one.sh. Yes, and yeah, let's put the usual liturgy there. So in this case, what we want to do is like, okay, we want to ground multiple different seeds, for example, and to do that, let's use the case statement. So let's, in the previous possibilities of how you can use the slurm array task ID, there's this case statement. So this is like Richard mentioned, like bash has its own way of like, it has its own programming language so you can use these case statements here. So based on the value of the slurm array task ID, let's choose numbers for the iterations and for the seeds. Does this look good? Yeah, well, we can try it out, doesn't hurt. Yeah, and let's run a few of these. Okay, and then let's run. We probably have to close the case statement and change the numbers, not everything zero. Yes. And of course, we want to apply the array statement at the top. Yes, okay, so this array will be zero, two, three. Yes. So yeah. SlurmPy.py, and then that number of iterations. Yeah, let's put like what we had in the exam exercise, let's put the output files and the error files. So in the code, like some of the stuff is printed the standard error. In the future, we'll add a flag there that you can silence like doing this number of iterations, this kind of debug flags. So let's, for now, let's just like put all of the, like the output here is the, like the output of the code. So the output that we want to chase an output on and then the error that needs to be the espatch error. Oh, yes. Yeah, so let's put the rest of the printing there. Yeah, we'll add up this in the future. Yeah. But you can do this, like you can do in any code, like it doesn't have to be aggregate code. So if you want to put the errors somewhere else than the output, because sometimes stuff can print like, like a debug stuff and the standard error. Yeah. So yeah. Okay. Let's run this. Does this look good? How many homes do you think they'll be when we try running it? It looks, I guess the main, it looks good to me. I see. Oh yeah. Yeah. That's good. Yeah. And I need to make sure I'm in the correct directory where slurmpy.ply is fine. So I will exit and save, control X, Y. Yes. I will list slurmpy.ply to make sure it's there because if I wasn't in the right working directory, this wouldn't be found. Okay. Should I espatch it? Yes. Are we ready? Yes. Let's go. So two and three are running. Those are the larger number of iterations. So I guess it makes sense they're taking a bit longer. So maybe we want to check one of the output files that already finished. They're done. Oh yeah. So I list, so we see array, exercise one, error and output two, three. So yeah, this first column is what we expected. So should we first list the output and this time I will use the cat program so we can see them both together. So cat has just that in it and if we cat the error, so this is actually doing what we'd expect it to do. So. Yeah. So we shouldn't write in the standard error. I think we should add a flag to silence. Yeah. So this is how some programs work. So the idea there's two different output streams. So the output has only the primary output of the program that will be reread and the error contains these other status messages. If you needed them both and you didn't quite it. Okay. So. Yeah. In the. Should we list a different one? I think we need to move forwards but maybe I could answer one of the questions in the array, I got some questions that was really good. If you'll share my screen. So. Okay. There was. Oh, I'll have to start this. You need to share. Yeah. I'll take this over. Yes. So there was a question in the notes about like how could you use, for example, use our great jobs when you're doing like training a neural network or something. And this is like very common thing. Like the answer was that you can basically like run different type of parameters and that sort of things like learning rates and that sort of stuff on different array jobs. So you can test out different parameter combinations. So this is quite often what people do with that array job. So let's look at these code, for example. So if you quite often, if you're quoting something, you notice that you might have a follow outside there. That you do a for loop over some parameters or something. And then you do run some code and then based on the code, you like maybe plot out the picture or something like that. And this is quite, yeah, okay. It's not showing pulling. So this is quite common. Sorry. So you might have something like this. And for that, you might have a slurms script that submits it that looks like this. But this is not optimal because here you cannot like adapt the like the parameters. You have to hard code the parameters into the code. And that's usually not a good idea because that's inefficient. You cannot easily change parameters and you don't know which parameters you used to run or whatever. And usually the better idea is to use something like this that you've write some sort of like argument handling to your code. So your code read some arguments from a command line, maybe from a file, I don't know. Like it creates a file name. It usually needs to get some argument from the command line so that it can like determine what to do. And then it does whatever it's supposed to do. And if you have your code written like this, you can easily write an array job where like you have an array job and then you choose the parameter and then you run the code with that parameter. So this is like pseudo code. So it doesn't work actually, but like you get the idea probably. So you can now run this array job and because we have the array, you suddenly have twice the like speed, twice the throughput of your program. So now you can run two parameters independently at the same time. If they're completely... Without modifying the code, which is the important part. The only modifications you need, you need to have some logic in the code so that you can give it stuff based on some arguments or some other thing. And this is good in a technical sense. You always want to write stuff that you can like, you don't always have to hard code the parameters of your models and stuff into your code itself. But okay, let's move forward. I think we need to... Yeah, yeah. So can you show exercise five? There's a good question there. Or should I share my screen? Yes, yes. I'll come back to my screen. Yeah. Okay, so exercise five, someone had a question that we said we should go on soon but we'll point out the point here. So the point of the array jobs is that all the boxes are the same size, the same number of CPUs, the same amount of memory. So even though you can tell this memory user program to use different amounts of memory, they all have the same amount of memory requested. So either some of them run out of memory or some of them are requesting too much. And this is not sort of the right way of using array jobs or at least it's not how it's expected if the memory differences are this much. You could put another way. So postal services give box bulk discounts for shipping boxes of the same size and just like this. Yeah, so if you have programs that are like or if you have posted stuff that is very small and you put it in the huge box because some of your stuff fits into the huge box or needs the huge box, that's not the most efficient way of posting stuff. Yeah, like the memes about getting something from Amazon and it came in a box that's far too big. Yeah, so the reason is that they have like a bulk discount on these huge boxes and then they fill it with like environmentally unfriendly and like plastic or something to make up the difference. So it's the same thing. So when you're submitting an array job you would want the jobs to be similar in size. In some cases if you have like, I don't know you want to go through a million things and all of these million things they depend how long they run so that they can be like you don't know when they finish or something. So there are ways of like grouping the jobs together so that you get like similar runtimes for all of these jobs or you can make it so that like each job will run a certain amount of jobs and then you wish that the mean goes towards like a certain value. So if your jobs might take a minute or 10 minutes and then you run like 100 of them the average starts to turn out because of large numbers you start to reach towards the average so you can like decide the time based on the average of like all of the small jobs. But this is like again more complicated stuff and let's not go there but there are ways of tackling these things but usually you want your array jobs each job needs to be similar to the other ones if one has a higher memory consumption then it forces everybody to have a higher memory consumption and that's not good.