 Yeah, let's let's look at like let's just run a few examples and let's see what what how do you actually Would normally run like a real stuff in the queue. So Do you want to start Richard or? Maybe you can't you have the array job first prepared. Yeah Yeah, so so this is like a freshly minted this morning An average of example of an actual use case of one of our users So this there's one of our users wants to use this deep learning Framework that basically does this kind of a feature extraction of musical genres and genres and you can like Identify different genres based on their well features such as probably rhythm and what kind of instruments they are And I don't know how it works actually, but like it it basically It's a machine learning or deep learning model that the based on this highly Advanced a arm models you can do this feature extraction and We help this or trying to help this user to get this working on the system and today I finally figured out how and So I will present Okay, yeah, it's a real example I will post it into the issue. So if the user is watching then you will have the solution soon Okay, so I haven't had time to do it yet. So I'll I'll share the screen. Okay, so did you already mention what array jobs are? Not yet. So so well basically what this Well, I guess I can I can mention it while you're getting ready since yeah, you're the type of now So Yeah, go run so oftentimes when you need to run things you don't take one program and run it With more processors, but you run many copies of the same program And this is the simplest and thus best. Oh, maybe people would argue that but Very good way of scaling up to more work because well, it's simple So that's what we've that's what we're using here so Seem was gonna find a way to run the code and He's opening the batch script. What's yeah, yeah Clear it up a bit. So here he's defined what the cluster will run So would you like to walk us through what the different things mean? Yes, so so this model me. Okay. Yeah, so this is a script file Which will tell the cluster what it needs to run so it connects to the slurm queuing system So we submit this and the cluster reads this and then does the instructions and then leave the results where it's configured Okay. Yes. Yeah, so so basically typically these kinds of scripts have like Two or two parts or maybe three parts in this case it would because it's an array job So the first part at the top here Is Are the instructions for the queue system? So so here we have different resource Things that we want for this job. So in this case, we want to do this feature extraction for multiple different Genres at the same time. So we use this array job to do it so What we want for this job is to to run it for 30 minutes With 30 gigabytes of memory Produces an output file We we run it on a GPU and then we run it with this array job 10 10 times at the same time So we run all of these different 10 options at the same time Maybe in the different nodes maybe in the same node So we will go through the array job in detail later on but basically Where you can think of it like like we had the pasta analogy yesterday Like we had 10 pots of pasta, but here we basically have the situation where we want to we want to cook 10 Different pasta pastas or 10 different musical genres. We want to Feature do the feature extraction for all of these different genres So we can do it parallel. We can do every genre in and of itself In a different different machine or different system or whatever We can do it separately and the the array job makes it really easy to do this kind of a kind of a thing At the end we actually have what the program is running So this is running the single array container. So it's a bit complicated We yesterday mentioned these containers This is the kind of application that actually benefits from these containers because that's what the original authors had provided You don't really have to know the details here. The main thing is that there's something here at the end and there's some variable That that defines what Data we are using as an our input And the data is determined Well, it's basically like when when we give this script to the queue the queue will Uh Queue will split it up into Multiple chops and each chops gets their own flavor of pasta and each of them cooks their own flavor of pasta So how does it actually how do you actually run it? Well, you give it to the queue using this s batch command So you give this script to the to the queue and the queue will do it We will do this in more detail today with simpler examples, but But for now, we just type this s batch and then we run this Run extract so now we have this batch job there Running we submitted the job and we can use this slurm queue To see how it's what's it doing the queue So currently it's it's pending We can also add this watch here so that we can watch the queue. So let's wait for a second So once the chops Get going so now they're queuing so they're not actually running Anywhere yet, but now they are like basically now I can just Take my hands of the keyboard and I know that they will be done eventually So now one of them is running Over there in one of the nodes So you will see that it's running on this dgs node this gpu node that we have So if we look under job id it says Like zero and then there's one to nine in the pending state on the first line Yeah, and if we look at if we close this view and look at like what we have these Files here we can look at the Features that we get well, there's currently not yet any but we can It will start producing these outputs here once it has run enough and now we have multiple of these jobs already running So basically now we're cooking nine pots of pasta at the same time basically So we have we have plenty of stuff going at the same time Uh So so this kind of a workflow where you want to like Do multiple things at the same time you want but they can be separated It can be done easily with this iRatio I think these These might be a good endpoint for this demo, but this like this just a demo like normally this would take You could run this basically the one thing I could emphasize is that this requires 12 gigabytes of Memory from the tpu to run so you don't have that kind of tpu it costs like you can buy that kind of a tpu But it costs you like 5000 euros if you are lucky to get one But basically you cannot get this kind of a resource anywhere besides spc clusters so You need to use the hbc cluster to run it and then it will take usually it can take hours to run these all of these uh Feature extractions, but if you use this kind of a great job, you can get it done Pasta you can get it done in in half an hour all of the Different pots of pasta Okay, okay great So Now let's go whoa, that's a nice effect