 Okay, so our final presentation of the lightning talk round is George Roth who will be talking about slurm So take it away George Okay. Hi. I'm George. I work for I'm gonna present slurmin action batch processing for the 21st century. We're gonna make batch processing cool again apparently I Work for nurse, which is the primary scientific commuting facility of the office of science. We have two supercomputers Cory and Edison and three smaller clusters and together that's Around 800,000 course 50 petabytes of storage we serve 6,000 scientists from Very different fields and everything we do is open science So what is slurm slurm was formerly called the simple Linux utility for resource management when they reached half a million lines of code They got rid of the simple so it's only called slurm now It's a highly scalable workload manager. It runs on I think 10,000 nodes on a quarry and it Runs on bigger systems than that as well It's like more than 60% of the 500 fuss machine the world runs slurm Development started in 2001 at Lawrence Livermore National Lab and it's now maintained by a company called scat MD and a pretty active open source community So How many here do have an HPC background? Oh Well, we're gonna start at the basics. So you might know a lot of that stuff, but The basic functionality of slurm is you're at the job script you submitted to slurm Inside the job script you do whatever you want to do your job is cute and the priority is applied So if you submit a higher priority job, it runs first and then when the job gets to run It enforces resource limits using C groups checks the usage who ran the job along it took etc etc and then tears it down When the job is finished So some of the features of slurm which make it nice to use from the administrative side It's very easy to deploy it as a single configuration file the documentation is extensive It's a pretty user-friendly If you're coming from HPC, I guess it's a highly scalable and extremely configurable Which is a good and the bad thing so you can do anything you want to do with it, but It can get out of hand if you're not careful to the architecture is Pretty simply if a controller a database server that keeps track of the of the jobs and the resources to use run times Etc. You have a slurm D the drums on the compute nodes and it talks to the other compute nodes in a hierarchical fashion So it scales pretty well and then the programs use like s control which is used to Check the state of the nodes and sq to check how the jobs are queuing go against the controller It's I think prudent to note that there is no single point of failure so you can have it running Pretty much disruption free so I'm gonna give you an example of a job array. So let's say you want to do some image processing as It's fashionable now and you want to get so you put a batch script Really plain batch script you put the aspects directives that slurm interprets on top and Then on the bottom rear you run nowadays your docker container, right? So you give the job a name called image resize you say I want to run a hundred of Those jobs so this is gonna get executed a hundred times you want to CPUs per task and you want 10 gigs of memory per task So and then you type docker on your docker container and you set the environment variable input to the task ID Of the tasks that you're currently in so the first task is gonna have input equals one then input equals two three Etc etc because this is reduced example. You would usually get the input files out of a file like The first second third entry So what you also can do is you can Build workflows on top of the scheduler so you can say I have a high priority. That's my AI job also fashionable now It should use five nodes and it can only run after let's say the image Pre-processing job has finished and the storm is gonna take care that it executes in order It also has advanced features which are pretty nice. It has burst buffers It integrates with containers not only the way we just saw by exceeding docker directly, but by Using a plug-in you have a job submit plug-in that can rewrite the job and you submit it It can federate which means you can run several controllers and you submit the job and it gets submitted to any controller And there's a relatively extensive Amount of plugins So burst buffers are a thing that was developed with input of NERSC. We use it to Provision storage for the create data work system. So when you launch a job, you can say, okay I want I don't know 500 terabytes of SSD and then you can use that inside of the job So that's not useful for I guess anybody in the gym except one guy But you can also use a generic plug-in that uses shell scripts and then you can prove to provision whatever you want to In in this and you can automatically stage data in and out for example you could so I Do have to say full disclosure that's not implemented yet You would need to do that yourself using shell scripts but you could pull data in from s3 and stage it out to s3 and then Run your container and say, okay This is my input. It's there because it was staged in by the scheduler and when you're done stage it out again Container integration is realized by what's done causes bank plug-in, which is a bit of an awkward name. I guess as Shifter, which is a tool to run talker on HPC systems is developed at NERSC There is a plug-in for storm of course There is also one for singularity and it's very easy to use if you want to have an interactive session that runs in some custom Pipeline that you put together You just do an as a log specified the image you want to run in when you enter this interactive session You're going to be inside of that container There's a job submit plug-in as learn is very heavily C based It's based on lure so when you Submit a job request you can modify it on the fly and make let's say business decisions which can be very powerful and Very very flexible as you can access everything that the internal state of storm if you want to do that There are more plugins like this which you can write either in C or in Lua as well The lure part is also done by Lawrence Livermore lab So there are a few plugins floating on for let's say X11 support that forwards the connection back to your login node If you have different network setups, you can notify the user if he gets an OM A nurse we have a thing that loads Performance tools like the kernel module you can request the kernel module to be loaded when the job runs and then then strips it down When the job is finished as I'm running out of time I'm not gonna go too much into this but and as you all have an HPC ground I guess you know how backfill works, but basically the scheduler If you have a big job that uses a lot of resources You need to drain those resources first to be able to run it and the scheduler uses that Draining to backfill jobs that can take a shorter time to execute those you get a higher utilization of your resources and That was it any questions All right any questions Hey, yeah Sturm does support restart we have Intel K&L system So you can run that in different nodes system does support node restart, but we do not restart it after every job All right, and that concludes our lightning talk round. Let's thank George