Job Scheduling with Slurm

What is Slurm? 

Many software research applications these days tend to consume large amounts of computer resources– CPU, GPU, or RAM. In SCI, there are a number of high-performance machines that have large amounts of these resources and we use a job scheduling infrastructure, based on Slurm, to manage access to those resources so that they can be used consistently and efficiently. 

The general approach when using Slurm is that instead of manually starting up processes that consume large amounts of resources, you submit a “job” to the scheduling infrastructure (Slurm), along with a spec of the resources needed to run the job (e.g. GPU, CPU, RAM), and then that infrastructure is responsible for finding appropriate computer resources as soon as they are available, and running the job on your behalf. This prevents computers from becoming overloaded and bogged down, eliminates hunting around and discovering where free resources might be available, etc.

How do I use it?

Any powerful tool requires a bit of learning, understanding, and familiarity to use well, and Slurm is no exception. There are many resources to be found online, but a basic quick-start guide is given here. 

Overview

Instead of logging directly into individual computers to run jobs, the job scheduler is orchestrated from one central node, which in SCI is compute.sci.utah.edu. Once you ssh into compute, which is just a normal interactive Linux computer, you then have access to a number of commands that allow you to start and stop jobs, look at what jobs are running now and how resources are being used, etc. When a scheduled job completes, your program’s output will be logged to a file. 

A few basic pieces of Slurm terminology: 

Node – an individual computer that provides some compute resources such as CPU, GPU and/or RAM
Partition – a group of nodes with similar sets of resources across which Slurm can schedule jobs
Job – a submitted task or set of tasks given to Slurm to be scheduled and run whenever resources are available; job steps are discrete (and possibly parallelized) tasks that make up a job
Queue – the list of jobs waiting to be scheduled

Basic Commands

There are a lot of commands and options for using Slurm, and it can be a bit overwhelming to get started. However, the basics are not too complicated. 

One thing you probably want to do the first time you log into compute.sci.utah.edu is see a list of resources that are available. This is done through the sinfo command: 

Output of sinfo command

As you can see here, there are a number of partitions, each of which are comprised of several nodes each of which have their own current state (e.g. idle, fully or partially allocated, and down/unavailable). You can get detailed information about nodes by running scontrol show node <nodename>. 

To see the list of currently running jobs, use the squeue command:

Here you can see that there are two jobs currently running, one on the node chimera and one on the node pegasus.

To run a job, you can use the sbatch command. To do so, you write a file that is basically a bash script with some extra headers thrown in there that tell Slurm how to run your job. The full suite of features and capabilities is quite large, but here is a simple example script that prints out the computer hostname and date, pauses for 5sec, then prints out the date and exits. It also requests a reservation of 1G of RAM and one GPU (totally unnecessary for this simple script, but just serving as an example of the kinds of things you can do):

#!/bin/bash
#SBATCH --job-name=chadtest
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --gres=gpu:1
#SBATCH --time=5
#SBATCH --mem=1G
#SBATCH --partition=titanrtx-24
#SBATCH --nodelist=atlas
#SBATCH --output=log_%J.txt
#SBATCH --mail-user=
srun echo "Running on:" $( hostname )
srun echo "Time start:" $( date )
srun sleep 5
srun echo "Time end  :" $( date )

 

And here is what running the job looks like:

As you can see, Slurm was requested to run this job on the wormulon-cpu partition and it picked an available node, wormulon1 to run the script on (which is verified by the output of the hostname command in this script). Note how the script also included an #SBATCH directive to write the output of the job to a file called log_%J.txt (the %J means the Job ID as given by Slurm). 

Additional Resources