What is Slurm?

Many software research applications these days tend to consume large amounts of computer resources—CPU, GPU, or RAM. At SCI, there are a number of high-performance machines that provide substantial resources. We use a job scheduling infrastructure, based on Slurm, to manage access to those resources so that they can be used consistently and efficiently.

The general approach when using Slurm is that instead of manually starting processes that consume large amounts of resources, you submit a “job” to the scheduling infrastructure (Slurm), along with a specification of the resources needed to run the job (e.g., GPU, CPU, RAM). That infrastructure is then responsible for finding appropriate computer resources as soon as they become available and running the job on your behalf. This prevents computers from becoming overloaded and bogged down and eliminates the need to hunt around for free resources.

How does it work?

While there are several computers and services that comprise the Slurm infrastructure, there are really only two components that you need to be aware of when submitting jobs:

• User Node: Instead of logging directly into individual computers to run jobs, the job scheduler is orchestrated from one central machine (compute.sci.utah.edu). When logged into this machine (via SSH), you have access to all the Slurm commands, such as sinfo, sbatch, etc. This is where you go to do your work and manage your jobs.

• Compute Server: Machines that have the capacity to run jobs (large amounts of CPU/RAM, GPUs, etc.) are the compute servers. As a user, you should be less concerned about what specific machine you want to run on and instead think about what resources you need to get work done. Slurm figures out the best resources to use given the stated needs and allocates those resources for your job alone.

How do I use it?

First off, you need a Slurm account created for you. If you are new to SCI, your account has already been created. If you started at SCI before June of 2025, you may need to contact SCI IT to get your account created.

Once you SSH into the user node, you’ll have access to commands that allow you to start/stop jobs and monitor usage.

Basic Terminology

• Node: An individual computer that provides compute resources (CPU, GPU, RAM).

• Partition: A group of nodes with similar sets of resources (e.g., general-cpu, general-gpu).

• Job: A submitted task given to Slurm to be scheduled.

• QOS (Quality of Service): A priority level assigned to a job (e.g., normal, pi-qos).

Basic Commands

One of the first things you may want to do is see a list of available resources using sinfo:

sinfo -o '%16P %12n %.6t %.4c %10l'

To see the list of currently running jobs, use squeue:

squeue -u $USER

To run a job, use the sbatch command. You’ll need to write a file that is essentially a bash script with special #SBATCH directives.

Example Script:

#!/bin/bash
#SBATCH --job-name=test_job
#SBATCH --partition=general-gpu   # Submit to the shared GPU partition
#SBATCH --constraint="gpu_ada"    # Request a specific generation (Ada Lovelace)
#SBATCH --time=1:00:00            # Request 1 hour (Max is 1 Day for general)
#SBATCH --nodes=1
#SBATCH --gres=gpu:1              # Request 1 GPU
#SBATCH --mem=4G
#SBATCH --output=log_%J.txt

srun echo "Running on:" $( hostname )
srun echo "Time start:" $( date )
srun sleep 30
srun echo "Time end  :" $( date )

What resources do I have access to?

The SCI cluster is divided into General (Shared) resources and PI (Dedicated) resources.

1. General Partitions (The “Commons”)

These partitions are open to all SCI users. They run on shared hardware or unused PI hardware.

• Account: Jobs here run under the common account. If you try to run here using a research group account, Slurm will automatically force your job into the common account.

• Fairshare: Priority is determined by User usage. If you run many jobs, your priority drops relative to other users to ensure fairness.

• Time Limit: Strictly 1 Day.

2. PI Partitions (Dedicated)

These are owned by specific research groups (e.g., pi-group).

• Access: Only members of that specific group can submit here.

• Time Limit: Custom/Unlimited.

• Priority: Submitting here allows you to use pi-qos, which gives you extremely high priority and the ability to preempt scavenger jobs.

Dealing with Preemption

If you submit to preempt-cpu or preempt-gpu, you are “scavenging” idle cycles from research groups. If a research group member needs their hardware back, your job will be Preempted.

What happens during preemption?

1. Your job receives a SIGTERM signal.

2. Your job is stopped and automatically Requeued.

3. When resources are free again, your job starts over from the beginning.

Strategies for Success

• Checkpointing: Your code must save its state periodically (e.g., to a file). If your job restarts, it should read that file and resume where it left off. Without this, you lose all progress.

• Job Arrays: Instead of one massive job, break it into smaller tasks using job arrays. If preempted, only the current task needs to restart.

• Check Queue Status: Before submitting to preempt partitions, check if the PI owners are active using squeue.

SCI Cluster Feature Constraints

Slurm Constraints allow you to request specific hardware properties for your job. This is essential when your code requires a specific GPU model, CPU instruction set, or when you simply want to target the newest hardware available.

To use a feature, add the --constraint flag to your job submission:

# Example: Requesting an H200 GPU
sbatch --constraint="h200" my_job.sh

GPU Features

You can request GPUs by their Specific Model (best for reproducibility) or by their Architecture (best for flexibility).

By Architecture (Recommended)

Use these tags to request any GPU of a specific generation.

By Specific Model

Use these tags if you need an exact GPU model.

CPU Features

You can constrain your job to specific CPU manufacturers or generations.

By Manufacturer

• cpu_amd: Any AMD CPU

• cpu_intel: Any Intel CPU

• cpu_epyc: Any AMD EPYC Server CPU

• cpu_xeon: Any Intel Xeon Server CPU

By Generation (Newest to Oldest)

• AMD: epyc_bergamo (9755), epyc_genoa (9000 Series), epyc_milan (7000 Series)

• Intel: xeon_emerald_rapids (Gen 5), xeon_ice_lake (Gen 3), xeon_cascade_lake (Gen 2), xeon_skylake (Gen 1)

Example Constraints

• Fastest Available GPUs: sbatch --constraint="gpu_hopper|gpu_ada" job.sh

• Specific Combo: sbatch --constraint="l40s&cpu_amd" job.sh

• Exclude Old Hardware: sbatch --constraint="gpu&!gpu_pascal" job.sh

Interactive Jobs and Monitoring

While batch jobs are standard, you sometimes need direct access to a node to debug code.

Interactive Sessions

To get a shell on a compute node:

# Safe interactive session (General)
srun --partition=general-cpu --pty bash

# High-risk/Scavenger interactive session (Preempt)
srun --partition=preempt-cpu --pty bash

Using tmux

When using an interactive job, tmux is highly recommended. It allows you to create persistent sessions, split windows, and keep processes running even if your SSH connection drops (though the job will die if the time limit is reached or it is preempted).

Connecting to a Running Job

If you have a batch job running and want to check on it:

1. Find the nodelist: squeue -u $USER (e.g., job is on glurmo-1)

2. Join the job: srun --jobid=<JOB_ID> --overlap --pty bash

Compute Node Update Schedule

The compute nodes in Slurm need to be updated regularly. Individual compute nodes will be scheduled for updating by putting them into DRAIN mode throughout the month.

• Jobs currently running will continue to run.

• New jobs will not be scheduled on that node.

• Once empty, the node updates, reboots, and returns to service (usually < 15min).

Additional Resources

• Official Slurm User Documentation, especially the Quick Start User Guide

• CHPC’s Slurm Documentation, including YouTube videos on Slurm Basics and Slurm Batch Scripting

• CHPC also offers hands-on training for Slurm scripting. While the SCI and CHPC environments differ, the material is transferable between them.