Using SLURM¶

Login Node¶

When you connect to the cluster, you are connected to a single shared computer with all your fellow users, known as the “login node”. The purpose of the “login” node is for you to submit jobs, copy data, edit programs, etc. You must not launch any computations on this computer directly.

Compute Nodes¶

These computers do the heavy lifting of running your programs. However you do not directly interact with compute nodes. You ask for the scheduler for compute nodes to run your application program using SLURM, and then SLURM will find available compute nodes and run your application program on them.

Computing Resources¶

An HPC cluster is made up of a number of compute nodes, each with a complement of processors, memory, and (possibly in some future) GPUs. You submit jobs that specify the application(s) you want to run along with a description of the computing resources needed to run the application(s).

You may submit both serial and parallel jobs. A serial job is executed on a single node using exactly one CPU. Parallel jobs can either share the memory on a single node or run as separate tasks communicating with each other. In order to choose a desired type of your job, you need to specify the number of tasks and the number of CPUs per each tasks. The table below summarizes the possible types of jobs.

Note that the number of tasks multiplied by the number of CPUs per task may not exceed 108, unless you submit your job in the free QOS.

In addition, for the dragon there is more granularity in available resources. Each user have a possibility to use up to 3240G of RAM and resources are depending from queue used as specified in table bellow.

Batch Jobs¶

To submit a batch script to SLURM, you need to use the sbatch command. It is designed to reject the job at submission time if it requests resources that cannot be provided as specified. This gives you the opportunity to examine the job request and resubmit it with the necessary corrections.

Interactive Jobs¶

An interactive job is a job that returns a command line prompt (instead of running a script) when the job runs. Use salloc command to submit an interactive job to SLURM. When the requested resources are allocated, a command line prompt will appear and the you can launch your application(s) using the srun command.

The following example requests a node for one task, and allocates 2GB RAM and 4 CPUs for an interactive job that is allowed to last one hour. Then a sample command launching your_application on the allocated node is shown:

user@dragon:~$ salloc --time=1:00:00 --mem=2G -n1 -c4
salloc: Granted job allocation 1234
[1234] user@dragon:~$ srun your_application

Anatomy of a Batch Job¶

A batch job requests computing resources and specifies the application(s) to launch on those resources along with any input data/options and output directives. You submit the job, usually in the form of a batch job script, to the batch scheduler. This is the preferred way of using the cluster.

The batch job script is composed of four main components:

• The interpreter used to execute the script

• #SBATCH directives that convey default submission options.

• The setting of environment variables and the modules (if necessary).

• The application(s) to execute along with its input arguments and options.

Here is an example of a batch script that requests 8 task on two nodes, allocating altogether 2GB RAM on each node and is allowed to run for 1 hour. It is assigned to the QOS normal. When the job is launched, it loads environmental module my_module and launches the requested tasks of my_application on the allocated nodes:

#!/bin/bash
#SBATCH -n 8 -N 2
#SBATCH --time=1:00:00
#SBATCH --mem=2G
#SBATCH --qos=normal
module add my_module
srun my_application

When the job is scheduled to run, the resource manager will execute the batch job script on the first allocated node. The srun command will launch additional requested tasks on all the allocated nodes.

The specific instruction how to launch available programs, are presented elsewhere.

Resources Specification¶

On the cluster you must manually specify requested resources. Should you forget to do this, your job will be allocated 1KB RAM and will be allowed to run for 1 minute. You specify the resources as command-line arguments to sbatch or salloc or in the #SBATCH directive of your script.

The basic resources are as follows:

When submitting your jobs you should carefully consider required resources. Specifying too low values puts your job at risk of being terminated and setting it too high wastes cluster resources and decreases priority of your job. Furthermore, small jobs can be launched earlier in order to fill-in the holes is cluster resources, using backfill algorithm.

$ seff JOBID

$ seff 1592
Job ID: 1592
Cluster: dragon
User/Group: user/user
State: COMPLETED (exit code 0)
Cores: 48
Nodes: 4 (12 cores per node)
Tasks: 24 (2 cores per task)
Elapsed Time: 00:08:02
CPU Utilized: 04:36:28
CPU Efficiency: 71.70% of 06:25:36 core-walltime
Max Memory Utilized: 2.74 GB (up to 934.75 MB/node)
Memory Efficiency: 68.46% of 4.00 GB (1.00 GB/node)

Job Names and Output Files¶

All jobs are given both a job identifier and a name, for easier identification in the batch-system. The default name given to a job is the file name of the submit script, which can make it difficult to identify your job, if you use a standard name for your submit scripts. You can give your job a name from inside the script by using the -J option:

#SBATCH -J parameterTest

This will name your job “parameterTest”.

By default — for jobs submitted with sbatch — the output which your job writes to stdout and stderr is written to a file named slurm_%j.out. The %j in the file name will be replaced by the job number SLURM assigns to your job. This ensures that the output file from your job is unique and different jobs do not interfere with each other’s output file.

In many cases the default file name is not convenient. You might want to have a file name which is more descriptive of the job that is actually running — you might even want to include important meta-data, such as physical parameters, into the output filename(s). This can be achieved by using the -o and -e options of sbatch. The -o option specifies the file containing the stdout and the -e option the file containing the stderr. It is good practice to include the %j string into the filenames. That will prevent jobs from overwriting each other’s output files. The following gives an example:

#SBATCH -o calcflow-%j.out
#SBATCH -e calcflow-%j.err

You can give the same filename for both options to get stdout and stderr written to the same file. Also if you simply skip the -e option, the stderr will be written to the same file as the stdout.

If you use job arrays, you may replace the job identifier %j mark with %a, which gives the whole-array identifier and %A for the array element number:

#SBATCH -o myarray-%a_%A.out

Partitions¶

SLURM jobs are executed on compute nodes. Amount and types of resources (CPUs, Memory, disk storage, etc.) may vary between nodes of the clusters. See the HPC Resources page for an overview. To get an overview of the nodes and see how they are doing, login to a cluster and execute:

$ sinfo -N -l

Partitions are used by SLURM to group nodes with an identical or similar configuration. On our cluster—where nodes are identical—you should not worry about partitions. Your job will be assigned to the partition best suited for your purpose.

To manually select a partition use -p option in sbatch or salloc or in the #SBATCH directive of your script:

Job Priorities¶

When you submit a batch job or request resources for interactive one, it is not necessarily started immediately. Instead, Jobs will be ordered in the queue of pending jobs based on a number of factors. The scheduler will always be looking to launch the job that is at the top of the queue. However, if your job does not require many resources, it may be launched earlier, as long as it does not delay the start of ony higher-priority job (this is known as conservative backfill).

The factors that contribute to a job’s priority can be seen by invoking the sprio command. These factors include:

• Fair-share: a number indicating how much you have been using each cluster recently. This is the most important factor, as it ensures that all users will be able to have their job started in reasonable time. You can check your current usage using sshare command:

  $ sshare
     Account       User  RawShares  NormShares  EffectvUsage  FairShare
  ---------- ---------- ---------- ----------- ------------- ----------
  root                                1.000000      1.000000   0.500000
   physics                       1    0.500000      1.000000   0.250000
    physics    username          1    0.125000      0.874713   0.007825
  

• Job size: a number proportional to the quantity of computing resources the job has requested. Smaller and shorter jobs will have higher priority.

• Age: a number proportional to the period of time that has elapsed since the job was submitted to the queue.

• QOS: a number assigned to the requested Quality-Of-Service. See below for details.

Submitting Job Arrays¶

Job arrays are submitted by including the -a or --array option in a call to sbatch, or by including the #SBATCH -a command in your SLURM script. The -a option takes a comma-delimited list of job ID numbers or of one or more pairs of job ID numbers separated by a dash.

Each job in the job array will be launched with the same SLURM script and in an identical environment–except for the value of its array ID. The value of the Array ID for each job in a Job Array is stored in the SLURM_ARRAY_TASK_ID environment variable.

For example, if a job array is submitted with 10 elements, numbered from 1 to 10, the submission command would be the following:

$ sbatch -a 1-10 array_script.sh

You may also specify explicit array indices separated by commas of add a step to the specified range, using “:”. Hence, the following commands are equivalent:

$ sbatch --array 0-20:4 array_script.sh
$ sbatch -a 0,4,8,12,16,20 array_script.sh

An optional parameter, the slot limit, can be added to the end of the -a option to specify the maximum number of job array elements that can run at one time. The slot limit is specified by appending a “%” to the -a option followed by the slot limit value. A twelve element job array with non-sequential array IDs and a slot limit of 3 could be specified as follows:

$ sbatch -a 1-3,5-7,9-11,13-15%3 array_script.sh

Each job included in the job array has its own unique array element value stored in the SLURM_ARRAY_TASK_ID environment variable. The value of each job array element’s array ID can be accessed by the job script just like any other shell environment variable. If the job ran a bash shell script, the job’s array ID information could printed to STDOUT using the following command:

echo "Current job array element's Array ID: ${SLURM_ARRAY_TASK_ID}"

Customizing Data for Job Array Elements¶

A more useful task for the array ID–and the real power of job arrays–would be to use the job’s Array ID as a direct or indirect index into the data being processed by the job array.

One approach to accomplish this would be to use the SLURM_ARRAY_TASK_ID value to provide a custom set of input parameters for job in the job array. To do this, a text file would be created containing multiple lines each of which would consist of a series of space delimited values. In this approach, each line in the data file would contain the input parameters needed by one element of the job array. The SLURM script would then be modified to include a command that would read in the correct line of the data file–based on the SLURM_ARRAY_TASK_ID value of that particular job. While there are many ways to read the appropriate line from the data file, the following serves as a sample implementation assuming that the data file was called data.dat and was located in the same directory as the script that was run for each element of the job array:

PARAMETERS=$(sed "${SLURM_ARRAY_TASK_ID}q;d" data.dat)

Assuming that the executable program/script for the jobs in this array was called command.sh, the SLURM script would launch the program with a line like the following:

./command.sh ${PARAMETERS}

An alternate approach is possible if the unique input parameters needed by each job in the array can be calculated arithmetically. For example, if each instance of the command.sh script needed to loop over a range of values, the SLURM script could calculate the max and min values needed for each job directly–based on the value in the SLURM_ARRAY_TASK_ID environment variable. If each job’s range needed to include 1000 values, this could be done by including commands like the following in the SLURM script:

MAX=$(echo "${SLURM_ARRAY_TASK_ID}*1000" | bc)
MIN=$(echo "(${SLURM_ARRAY_TASK_ID}-1)*1000" | bc)

The data file referred to above (data.dat) would not be needed in this approach, and the SLURM script call to command.sh would be something like the following:

./command.sh ${MIN} ${MAX}

Most Common Environmental Variables¶

Job Specification¶