Submitit with SLURM sub-scheduling

[giorgos117]

Is it possible to use submitit with SLURM sub-scheduling? I work on a system where each new SLURM job allocates at least one node. If possible, I would like to use submitit with SLURM sub-scheduling to get e.g. 4 small SLURM jobs per node.

Best,

George

[gwenzek]

Hi, I'm not aware of this SLURM feature, could you share some link to docs and example slurm commands ?
In general submitit has escape hatches to pass arbitrary CLI argument to slurm so if you know the right sbatch/srun command, you should be able to write them with submitit

[giorgos117]

Hello and thanks for the reply,

With the following script, 4 small jobs run in a single node:

`#!/bin/bash -l
#SBATCH --time=1:0:0
#SBATCH --account=def-your-project-account
#SBATCH --nodes=1
#SBATCH -p cpu
#SBATCH -q test
#SBATCH --ntasks=4 # number of tasks
#SBATCH --ntasks-per-node=4 # number of tasks per node
#SBATCH --cpus-per-task=32 # number of cores per task

srun -n 1 --exact ./test-task1 > output1.txt &
srun -n 1 --exact ./test-task2 > output2.txt &
srun -n 1 --exact ./test-task3 > output3.txt &
srun -n 1 --exact ./test-task4 > output4.txt &`

I couldn't find a relevant link from the SLURM docs, so I include one from the docs of an HPC system:

https://docs.lxp.lu/first-steps/handling_jobs/#sub-scheduling

Can I make submitit include multiple srun commands in the sbatch file?