Bash – Command Line Macros in Comments using SLURM

bashshell-scriptslurm

In my opinion, comments are comments are comments. They should NEVER change program state. Sadly, the people over at SLURM disagree.

SLURM requires the following syntax (or something similar) to exist at the front of a .sh file:

#!/bin/bash

#SBATCH --time=5:00:00    
#SBATCH --ntasks=8
#SBATCH --mem-per-cpu=1024M

MyProgram.exe

The above will submit a request to run the program MyProgram.exe using 8 processors on my school's super computer. I'm currently benchmarking my code and would like to use a command line argument to change the number of processors. Effectually, I want to do the following

#!/bin/bash

#SBATCH --time=5:00:00    
#SBATCH --ntasks=%1
#SBATCH --mem-per-cpu=1024M

MyProgram.exe

where %1 would be a command line argument. I would then call it as sbatch myShScript.sh 123 which would use 123 processors. This, of course, does not work because bash interprets the #SBATCH --ntasks=%1 line as a comment (which it is) and never subs in my command line argument.

Is there a way to trick SLURM or BASH into subbing in my command line argument?

Best Answer

All the processing done by SLURM (by sbatch, specifically) is done before bash is invoked, so bash won't help you here. The script could be in any language, it wouldn't matter: the #SBATCH are only coincidentally bash comments, what matters is that they're sbatch directives.

Options can be specified in the file so as to provide a convenient way to always use the same parameters for a particular script. If you want to use different options, pass them on the command line of sbatch. You can write a wrapper script that runs sbatch if you want to build up options from certain specific parameters. You can pass the job script as standard input (a here document is convenient) instead of keeping it in a separate file if you prefer.

#!/bin/sh
sbatch --time=5:00:00 --ntasks="$1" --mem-per-cpu=1024M <<'EOF'
#!/bin/sh
MyProgram.exe
EOF
Related Question