Implement srun prefix for example-app scripts, for Slurm-managed instances?

jasonbuechler · March 28, 2023, 5:51pm

This isn’t really a feature request, but also sort-of is. Also, it’s kind of a PSA for anyone wanting more accurate sacct/sstat data.

I don’t know if other resource managers have a similar mechanic, but Slurm needs an extra touch of tlc for job-steps within sbatch jobscripts, to get full details on resource usage.

To partially quote the answer for this stackoverflow question:

First, it will bring better reporting of the resource usage ; the sstat command will provide real-time resource usage for processes that are started with srun , and each step (each call to srun) will be reported individually in the accounting.

For quick reference, prefixes would be positioned:

github.com

OSC/bc_example_jupyter/blob/94d29b4ba705cfe3b17df8ce78f584a4a2f986c5/template/script.sh.erb#L29


      
          
          
# List loaded modules
          module list
          <%- end -%>
          
          
# Benchmark info
          echo "TIMING - Starting jupyter at: $(date)"
          
          
# Launch the Jupyter Notebook Server
          set -x
          jupyter notebook --config="${CONFIG_FILE}" <%= context.extra_jupyter_args %>

github.com

OSC/bc_osc_matlab/blob/fee6600e6f934b109c5d2442091a90ef33f84e12/template/script.sh.erb#L44


      
          
          
# Launch MATLAB
          <%- if gpu -%>
          module load intel/16.0.3 virtualgl
          module list
          set -x
          vglrun matlab -desktop -nosoftwareopengl
          <%- else -%>
          module list
          set -x
          matlab -desktop
          <%- end -%>

NOTE: If you are using a newer version of rstudio-server and are running rstudio-server directly from template/script.sh.erb instead of using Singularity, you’d prefix the “rstudio-server” line. (And not the rsession line, which might initially seem the right target.)

github.com

OSC/bc_example_rstudio/blob/a8909ee46ccdc5577c22efd8ba6a3aa5e3694041/template/script.sh.erb#L57


      
          export TMPDIR="$(mktemp -d)"
          
          
mkdir -p "$TMPDIR/rstudio-server"
          python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
          chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"
          
          
set -x
          # Launch the RStudio Server
          echo "Starting up rserver..."
          
          
singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
           --www-port "${port}" \
           --auth-none 0 \
           --auth-pam-helper-path "${RSTUDIO_AUTH}" \
           --auth-encrypt-password 0 \
           --rsession-path "${RSESSION_WRAPPER_FILE}"
          
          
echo 'Singularity as exited...'

gbyrket · March 28, 2023, 7:49pm

Hi Jason.

Thanks for your post. I’ll discuss with the team to determine how we proceed.

Thanks,
-gerald