AWS parallel cluster(Slurm) + Open OnDemand, Slurm jobs stuck in “user env retrieval failed requeued held” during compute node boot-u

Get Help
1

Environment Details:
AWS ParallelCluster 3.16.1
Slurm: 25.11.8
OS: Ubuntu 22.04.5 LTS
Open OnDemand: 4.2.4

Issue Summary
When a job is submitted through Open OnDemand to the AWS parallel cluster, it is requeued and held with the reason “user env retrieval failed requeued held” whenever the submission triggers a new compute node to boot. Our compute nodes take about 4 minutes to become ready due to additional configuration steps performed at boot time.

If a compute node is already in UP state with sufficient resources, jobs submitted via sbatch from Open OnDemand complete the user-environment retrieval steps successfully and run without issue. The failure seems to be specifically tied to the node boot/launch window.

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
20 default sbatch foolbar PD 0:00 1 (user env retrieval failed requeued held)
19 default sys/dash foolbar PD 0:00 1 (user env retrieval failed requeued held)

Open OnDemand app log, it shows export=NONE
App 456545 output: [2026-09-29 15:32:10 +0000 ] INFO “execve = [{}, \“ssh\”, \”-oBatchMode=yes\“, \”-oUserKnownHostsFile=/dev/null\“, \”-oStrictHostKeyChecking=no\“, \“foolbar\”, \”/opt/slurm/bin/sbatch\“]”, \“–export\”, \“NONE\”, \“–parsable\”

I believe this may be related to a change in how Slurm handles the user-environment retrieval timeout:

https://support.schedmd.com/show_bug.cgi?id=5037

start from Slurm 25.05 deprecates GetEnvTimeout from slurm.conf and removes arguments from the sbatch --get-user-env option

https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.05.md

https://github.com/jupyterhub/batchspawner/issues/319

Failure Behavior changes so that any failure or timeout while loading the user environment automatically requeues and holds the job with a specific reason instead of letting it continue.

Is this a known interaction between AWS ParallelCluster dynamic node provisioning and the Slurm 25.11 user-environment retrieval or just Open OnDemand?

Is there a fix or supported configuration change planned?

https://aws.amazon.com/blogs/hpc/deploying-open-ondemand-on-aws-and-integrating-with-parallel-cluster/

However, there is a “Copy environment” option under Job Options for Open OnDemand Job Composer.
If NOT check this option, the job will fail with “–export NONE”.
If check this option, the job will complete with “–export ALL”.

Is there a way to set the system wide “Copy environment” option for Open OnDemand?

Thanks

2

Hi and welcome!

Yes, you can set copy_environment at the cluster level which will apply to all apps that use that cluster:

or at the applicatiion level in submit.yml.erb:

batch_connect:
  # ... your batch connect options
script:
  copy_environment: true
3 
Thanks a lot for the help.
The cluster level setting copy_environment is not working.

Setting at the applicatiion level in submit.yml.erb works.

cluster.yml:

v2:
  metadata:
    title: "HPC Slurm Cluster"
  login:
    host: "head.my_center.edu"
  job:
    adapter: "slurm"
    cluster: "my_slurm_cluster"
    copy_environment: true
    bin: "/opt/slurm/bin"
    conf: "/opt/slurm/etc/slurm.conf"
4

OK thanks, I’ll look into the cluster level config.