Problems using `--constraint` with booleans in `submit.yml.erb`

Get Help
1

In submit.yml.erb, I’m attempting to set a --constraint with some boolean operators. Something like this:

script:
  native: 
    <%- if node_type == "a100" -%>
    - "--constraint='[A100-40GB|A100-80GB]'"
    # ... etc ...

I’ve verified that this exact --constraint works for us in normal sbatch and salloc. But in OOD, the job submission fails with: " Failed to submit session with the following error: sbatch: error: Batch job submission failed: Invalid feature specification".

In my troubleshooting, it seems like the | character might not be passed correctly, but I am still not sure. Do any of you know if | needs special handling in an .erb file?

Thanks,
Ron

2

Could you post what OOD submitted with the job_scirpt_options.json file in the session data?

My guess is those single quotes are throwing things off.

3

Sure thing, here it is:

{
  "job_name": "sys/dashboard/dev/bc_jupyter_rh9",
  "workdir": "/storage/home/hcodaman1/rrahaman6/ondemand/data/sys/dashboard/batch_connect/dev/bc_jupyter_rh9/output/65fa2626-8888-4401-ad81-1c388446a25a",
  "output_path": "/storage/home/hcodaman1/rrahaman6/ondemand/data/sys/dashboard/batch_connect/dev/bc_jupyter_rh9/output/65fa2626-8888-4401-ad81-1c388446a25a/output.log",
  "shell_path": "/bin/bash",
  "native": [
    "--nodes=1",
    "--ntasks-per-node=1",
    "--gpus-per-node=1",
    "--constraint='[A100-40GB|A100-80GB]'"
  ],
  "wall_time": 3600,
  "email_on_started": false
}
4

Yeah I can see the issue with those single quotes. That file shows you exactly what is being handed to SLURM to run and it’s going to fall over when it hits those.

Thinking on this, I’ve seen this issue previously in many forms and I think I understand why now after searching the OOD code. Simply put, people fill that out as if a shell will be issuing a command from that String, but this is not true. Instead, OOD will use ruby to call exec() for sbatch.

This is fundamentally different than if we were using a shell and just calling sbatch as then we would need those single quotes to prevent the shell from interpreting | as a pipe operator. However, when we call exec() that’s not the case, instead we get an Array of arguments being passed (which you can see in what you posted under native) so, no shell is ever processing that and removing those quotes. They literally get passed and SLURM doesn’t understand what the single quote is and returns the “…Invalid feature specification” because SLURM has no such specification for single quotes.

That’s a lot, but i am trying to capture why this happens for future users to see as well.

The point is this: If using native then do NOT use shell quotes

For your particular problem just remove the single quotes:

script:
  native: 
    <%- if node_type == "a100" -%>
    - "--constraint=[A100-40GB|A100-80GB]"
    # ... etc ...