# Invalid gres with batch\_connect

**URL:** <https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577>\
**Category:** Get Help\
**Created:** [March 30, 2023, 4:20pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577 "2023-03-30T16:20:17Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrisdotcollins](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/chrisdotcollins/32/1563_2.png) [@chrisdotcollins](https://discourse.openondemand.org/u/chrisdotcollins)\
**Post date:** [March 30, 2023, 4:20pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/1 "2023-03-30T16:20:17Z")

</div>

Hi,

I have a Jupyter app based on the example batch\_connect approach which works fine, but when I try and make use of GPU by adding in gres, it is failing at submit with an invalid gres error:

“sbatch: error: Invalid generic resource (gres) specification”

The job\_script\_options shows the following, which from a slurm point of view looks like what I’d expect:

“native”: [  
“-n”,  
“1”,  
“-J”,  
“OODJupyter”,  
“-p”,  
“gpu”,  
“–gres=gpu:1”,  
“–exclusive”  
]

I’ve tried various combinations around how to specify the gres option, but to no avail.

A quick google for ondemand and the error reveals one page [ondemand gpu request error Nov 2021 | Ohio Supercomputer Center](https://www.osc.edu/resources/technical_support/known_issues/ondemand_gpu_request_error_nov_2021) but no hint at what that problem was (probably unrelated!).

Any assistance would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [March 30, 2023, 6:43pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/2 "2023-03-30T18:43:35Z")

</div>

Nothing is jumping out on the OOD side but I’d be curious to know if SLURM is configured for GPUs.

I think the field you want to ensure is set would be `GresTypes=gpu` to start.

---

<div class="post-metadata">

**Author:** ![chrisdotcollins](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/chrisdotcollins/32/1563_2.png) [@chrisdotcollins](https://discourse.openondemand.org/u/chrisdotcollins)\
**Post date:** [March 30, 2023, 8:30pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/3 "2023-03-30T20:30:00Z")

</div>

Thanks for following up. I think things are set up correctly. We have `GresTypes=gpu` set in our slurm config, and a typical node is set with `Gres=gpu:tesla:1`

A salloc started with a similar `--gres=gpu:1` included run on the same node/partition seems to work (taken from scontrol show job):

```auto
   JOB_GRES=gpu:tesla:1
     Nodes=gpu01 CPU_IDs=14 Mem=4000 GRES=gpu:tesla:1(IDX:0)

```

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [March 30, 2023, 9:11pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/4 "2023-03-30T21:11:37Z")

</div>

Thanks for the info. I am curious, what happens if you alter that script to fit the pattern in what worked for you, so:

```auto
...
"--gres=gpu:tesla:1",
...

```

---

<div class="post-metadata">

**Author:** ![chrisdotcollins](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/chrisdotcollins/32/1563_2.png) [@chrisdotcollins](https://discourse.openondemand.org/u/chrisdotcollins)\
**Post date:** [March 30, 2023, 9:35pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/5 "2023-03-30T21:35:37Z")

</div>

Unfortunately, same invalid gres result

```auto
  "native": [
    "-n",
    "1",
    "-J",
    "OODJupyter",
    "-p",
    "gpu",
    "--gres=gpu:tesla:1",
    "--exclusive"
  ]

```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 31, 2023, 1:37pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/6 "2023-03-31T13:37:57Z")

</div>

If you remove OOD from the picture, what’s are the arguments you’d have to give for just a shell script?

You can see in the `/var/log/ondemand-nginx/$USER/error.log` the command you’re actually issuing. You can search for `execve`. This is the command we’re issuing.

I’d suggest finding what the command should be just from command line testing. Also - you can try `--gpus-per-node` do the same thing?

---

<div class="post-metadata">

**Author:** ![chrisdotcollins](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/chrisdotcollins/32/1563_2.png) [@chrisdotcollins](https://discourse.openondemand.org/u/chrisdotcollins)\
**Post date:** [March 31, 2023, 2:58pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/7 "2023-03-31T14:58:23Z")

</div>

Thanks for the info -

The execve line looks correct as far as I can tell:

```auto
"execve = [{}, \"sbatch\", \"-D\", \"BLAH\", \"-J\", \"sys/dashboard/dev/jupyterprod\", \"-o\", \"BLAH\", \"-A\", \"viper-standard-user\", \"-t\", \"01:00:00\", \"--export\", \"NONE\", \"-n\", \"1\", \"-J\", \"OODJupyter\", \"-p\", \"gpu\", \"--gres=gpu:tesla:1\", \"--exclusive\", \"--parsable\"]"

```

And with --gpus-per-node=1 same error:

```auto
"execve = [{}, \"sbatch\", \"-D\", \"BLAH\", \"-J\", \"sys/dashboard/dev/jupyterprod\", \"-o\", \"BLAH\", \"-A\", \"viper-standard-user\", \"-t\", \"01:00:00\", \"--export\", \"NONE\", \"-n\", \"1\", \"-J\", \"OODJupyter\", \"-p\", \"gpu\", \"--gpus-per-node=1\", \"--exclusive\", \"--parsable\"]"

```

So far I’ve been using a jupyter batch\_connect but thought I’d test with a task submitted through job composer. A basic test job fails here with the same gres error, but the exact same script works when submitted directly with sbatch from the command line (and has the expected gres):

```auto
#!/bin/bash
#SBATCH -J testcc
#SBATCH -p gpu
#SBATCH --gpus-per-node=1

sleep 60
echo $HOSTNAME

```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 31, 2023, 3:05pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/8 "2023-03-31T15:05:11Z")

</div>

OK - then I’d suspect the flags `--exclusive` and `--EXPORT NONE`.

I assume you’re submitting the script with just.

```auto
sbatch delme.sh

```

Can you remove the `#SBATCH` comments and try with variations of command line arguments?

```auto
sbatch --exclusive -p gpu --gres=gpu:tesla:1 delme.sh
sbatch --export NONE --exclusive -p gpu --gres=gpu:tesla:1 delme.sh
sbatch --export NONE -p gpu --gres=gpu:tesla:1 delme.sh
sbatch -p gpu --gres=gpu:tesla:1 delme.sh

```

Also be sure that you’re submitting from the web host itself (the host that OOD is installed on). You may have a slurm conf mismatch or similar. In any case, it’s just good to replicate the commands on the same system to track down what’s going on.

---

<div class="post-metadata">

**Author:** ![chrisdotcollins](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/chrisdotcollins/32/1563_2.png) [@chrisdotcollins](https://discourse.openondemand.org/u/chrisdotcollins)\
**Post date:** [March 31, 2023, 4:04pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/9 "2023-03-31T16:04:11Z")

</div>

You are on to something there! Sorry, I should have tested on the same host but generally have our OOD server restricted to direct user logins so by force of habit have just been submitting the comparison test jobs from another host. But anyway, it does indeed fail from the command line on that host.

I’m running configless slurm, and it looks like for some reason, gres.conf hasn’t made it to the OOD server host. I just need to give it a kick and work out why!

Thanks so much for the help, hopefully this will help someone else avoid the same mistake in future!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [September 27, 2023, 4:04pm UTC](https://discourse.openondemand.org/t/invalid-gres-with-batch-connect/2577/10 "2023-09-27T16:04:35Z")

</div>

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.