# Slurm job submission errors

**URL:** <https://discourse.openondemand.org/t/slurm-job-submission-errors/2468>\
**Category:** Get Help\
**Tags:** question\
**Created:** [February 7, 2023, 3:19pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468 "2023-02-07T15:19:52Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 3:19pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/1 "2023-02-07T15:19:52Z")

</div>

I am trying to submit a Slurm job through OOD and I am getting the following error:

An error occurred when submitting jobs for simulation 1: sbatch: error: s\_p\_parse\_file: cannot stat file /opt/slurm/slurm.conf: No such file or directory, retrying in 1sec up to 60sec  
sbatch: error: ClusterName needs to be specified  
sbatch: fatal: Unable to process configuration file

I am using the following as documentation:  
[Setup SLURM](https://osc.github.io/ood-documentation/latest/installation/resource-manager/slurm.html)

The OOD server is a separate server from the cluster.  
The contents of my .yml file for the cluster on the OOD server is as follows:

```bash
---
v2:
  metadata:
    title: "F&M Research Cluster"
    url: "https://dorcfandm.github.io/rcs.github.io/"
  login:
    host: "rcs-scsn.fandm.edu"
  job:
    adapter: "slurm"
    cluster: "rcs-sc"
    bin: "/usr/bin"
    conf: "/opt/slurm/slurm.conf"

```

I also tried to test the setup via the command-line using:  
su $USER -c ‘scl enable ondemand – bin/rake test:jobs:cluster1 RAILS\_ENV=production’

with the following output:

```bash
 Rails Error: Unable to access log file. Please ensure that /var/www/ood/apps/sys/dashboard/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /var/www/ood/apps/sys/dashboard/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
mkdir -p /home/user/test_jobs
Testing cluster 'cluster'...
Submitting job...
[2023-02-07 10:04:42 -0500] INFO "execve = [{\"SLURM_CONF\"=>\"/opt/slurm/slurm.conf\"}, \"/usr/bin/sbatch\", \"-D\", \"/home/user/test_jobs\", \"-J\", \"test_jobs_cluster\", \"-o\", \"/home/user/test_jobs/output_cluster_2023_02_07t10_04_42_05_00_log\", \"-t\", \"00:01:00\", \"--export\", \"NONE\", \"--parsable\", \"-M\", \"rcs-sc\"]"
rake aborted!
OodCore::JobAdapterError: sbatch: error: s_p_parse_file: cannot stat file /opt/slurm/slurm.conf: No such file or directory, retrying in 1sec up to 60sec
sbatch: error: ClusterName needs to be specified
sbatch: fatal: Unable to process configuration file
/opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.29/gems/ood_core-0.22.0/lib/ood_core/job/adapters/slurm.rb:477:in `rescue in submit'
/opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.29/gems/ood_core-0.22.0/lib/ood_core/job/adapters/slurm.rb:415:in `submit'
/var/www/ood/apps/sys/dashboard/lib/tasks/test.rake:30:in `block (4 levels) in <top (required)>'

Caused by:
OodCore::Job::Adapters::Slurm::Batch::Error: sbatch: error: s_p_parse_file: cannot stat file /opt/slurm/slurm.conf: No such file or directory, retrying in 1sec up to 60sec
sbatch: error: ClusterName needs to be specified
sbatch: fatal: Unable to process configuration file
/opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.29/gems/ood_core-0.22.0/lib/ood_core/job/adapters/slurm.rb:335:in `call'
/opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.29/gems/ood_core-0.22.0/lib/ood_core/job/adapters/slurm.rb:244:in `submit_string'
/opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.29/gems/ood_core-0.22.0/lib/ood_core/job/adapters/slurm.rb:475:in `submit'
/var/www/ood/apps/sys/dashboard/lib/tasks/test.rake:30:in `block (4 levels) in <top (required)>'
Tasks: TOP => test:jobs:cluster
(See full trace by running task with --trace)

```

/opt/slurm/slurm.conf is indeed the correct location for slurm.conf on our cluster and the file permission is set to world readable. There is only one cluster setup and it’s name is rcs-sc

Thank you in advance for any help

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 4:24pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/2 "2023-02-07T16:24:37Z")

</div>

Hi and sorry for the trouble.

Looking at the log outputs, it looks like something is off with the `cluster` name you are submitting to, which needs to match the name given in SLURM.

What is the cluster name in the `slurm.conf` file?

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 4:29pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/3 "2023-02-07T16:29:30Z")

</div>

On the cluster in slurm.conf the cluster name is:

```bash
################################################
# CONTROL #
################################################
ClusterName=rcs-sc

```

On my OOD server (which is not part of the cluster), I set cluster: “rcs-sc”

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 4:56pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/4 "2023-02-07T16:56:15Z")

</div>

Do you have those `slurm.conf` files on the actual web node running OOD? They will need to be there.

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 5:01pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/5 "2023-02-07T17:01:07Z")

</div>

No I don’t have it on the OOD webserver. It wasn’t clear from the documentation that it needed to be on the webserver. Does Slurm also need to be installed on the webserver too or do I just need to copy the slurm.conf from my cluster to the webserver?

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 5:02pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/6 "2023-02-07T17:02:28Z")

</div>

You won’t need `slurm` on the web host.

I apologize that wasn’t clear in the docs. The way to think about it is Open OnDemand needs to know the configuration details of the cluster in order to submit jobs to it, and the cluster needs to know its own configuration details in order to manage the jobs that are submitted to it.

So, this is why you will need that `slurm.conf` in two places, but won’t need to do a full SLURM install on the web host with OOD.

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 5:04pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/7 "2023-02-07T17:04:01Z")

</div>

Ok, sounds good. I will copy it over and close the ticket if that solves the issue

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 5:19pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/8 "2023-02-07T17:19:51Z")

</div>

My apologies again, I can’t even understand our own docs on this either.

So, for the current setup you have you _will need the SLURM binaries on the web host with OOD_.

I’m really sorry for the confusion.

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 5:25pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/9 "2023-02-07T17:25:35Z")

</div>

Now, there is a way around having that binary installed on the web host with OOD, but it is not totally documented yet though it works in `2.0`.

If you look here:  
[https://osc.github.io/ood-documentation/develop/installation/resource-manager/slurm.html](https://osc.github.io/ood-documentation/develop/installation/resource-manager/slurm.html)

You’ll see an option for `submit_host` that does allow you to skip having the binary on the web node with OOD and instruct the `sbatch` commands to be run on another host.

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 5:27pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/10 "2023-02-07T17:27:12Z")

</div>

I will add the submit\_host option to my cluster.yml and see if that helps

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 5:39pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/11 "2023-02-07T17:39:14Z")

</div>

My current cluster.yml file is:

```bash
---
v2:
  metadata:
    title: "F&M Research Cluster"
    url: "https://dorcfandm.github.io/rcs.github.io/"
  login:
    host: "rcs-scsn.fandm.edu"
  job:
    adapter: "slurm"
    submit_host: "rcs-scsn.fandm.edu"
    bin: "/usr/bin"
    conf: "/opt/slurm/slurm.conf"

```

But now it errors:  
An error occurred when submitting jobs for simulation 1: sbatch: error: get\_addr\_info: getaddrinfo() failed: Name or service not known  
sbatch: error: slurm\_set\_addr: Unable to resolve “rcs-sc.cluster”  
sbatch: error: slurm\_get\_port: Address family ‘0’ not supported  
sbatch: error: Error connecting, bad data: family = 0, port = 0  
sbatch: error: slurm\_persist\_conn\_open\_without\_init: failed to open persistent connection to host:rcs-sc.cluster:6819: No error  
sbatch: error: Sending PersistInit msg: No error  
sbatch: error: get\_addr\_info: getaddrinfo() failed: Name or service not known  
sbatch: error: slurm\_set\_addr: Unable to resolve “rcs-sc.cluster”  
sbatch: error: slurm\_get\_port: Address family ‘0’ not supported  
sbatch: error: Error connecting, bad data: family = 0, port = 0  
sbatch: error: Sending PersistInit msg: No error  
sbatch: error: DBD\_GET\_CLUSTERS failure: No error  
sbatch: error: Problem talking to database  
sbatch: error: ‘rcs-sc’ can’t be reached now, or it is an invalid entry for --cluster. Use ‘sacctmgr list clusters’ to see available clusters.

1. Do I need to rebuilt the portal every time I change cluster.yml?

2. I suspect the issue is with this line in our slurm.conf  
AccountingStorageHost=rcs-sc.cluster

That name (rcs-sc.cluster) resolves fine on the cluster (to a local 10. address) but it won’t resolve from any other server outside the cluster

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 5:50pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/12 "2023-02-07T17:50:15Z")

</div>

You shouldn’t have to rebuild the portal changing the `cluster.yml`, though restarting `httpd` may not be a bad idea just to be safe.

I’m not sure why that name resolution is failing, though. That nothing outside the cluster can resolve it is clearly the problem though.

What happens if you set the cluster `ip` and `host` in `/etc/hosts` and try again? That will at least clue you in on if it’s a config problem or something with name resolution.

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 5:59pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/13 "2023-02-07T17:59:13Z")

</div>

So I’m slowly moving in the correct direction. I edited the slurm.conf on the OOD server  
and just removed the .cluster portion on that line and it was closer to working. Now I’m getting an error about keys:

An error occurred when submitting jobs for simulation 1: No ECDSA host key is known for rcs-scsn.fandm.edu and you have requested strict checking.  
Host key verification failed.

Through the OOD webportal I can start an ssh session to the cluster as an actual user. I also know that the username I am using has all the keys synced up on the cluster. I can also ssh directly from the commandline on the OOD server as the username.

Is the request being made under a different user (like ood) and I need to add that user to our cluster?

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 7, 2023, 7:23pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/14 "2023-02-07T19:23:06Z")

</div>

I’m not sure. This may a problem where something failed to sync.

Do you see an entry for the cluster in the `known_hosts` on the OOD web host?

If you manually add the public key for the cluster to the web node’s `known_hosts`, what happens if you try to submit then? It looks like it just needs that host and its key added. You can use `ssh-keyscan` with the cluster `hostname` to grab them.

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 7, 2023, 8:30pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/15 "2023-02-07T20:30:08Z")

</div>

I tried the following

1. From the command line on the OOD server do ssh kegen -t ECDSA
2. SSH copy id to our cluster
3. SSH’s to cluster and verified I didn’t need password.
4. From the OOD webportal started an ssh session to the cluster and verified it didn’t ask for password.
5. Tested job submit and it still failed same issue

I dug around the OOD discourse pages and found that apparently there’s an option (one that I didn’t see in the documentation) you can add to the cluster.yml file  
strict\_host\_checking:

I set this value to false and then I could submit jobs and also see jobs in the queue.

I also noticed that when running the rake test mentioned above it seems to be setting "UserKnownHostsFile=/dev/null" which I think is also a problem.  
Is there a setting I can add in the configuration file to host file to something like /home/$USER/.ssh/known\_hosts and then maybe I can turn strict checking back on? My concern is that by turning off strict checking that it presents some security risks

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [February 8, 2023, 5:23pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/16 "2023-02-08T17:23:59Z")

</div>

I don’t think I see anything about ensuring the cluster and its public key have been added to the web host’s `known_hosts` file in the updates above though.

Could you see if the cluster has an entry in the web host’s `known_hosts` file to start?

---

<div class="post-metadata">

**Author:** ![aweaver1fandm](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/aweaver1fandm/32/1438_2.png) [@aweaver1fandm](https://discourse.openondemand.org/u/aweaver1fandm)\
**Post date:** [February 9, 2023, 2:51pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/17 "2023-02-09T14:51:23Z")

</div>

That did indeed work and I did not need strict\_host\_checking: false.

For anyone else with this problem, trying to submit jobs through Slurm to a cluster when OOD is hosted on a different server from the cluster here is what I believe to be a full fix.

1. Copy the cluster’s slurm.conf file to the server hosting OOD  
Note: In our slurm.conf file we had one line that referenced \<servername\>.cluster  
We modified the slurm.conf on the OOD server to remove the .cluster part  
because it caused some errors as that domain name was local to our  
cluster

2. In the cluster .yml file make sure under the job section you define submit\_host:  
According to one of the comments in this thread from OOD folks  
if you don’t define that, then you need Slurm running on the OOD server

3. On the OOD server, check /etc/ssh to see if a known\_hosts file exists.  
a) If not create one  
b) Add the cluster’s public key to the known\_hosts file on the OOD server.  
When I ran some of the rake tests I noticed some comments about  
ECDSA keys so we manually copied/pasted our cluster’s public ECSDA  
key into the OOD server known\_hosts. The cluster’s key file was also in the  
/etc/ssh directory.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [August 8, 2023, 2:51pm UTC](https://discourse.openondemand.org/t/slurm-job-submission-errors/2468/18 "2023-08-08T14:51:42Z")

</div>

Closed on August 8, 2023, 2:51pm UTC

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.