Auto_accounts and auto_qos blank after upgrading to 4.1

Get Help
1

Hello,

We just upgraded from 4.0 to 4.1 today. After upgrading, the account (auto_

accounts) and qos (auto_qos) fields no longer populate.

Strangely, the partition (auto_queues) field does populate.

I have looked at /var/log/ondemand-nginx/$USER/error.log but do not see the cause of the issue.

As the $USER in question, i ran the following command which completed successfully, verifying the databases can be read:

sacctmgr -nP show users withassoc format=account,cluster,partition,qos where user=$USER

I will attempt to upload our cluster config, form.yml, and submit.yml.

I appreciate any help. Thank you!

Dave

env.txt (184 Bytes)

submit.yml (27.1 KB)

Storrs_HPC.yml (27.1 KB)

form.yml (27.1 KB)

2

Maybe the same issue as here:

3

Yes I would guess it’s issuing cluster=storrs_hpc when it should be just cluster=storrs. I think it is as @Jacob-BM suggests - you need to change the cluster filename in from storrs_hpc to just storrs.

4

The cluster name is “slurm” and is defined in the Storrs_HPC.yml file:

job:
cluster: "slurm"

So, starting with 4.1 it’s no longer issuing the cluster name based on this setting, but rather the filename? Wouldn’t that cause auto_queues to fail also? Because auto_queues was working, as was shell access. When I change the name if this file, I lose the apps list on the dashboard, even after restarting the webserver. Does this require a rebuild of the dashboard?

5

If you have cluster defined there then it should be using that. you wouldn’t have to rebuild the dashboard, only reconfigure all the applications to use the new cluster name.

I’ll take a look at the code and report back.

In the interim, can you take another look at this file and see the commands it’s issuing with storrs_hpc.yml or just storrs.yml

6

OK, I was wrong here. Defining cluster here sets the -M <cluster> flag which is only used for a few commands.

Is slurm the actual cluster name? I.e., the ClusterName as defined in the slurm .conf file?

The name of this configuration file should now match the ClusterName configuration from the slurm .conf file. Terribly sorry about the issue and confusion, we had to make that change to make the adapters less global and more instance/cluster specific.

7

Hi Jeff,

From the user error file, it indeed looks like the apps are trying to assign the cluster name as “Storrs_HPC” so that makes sense.

You are correct, “slurm” is the name defined by ClusterName in the slurm.conf file. When I change the name of:

/etc/ood/config/clusters.d/Storrs_HPC.yml

to

/etc/ood/config/clusters.d/slurm.yml,

all the apps disappear from the dashboard. I assume that is because I need to edit all the apps’ form.yml to change:

cluster: "Storrs_HPC"

to

cluster: "slurm"

I will make these edits when I get a chance and report back.

Thanks!
Dave

8

Jeff,

I just wanted to report back that I made the changes and everything appears to be working correctly. Thanks again for your help!

Best,
Dave

9

Glad to hear it, sorry about the issues!