Federation support for slurm

Get Help
1

We have slurm federation enabled on our production cluster, with three separate slurm clusters. The idea behind federation is that users can either submit without specifying cluster, in which case each of the clusters is eligible to run the job depending on resources and availability. We’re trying to make that less exposed to users in OOD. I’m aware we can add a cluster dropdown and program that. But that’s not what we want. What are the implications of removing the cluster field from our cluster definition file? Based on this, it’s not clear that this will work or be desirable: Slurm — Open OnDemand 4.2.0 documentation

Essentially, I want a way to change the behavior so that the -M “cluster” flag is removed from the sbatch submission when an OOD job is submitted.

To make matters more interesting, we need to keep separate cluster definitions in OOD as it is - we have a dev cluster and a teaching cluster that are not part of federation. But for the slurm federated clusters, I’d like to just have the “master” cluster be configured in OOD, and let the sbatch command go into the federated environment and let slurm determine where to send the job.

2

If you get rid of cluster configuration, you’ll need to replace it with the conf configuration and place a copy of the slurm .conf file for each corresponding cluster somewhere on the web node. That’s how we run it at OSC.

3

Thanks Jeff. Could you expand on that? We have a yaml file for our prod cluster (which is now federated with the 2 sibling clusters, as I mentioned), a yaml file for our dev cluster, and a yaml file for our teaching cluster all in /etc/ood/config/clusters.d. Each cluster yaml has a submit host defined under “login”, and under “job” we have variations of this:

adapter: “slurm”
cluster: “slurm_cluster”
bin: “/usr/bin”
conf: “/somepath/slurm.conf”

So the conf files are all defined in the yaml files. What would we be changing in your example?

4

If you already have the conf supplied, I think you should be able to just remove the cluster entry for all YML files, though @tdockendorf would know more.

A quick read of federation documentation says the you get unique job ids across all clusters, so I think that’s the only real requirement here. If OOD issues squeue -j 123 against the “origin” cluster and it returns information from a sibling cluster you should be good to go.

Do you already have the sibling clusters enabled in OOD in some form or fashion? Beyond just getting OOD to communicate to it correctly (which I think it will given unique IDs) then you may need to update allowlists like the shell allowlist to allow shelling into those compute nodes and the /rnode_uri and /node_uri allowlists to allow proxying to them.

5

If you want the submission to not use -M <cluster> then omit cluster key. This is what we do:

  job:
    adapter: slurm
    bin: "/usr/bin"
    conf: "/etc/slurm/slurm-cardinal.conf"

Each cluster submits jobs without -M and uses the cluster-specific config so Slurm client knows where to submit without the -M flag.

6

Thanks Jeff and Trey. I think that did it. I’ll do some more testing and reply if I encounter problems. But my initial test job with the cluster line removed indeed omitted -M <cluster>

7

We have started seeing some odd behavior after implementing multiple clusters on OOD. Some jobs are seemingly failing silently. Slurm keeps them running but the OOD interface shows “Completed” and the output.log file does not indicate any noticeable failure.

I have configured several of our apps to send “-M all” as a hardcoded option. Early testing with that showed that it allowed jobs to run within OOD, but additional tests are failing silently again.

It’s worth noting that ood itself points to the first cluster in the Federation, and the submit host also uses that first cluster. Jobs submitted there (especially with “-M all”) will submit to all the clusters in the federation. From what we’ve seen, jobs submitted to the original cluster work fine in OOD, but jobs that land on the second federated cluster sometimes do not work.

Is there anywhere else I can look?

8

Do these jobs return with a simple squeue -j 1234 from the command line? If Slurm doesn’t return anything we assume it’s completed.

So I guess I would try to recreate the same behavior on the CLI first with or without the -M option.

9

Thanks Jeff. I have reached out to SchedMD as well to confirm that our federation config isn’t causing unintended problems that would lead to this behavior. Today our testing is showing correct behavior, but we’ll keep testing. Thanks.