OnDemand is running in a vm that does not have the scheduler installed on it (lsf in this case) so for OOD to submit/query lsf I will need to set “bin_overrides” in my cluster config to wrapper scripts for the various lsf commands that ssh into a login node that does have lsf to run.
In OnDemand 1.7 or 1.8 (still undecided) we should support a config option to execute these commands via ssh on the specified host, at least for Slurm and LSF.
After reading through a fair amount of the ood_core source code, I can see it would be a bit of a pain to add this in (properly, not hacky). Wrapper scripts will be fine for now.
FWIW, The reason I am in this weird scenario is the cluster in question is running on Power9 architecture so to get OOD involved, I have a single x86 node that runs it. I figure sending the jobs to one of the power9 head-nodes is easier than trying to get lsf installed on the x86 node and playing nice, even if it is sshing into the head-node each time it needs to interact with it.