Run application on OOD head node when job starts

Get Help
1

Is it possible for OOD to launch an application (specifically a separate reverse proxy) when a specific job starts, then cancel it once the job ends? I’d like to forward specific ports through OOD head node if possible.

2

OOD user applications are, essentially, bash scripts that are are run within a scheduler job. Thus, if what you want to do can be manually scripted then it is very likely that you can also accomplish it via OOD.

That doesn’t mean that it is scalable/good idea/best practice, of course.

If it were me:
If you use Slurm, you could make a hidden partition with your OOD head node in it. You can restrict the available resources, POSIX groups, accounts, etc. When the specific job gets submitted, it will in turn submit the second job that has the reverse proxy in it. It will save the SLURM_JOB_ID for the second job.

You can create a cleanup script (template/clean.sh) that will then cancel the secondary job when the main application is finished. Read about this here and here.

3

That’s what I was afraid of.

I’m essentially looking for a way for OOD to be able to launch an application locally to proxy a web ui, since OOD’s reverse proxy capabilities always result in a blank screen. Then, the connect card would forward to the local proxy instead of the OOD url.

4

Actually, could the connection.yml file be modified to launch the proxy on the OOD host?

5

I would definitely ensure that the OOD reverse proxy is properly configured and working, and that your application can indeed be served from a specific root path before trying something totally custom.

Being able to specify the root path is what allows the reverse proxy conversion from

web.<institution>.edu:<port number>/login

to

web.<institution>.edu/node/<host>/<port number>/login

6

It is. VSCode server, Jupyter Lab, MATLAB, and RStudio all work fine. OOD isn’t passing the right apache rules, so unless those can be changed at job submit time, we need to go through a secondary proxy.

7

I see.

Maybe – with some careful scripting – you could use bin_overrides to create a wrapper script for the Slurm commands that will submit the job and then run your custom proxy. See:

That could, in theory, get you the right behavior when cancelling a job. You’d still need to figure something out to kill your custom proxy if the application exits or the job times out…

8

Is it possible to use bin_overrides at the job level? Every other application is working fine at the moment and I wouldn’t want to disrupt that if possible

9

I don’t think so, but maybe with some careful scripting you could pass some option or read in a variable so that the custom proxy would only go into effect with certain apps but not by default?

10

What app is breaking through the reverse proxy? Please don’t say Cryosparc… (although there is a solution for that app) See the nginx rewrite suggested here: mod_subsitute to correct assets/links on proxied applications · Issue #2311 · OSC/ondemand · GitHub

1 Like
11

I thought on this a little more.

It doesn’t sound like you are against SSHing to the OOD host and running your custom proxy. If that is true, then you can inject some extra scripting on a per-job basis at submission time. Look here, particularly at these:

  header: ""
  footer: ""
  script_wrapper: "%s"
  before_script: "..."
  run_script: "..."
  clean_script: "..."

Put different shell commands in each of those and see how they affect the resulting job_script_content.sh file for the job. You can chose the best option to put your command to the OOD node to launch the custom proxy and record the PID. Then you would probably want to use clean_script or footer to kill the custom proxy when the application is closed.

This doesn’t help you if they cancel or delete the job which is what you were asking in the other thread. It seems like that could be taken care of by your Slurm prolog (or a bin_override for scancel) with some very careful programming.

12

Yeah, ssh-ing back to the OOD host is what I decided on yesterday (hence the clean.sh question). I’ll probably put together a slurm epilog and enforce a standard name so it will clean out the proxy PID on job deletion.