How to tunell interactive desktop over ssh (jump host)

Get Help
1

In our case we can’t submit jobs directly from the host where ood runs. Instead for a few clusters we have a wrapper scripts that call slurm commands over ssh.
I did a POC running interactive deskotp app and forwarding ports over ssh using -L option and changing connection.yml appropriatelly and it worked fine. Looking at the batch_connect I can’t find a “hook” executed on the host when sbatch is called after sbatch or best afeter VNC started.

Did anyone accomplish that? I’m looking for hints of how to do that with as low number of changes as possible.

Thanks in advance!

2

I remember that there used to be tweak/configuration for OOD desktop app to show the exact ssh command to use to get the forwarding working.

EDIT: mea maxima culpa. I thought that you want to tunnel the ssh from your local machine, not between ood host and vnc host. Forget what I wrote above, pls.

3

Thanks for the reply!

I can figure out the command, but I don’t know how/where to inject it. I was able to run the ssh tunnel manually and “hack” connections.yml so I was able to open the deskotp in the web, but I don’t know where to configure the “after” script that would do the same on every desktop creation.

4

I have seen folks just write wrappers around sbatch. I.e. sbatch ends up being a script like:

#!/bin/bash
ssh login-node.mysite.edu $@
5

Thanks! Yes, the wrapper works fine, but the interactive deskotp app requires connectivity from the host running OOD to the computing node, which isn’t possible in my case. I have to setup a tunnel (POC with ssh tunnel done by ssh -L works fine). I just need something that will be executed once the VNC and websockify are running to setup a tunnel.
I can’t find anything like “after_script” neither in batch_connect nor in the vnc app configuration.

6

Ah right, so your OOD host is not on the same network as the compute nodes?

1 Like
7

Just want to jump in with a few quick comments:

If you look at the OOD installation docs (Installation — Open OnDemand 3.1.0 documentation) we explicitly say

The OnDemand host machine needs to be setup similarly to a login node. This means that it will need:

    RedHat/RockyLinux/AlmaLinux 8+ or Ubuntu 20.04-24.04 or Debian 12 or Amazon Linux 2023

    the resource manager (e.g., Torque, Slurm, or LSF) client binaries and libraries used by the batch servers installed

    configuration on both OnDemand node and batch servers to be able to submit, status, and delete jobs from command line

    signed SSL certificate with corresponding intermediate certificate for your advertised OnDemand host name (e.g., ondemand.my_center.edu)

While it’s certain possible to setup in other configs, having it act like a login node is the most standard way to setup and the easiest to support long term.

The other thing you might want to do is look at our Linux Host Adapter, which allows you to execute SSH commands to remote hosts: LinuxHost — Open OnDemand 3.1.0 documentation

8

cluster.d file support the attribute submit_host which is A different, optional host to ssh to and then issue commands. You can also use bin_overrides if you want something fancier.

https://osc.github.io/ood-documentation/latest/installation/resource-manager/slurm.html?highlight=submit_host

This is going to be tough without this connectivity. You can’t get a firewall rule to allow traffic from this host?

Seems like you’ll have to rewrite the bc_desktop application. You can make changes to it directly - but they’ll be overwritten on updates. So best thing to do is to cp

cp -R /var/www/ood/apps/sys/bc_desktop /var/www/ood/apps/sys/bc_desktop2
# disable the original bc_desktop. you'll need to disable it again on subsequent updates.
chmod 700 /var/www/ood/apps/sys/bc_desktop

Now you’ll be able to freely edit /var/www/ood/apps/sys/bc_desktop2/template/script.sh.erb to do what you like.

9

Thanks @jeff.ohrstrom it would be great to get a kind of hook that can run afterwards, but knowing that there is no other option today is also a solution.

10

Finally I think I’ll do it the opposite way - ssh’ing back from the computing node over the jump host to ondemnad, this way I’ll be able to just add a command to the wrapper.

I’ll report back if it works fine.

11

Thinking about this a little more, I think you can also override websockify_cmd to be some sort of wrapper that does a bunch of stuff and boots websockify.

https://osc.github.io/ood-documentation/latest/reference/files/submit-yml/vnc-bc-options.html

12

OK… I was able to get a little bit further using @jeff.ohrstrom suggestion to wrap websockify_cmd, by the script like:

ssh      -R "$1:localhost:$1" -J CLUSTER_LOGIN_HOST USER@OOD_ADDRESS -f -N -v
/path/to/websockify $@

The only change I need to make manually now is substituting host in connection.yml to appropriate address of my ondemand server. One dirty way I’m thinking of doing it is a infinite loop like:

MYHOSTNAME=$(hostname -s)
OODADDRESS="x.y.z.y"
while /bin/true
do
        sleep 5
        sed -i "s/$MYHOSTNAME/$OODADDRESS/g" connection.yml
done &

Any suggestions on how to make it “clean”?

13

I think you can use set_host another configuration here. When the ood script runs, it issues the command for set_host which is generally $(hostname). If you need to keep it static, use $(echo 'x.y.z.y').

This means that x.y.z.y will show up in your connection,yml.

https://osc.github.io/ood-documentation/latest/reference/files/submit-yml/basic-bc-options.html