Connect to external JupyterLab server

Get Help
1

Hi all,

I have a JupyterLab apps working well in our OOD v1.8 platform.
Now I want to use this OOD platform to config and access a remote JupyterLab server running outside of the OOD cluster. So I cloned the JupyterLab apps and replace the “jupyter lab” command in “script.sh” with a batch of commands. These commands send config parameters to the remote host to start the JupyterLab server and then conduct the ssh port forwarding to connect to the remote JupyterLab server from the OOD node. However, it returns the “Failed to connect” to the OOD node/port error message after clicking “Connect to Jupyter” button from the OOD portal.

Here is the view.html.erb file:

<form action="/node/<%= host %>/<%= port %>/login" method="post" target="_blank">
  <input type="hidden" name="password" value="<%= password %>">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-eye"></i> Connect to Jupyter
  </button>
</form>

Looks like OOD reverse proxy can not connect to the forwarded port of a OOD node. Is there a way to access the external JupyterLab server via the OOD portal? Thanks.

2

Hi Metyr. I have a couple requests please.
1). Can you please post any logs that may have been generated.
2). Can you please post the updated script.sh? If there is information that you do not want made public, please remove that information from the post. If we need that info, we’ll reach out to you.

3

Hi gbyrket,

Many thanks for your reply.

The “script.sh” is pretty simple as below

#!/usr/bin/env bash
echo "running jupyter"
ssh-agent MY_HOME_DIR/ondemand/dev/remote_jupyter/bin/remote_jupyter ./pgo.yml 
echo "Jupyter job is done!"

The pgo.yml contains necessary PBS job parameters at the remote cluster which are specified by the user at the OOD session config page. The command “remote_jupyter” starts the remote JupyterLab server with the configure file “config.py” and then conduct the ssh port forwarding at the OOD node.

The config.py file:

c.NotebookApp.ip = '*'
c.NotebookApp.port = 29295
c.NotebookApp.port_retries = 0
c.NotebookApp.password = u'sha1:7SU3hQY0cqH1jMYF:2cac5f8fd37ba1482692894e3117451d83657c85'
c.NotebookApp.open_browser = False
c.NotebookApp.allow_origin = '*'
c.NotebookApp.notebook_dir = '/home/xyz'
c.NotebookApp.disable_check_xsrf = True

Both pgo.yml and config.py are created in before.sh.erb.

The log file “output.log”:

Script starting...
Waiting for Jupyter Notebook server to open port 29295...
TIMING - Starting wait at: Sat Nov 20 12:47:08 AEDT 2021
running jupyter
Discovered Jupyter Notebook server listening on port 29295!
TIMING - Wait ended at: Sat Nov 20 12:47:29 AEDT 2021
Generating connection YAML file...

It is noticed that the function “port_used_nc” can not discover the forwarded port 29295 so I have to use “port_used_lsof” to discover it and then generate the connection YAML file.

I can manually access the remote JupyterLab server with the url “localhost:29295” at the local OOD node. However, I get the “Failed to connect” error after clicking “Connect to Jupyter” button at the OOD session page.

Please let me know if you need more information. Thanks.

4

Hi, I’m trying to figure out the topology here and where you’re able to access it.

You have an OOD host that you’ve submitted a job to. ssh-agent is running on your compute node of this job.

Am I reading this right - You can access localhost:29295 on the OOD web server instance. Not the compute node that your job is running on?

5

Hi Jeff,
Thanks for your reply. Here is the workflow:
I firstly start a job from OOD web portal and get it running at a OOD compute node, say ood-vn1. Then the job use “ssh-agent” to run a script at ood-vn1 to login the other cluster to start a JupyterLab session there. At the end, the script runs the ssh port forwarding command to forward remote JuputerLab port to the local port 29295 of ood-vn1.
I can confirm the port forwarding works by loging in ood-vn1 directly and opening the url “localhost:29295” in the web browser to access the remote JupyterLab session. However, it doesn’t work when OOD web server try to open it with the url like “https://ood*****/node/ood-vn1/29295”.
So I was thinking whether I need some settings on the reverse proxy to make the OOD server connectable to the forwarded port at its compute node.

6

Got it, thanks. I don’t know if that port 29295 is external. I don’t use ssh forwarding often so I don’t know exactly how it works, but my guess here is that 29295 is only accessabile through the localhost (local to ood-vn1). Meaning when the ood server (let’s call it ood-web1) tries to contact ood-vn1:29295 it can’t.

When you start this job, can you connect to ood-vn1:29295 from any external host? (your laptop, a login node, another compute node, etc).

9

Hi Jeff, Thanks for your help. I finally explore the issue and make it working. This ticket can be solved.

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.