I’ve been at this for about a week now trying to get jupyter lab to launch in open ondemand. I’ve setup a slurm cluster and i have gotten to the point where i can launch a job from the OOD portal and it starts on the compute node and i can access it directly via a web ui (ie http://hpc-02.grit.ucsb.edu:12345/lab) but if i click the connect button in the OOD portal i get a 404 page at https://hpc.grit.ucsb.edu/node/hpc-02.grit.ucsb.edu/25933/lab. I am able to access the shell via the web ui and i’m at a lost as to how to further troubleshoot this. Any help would be appreciated. below is a dump of the config files:
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ ls
form.yml manifest.yml template view.html.erb
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat form.yml
---
cluster: "grit_hpc"
form:
- bc_account
- bc_queue
- bc_num_hours
- bc_num_slots
attributes:
bc_queue: "grit_nodes"
bc_num_hours: 1
bc_num_slots: 1
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat manifest.yml
---
name: Jupyter Notebook
category: Interactive Apps
subcategory: Servers
role: batch_connect
description: "Launch a Jupyter Notebook server via Slurm."
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat view.html.erb
<form action="/node/<%= host %>/<%= port %>/lab" method="post" target="_blank">
<input type="hidden" name="password" value="<%= password %>">
<button class="btn btn-primary" type="submit">
<i class="fa fa-eye"></i> Connect to Jupyter
</button>
</form>
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat template/
after.sh.erb before.sh.erb script.sh.erb submit.yml.erb
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat template/after.sh.erb
#!/bin/bash
echo "Jupyter launched successfully at http://<%= `hostname --fqdn`.strip %>:<%= ENV['PORT'] %>/"
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat template/before.sh.erb
# Get full hostname of compute node
host=$(hostname -f 2>/dev/null)
host=${host:-$(hostname).grit.ucsb.edu}
export host
# Use port from env or pick a random one in range
port="<%= ENV['PORT'] || rand(10000..65000).to_s %>"
export port
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat template/script.sh.erb
<%
port = ENV['PORT']
%>
#!/bin/bash
#SBATCH --job-name=jupyter-notebook
#SBATCH --output=output.log
#SBATCH --partition=<%= context.bc_queue %>
#SBATCH --account=<%= context.bc_account %>
#SBATCH --time=<%= context.bc_num_hours %>:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= context.bc_num_slots %>
set -euxo pipefail
export PORT=<%= port %>
export IP=$(hostname --fqdn)
# Activate your Python venv
source /opt/jupyterhub/bin/activate
echo "http://${IP}:${PORT}/"
# Launch Jupyter Notebook
jupyter-lab --no-browser \
--ip=0.0.0.0 \
--port=${PORT} \
--NotebookApp.token='' \
--NotebookApp.password=''
gritadm@hpc-grit:/var/www/ood/apps/sys/jupyter$ cat template/submit.yml.erb
<%
port = rand(10000..65000)
ENV['PORT'] = port.to_s
%>
submit: script.sh.erb
batch_connect:
template: "v2"
port: "<%= port %>"
redirect: true
Off the top of my head - I’d guess that you need to complete this step below.
I would also check this section out to verify that it works.
That’s my guess if the 404 page you’re referencing is from Apache.
If, on the other hand, 404 page is from Jupyter, then you likely need to configure Jupyter to recognize that it’s behind a proxy. Here’s our configuration for the same:
thanks for the quick response, i reviewed your suggestion, and here is the relevant contents of ood_portal.yml:
# Regular expression used for whitelisting allowed hostnames of nodes
# Example:
# host_regex: '[\w.-]+\.example\.com'
# Default: '[^/]+' (allow reverse proxying to all hosts, this allows external
# hosts as well)
#host_regex: '[^/]+'
host_regex: '[^/]+'
# Sub-uri used to reverse proxy to backend web server running on node that
# knows the full URI path
# Example:
# node_uri: '/node'
# Default: null (disable this feature)
node_uri: '/node'
# Sub-uri used to reverse proxy to backend web server running on node that
# ONLY uses *relative* URI paths
# Example:
# rnode_uri: '/rnode'
# Default: null (disable this feature)
rnode_uri: '/rnode'
following the suggestion to try and setup nc on a compute node and access it results in a 404 as well. (https://hpc.grit.ucsb.edu/node/hpc-02.grit.ucsb.edu/5432/) though from the command line i can nc from the ood host to the compute node and verify it can connect. The error for both is an apache 404 page.
found another post here with a similar sounding issue and tried copying the contents of /etc/apache2/sites-available/ood-portal.conf to /etc/apache2/sites-available/ood-portal-le-ssl.conf and now i get a jupyterhub 404 page. at least a new error is some progress.
Jupyter Server
404 : Not Found
You are requesting a page that does not exist!
further more if i try to conenct to an open nc port i see the expected data on the remote host so i’m dangerously close.
UPDATE:
i can now get it to load the correct page but it initially shows a 405 error, if i reload the page then i get my jupyter lab and all is well. I’m again stumped on this one.
UPDATE2:
got it sorted i had a post in my view.html.erb method instead of a get
Apache loads configs in alphabetical order so my guess is that the -le-ssl.conf one is getting loaded before the main ood one so whatever changes are in the le-ssl.conf are working. I would diff the files. Maybe give us the diff output?