Having trouble configuring Jupyter on OOD: "Not Found -- The requested URL was not found on this server."

Get Help
1

Hi all,

I am trying to set up Jupyter on OOD. I’m running OOD on a LSF10.1, RHEL8 cluster.

The Jupyter interactive session launches, and gets to the Running state. However, whenever I click :eye: connect to Jupyter, I get the following error:

Not Found

The requested URL was not found on this server.

I went ahead and enabled reverse proxy in the /etc/ood/config/ood_portal.yml file like so:

host_regex: '[\w.-]+\.my_center\.edu'
node_uri: '/node'
rnode_uri: '/rnode'

I also tried the default value for host_regex, but that did not work either.

Any help would be appreciated. Many thanks in advance! :smile:

2

Can you share the URL you’re trying to access? You can obfuscate what you need, but your intuition about those configs is correct.

What I would question is - in the URL do you have the FQDN or a simpler hostname.

3

Hi Jeff,

Thanks for your response. The URL that Connect to Jupyter button sends me to is this: https://svltondemand01.institution.org/node/nodegpu228/39003/login

In the URL, I don’t have the FQDN, just the domain name. A quick glance at output.log shows the following:

Script starting...
Waiting for Jupyter Notebook server to open port 39003...
TIMING - Starting wait at: Mon Nov  6 15:32:44 CST 2023
TIMING - Starting main script at: Mon Nov  6 15:32:44 CST 2023
TIMING - Starting jupyter at: Mon Nov  6 15:32:44 CST 2023
+ jupyter notebook --config=/home/wabuala/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/bf30c35c-adbb-46f2-a8e5-598ef668f046/config.py
[W 15:32:48.566 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[W 15:32:48.567 NotebookApp] Terminals not available (error was No module named 'winpty')
[W 2023-11-06 15:32:50.098 LabApp] 'config_file' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2023-11-06 15:32:50.098 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'port_retries' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'password' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'allow_origin' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'disable_check_xsrf' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-11-06 15:32:50.098 LabApp] 'disable_check_xsrf' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-11-06 15:32:50.101 LabApp] JupyterLab extension loaded from /home/wabuala/.local/lib/python3.6/site-packages/jupyterlab
[I 2023-11-06 15:32:50.101 LabApp] JupyterLab application directory is /path/to/userHome/wabuala/.local/share/jupyter/lab
[I 15:32:50.122 NotebookApp] Serving notebooks from local directory: /home/wabuala
[I 15:32:50.122 NotebookApp] Jupyter Notebook 6.4.10 is running at:
[I 15:32:50.122 NotebookApp] http://nodegpu228:39003/node/nodegpu228/39003/
[I 15:32:50.122 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Discovered Jupyter Notebook server listening on port 39003!
TIMING - Wait ended at: Mon Nov  6 15:32:50 CST 2023
Generating connection YAML file...

This line [W 15:32:48.567 NotebookApp] Terminals not available (error was No module named 'winpty') captures my attention, but despite winpty not being installed, seems like Jupyter is spinning up.

4

Yea that may bit it exactly - you’re expecting to match against the FQDN and that nodegpu228 doesn’t match that.

5

I tried the following, and all give the same 404 error:

host_regex: '[^/]+'
host_regex: '[\w.-]'
host_regex: '[\w.-]+'
host_regex: '[\w.-]+\.institution\.org'

I’m confused as to why the default is not working, should map to all alphanumeric chars until it sees “/”. Is there a way for me to check what the host value is resolving to after applying host_regex?

6

Sorry - you’re reconfiguring the yaml file - did you bounce httpd to be sure it takes affect? You can grep through the .conf files to see if it’s actually showing up for apache.

7

This is very permissive and seems like it should work (for just about anything lol) which leads me to think the configs aren’t being updated in apache.

8

Aha! I had forgotten to bounce httpd, thanks for the heads up.

I got a new error, using host_regex: '[\w.-]+'. The new error reads:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: DNS lookup failure for: nodegpu237

A quick google search shows this: Solved: Proxy Error Apache Proxy SSL. Seems like it’s in the right direction, but I’m confused where to add the “/” that is mentioned in the answer.

Have you seen this error before?

Thanks for your prompt help and support, Jeff! :smile:

9

I actually just realized that the hostname is not is /etc/hosts! :scream: lol

I will add it and let you know if that resolves the issue. I have a keen feeling this might be the issue.

10

That was the solution. Once I could directly ssh from the OOD host VM to the compute node, Jupyter spun up! Thanks again, @jeff.ohrstrom

1 Like