Using a MS Win desktop I’ve started 20+sessions today. I cannot reproduce the error.
I have one Mac user who consistently cannot launch Jupyter with it consistently returning:
Not Found
The requested URL was not found on this server.
The URL looks like the following (minus https:the servername)
node/compute-6-11.cluster/38478/login?next=%2Fnode%2Fcompute-6-11.cluster%2F38478%2F
This seems to occur only with MacOS Chrome intermittently, however one person today is locked out by this error. I have no other users reporting. It happens to this individual regularly, so they’re understandably frustrated.
Thank you for considering. I have spun through many like errors here in discourse without identifying a solution. Windows clients all seem to work flawlessly.
Andy
Hi and welcome!
Next time it happens I would try to hit that same URL the user provides. The user has the username/password combination, but you don’t so you’ll be prompted with a password you don’t know. I suggest this because I’m guessing it’s actually a server side issue. That is, if you’re prompted for a password by Jupyter then it may be local to that user, but if you receive the same 404 as them then it means the issue is on the server and their browser/OS combination is a red-herring.
That’s how this reads to me anyhow - the server (apache) is responding 404 because it can’t connect to that host:ip (compute-6-11.cluster:38478 in the example you’ve given) and that’s a server side issue - not a client side issue.
As a complete guess I would assume they are starting juptyer and it’s crashing for one reason or another. There may be something about that in the output.log of that particular job. If jupyter is running and apache times out I believe that would be a different error code than 404 (some 500 level code I’d guess).
At any rate - I’d suggest pulling the output.log for this users’ Jupyter jobs and see if it’s crashing or indeed just ssh’ing to the machine to see if it’s still running or just a simple squeue or similar to see if the job is still running (if jupyter crashes and stops, then the job should complete I’m pretty sure).
Connecting with “wget compute-6-11.cluster:38478” from the ondemand proxy host I get a “ERROR 404: Not Found”. A test to my own working jupyter session responds equally with wget. Both seem to connect but 404.
Interestingly we can connect to the user’s onemandhost/pun/sys/shell/ssh/compute-6-11 and it starts a shell for him so the proxy seems good.
I’ve been troubleshooting on the node and can see that port is open (netstat lnp | grep 38478 for example) where I find python running and the port open 0.0.0.0:38478 ...746601/python3
Then an lsof -p 746601 where I can see the process is in the “Working Directory” requested and jupyter appears to be started. The user’s processes frompsare (mostly) identical to my own working example, both having launched the dashboard/batch_connect script.sh and jupyter-lab config.py
The job’s output.log (a fresh session) ends the same as my own with:
Discovered Jupyter server listening on port 16629!
Generating connection YAML file…
Nothing stands out on the server’s per-user access.log or error.log. This just seems the strangest thing.
Thank you for considering!
OK - I think I see what’s going on now. The 404 isn’t from apache it’s from Jupyter.
Can you share your application or have a link to it on the web?
From your first message
if you look at the next query parameter it is /node/compute-6-11.cluster/38478/ - I.e., hitting the / root directory of Jupyter. It should in fact look like /node/compute-6-11.cluster/38478/tree or /node/compute-6-11.cluster/38478/lab, ending in /tree or /lab not just /.
If your jupyter application is like ours, we set this in the in the script
to pass back to the view
Somehow for this user it isn’t being set properly. Do you have the same option that we do that allows users to get jupyter notebook (that results in /tree) or jupyter lab (that results in /lab)?
Users can choose between Lab and notebook “tree”, both behave the same for this broken user. As with your example:
Beginning at line 56 /var/www/ood/apps/sys/jupyter/template/before.sh.erb:
JUPYTER_API="<%= context.mode == "1" ? 'lab' : 'tree' %>"
export JUPYTERLAB_WORKSPACES_DIR=$PWD
export jupyter_api="$JUPYTER_API"
In the user’s output directory before.sh line 46:
JUPYTER_API="lab"
export JUPYTERLAB_WORKSPACES_DIR=$PWD
export jupyter_api="$JUPYTER_API"
Could the
$PWD be ${PWD} or “${PWD}”? It seems to make no difference in quick tests.
We’ve tried both lab and notebook aka tree with the same result. Is there any notable response difference on a Mac? I am tempted to let blame fall on the user’s desktop but I cannot prove anything.
Due to security layers and MFA it would be difficult to share the app directly.
Thank you , Andy
I don’t believe it’s an issue with PWD.
Can you share your view.html.erb and make sure that conn_params is set using jupyter_api.
conn_params:
- jupyter_api
I omitted conn_params from our submit.yml.erb but just added it back as below:
batch_connect:
template: "basic"
conn_params:
- jupyter_api
I must have been struggling with getting something going back in December, I suspect it was removed during a process of elimination. I’ve asked the impacted user to retry but he reports no change. I found no change from having it or not while using my notebook.
Our view.html.erb:
<%-
base_url = "/node/#{host}/#{port}"
login_url = "#{base_url}/login"
next_url = "#{base_url}/#{jupyter_api}"
full_url="#{login_url}?next=#{CGI.escape(next_url)}"
form_id = "juypyter_form#{login_url.gsub('/', '_')}"
%>
<!-- Zoom on iPad doesn't like opening in a new tab. It won't foward the form parameters. -->
<script type="text/javascript">
function changeTarget() {
const agent = navigator.userAgent;
if (/Mozilla\/5\.0 \(Macintosh; Intel Mac OS X \d+_\d+_\d+\) AppleWebKit\/\d+\.\d+\.\d+ \(KHTML, like Gecko\)/.test(agent)) {
document.getElementById("<%= form_id %>").target = "_self";
}
}
</script>
<form id="<%= form_id %>" action="<%= full_url %>" method="post" target="_blank" onsubmit="changeTarget()" >
<input type="hidden" name="password" value="<%= password %>">
<button class="btn btn-primary" type="submit">
<i class="fa fa-cogs"></i> Connect to Jupyter
</button>
</form>
Odd - I assume when the user just adds /lab to the URL it does work for them? But the Connect to Jupyter link the card never has it?
That works for me, I can put either /lab or /tree in and it works. The affected user says:
This is the URL I get that fails.
/node/compute-6-14.cluster/39216/login?next=%2Fnode%2Fcompute-6-14.cluster%2F39216%2Flab
So I modded the URL
/node/compute-6-14.cluster/39216/lab
It gave the same Not Found msg.
I checked on the node and I can see the python http server running on port 39216 an lsof shows that process is in the expected/requested file location. The log file ends with
Discovered Jupyter server listening on port 39216!
Generating connection YAML file…
If I try to connect there the log shows
[W 2026-07-07 15:49:15.597 ServerApp] 404 GET /node/compute-6-14/39216/tree (@xx.xx.xx.28) 2.35ms referer=None
[W 2026-07-07 15:49:37.751 ServerApp] 404 GET /node/compute-6-14/39216/lab (@xx.xx.xx.28) 2.26ms referer=None
It displays a jupyter lab link in the upper right and
404 : Not Found
You are requesting a page that does not exist!
It seems sometimes it uses the fqdn and for me it does not
/node/compute-6-14/39216/lab
versus
/node/compute-6-14.cluster/39216/lab
On the ondemand node it resolves either way via DNS and I recently added them all to /etc/ho
sts
Given that it’s that single user, I wonder if there’s something in their ~/.bashrc that’s throwing this off?
I think I found the answer
host_regex: ‘(compute|largemem|gpu)-\d±\d+\’
doesn’t support the FQND compute-13-16.cluster but the server seems to be randomly delivering URLs with and without .cluster
That regex only works for compute-13-16 not compute-13-16.cluster
They warn not to wildcard it but what else might work? Can I get the server to produce one URL or the other?
I have been using
host_regex: ‘(compute|largemem|gpu)-\d±\d+’
For fishing reasons it would be good if the server provided the fqdn in the url each time, but this is not the case. Then .cluster could be used in the regex.
Yes you can configure set_host parameter to use maybe "$(hostname -A | head -n 1)" that way you force the FQDN every time and add .cluster to the host_regex.
You can set this option globally for the entire cluster so you don’t have to set it in every application.
Some compute nodes returned the fully qualified domain name from ‘hostname’ while some did not. Changing the command in the ood/config/clusters.d/cluster.yml batch_connect/basic stanza to set_host: "host=$(hostname -f)" makes what is returned consistent. In turn, the whitelisting regex needed to include the domain “.cluster” ‘(compute|largemem|gpu)\-\d+\-\d+\.cluster’ so that it will match consistently. Without a match you cannot establish a connection - the root of the intermittent error.
Thank you for your efforts on this Jeff.
Glad to hear it worked out!