I just install a Jupyter interactive app and success to run the app and open the Jupyter Notebook in browser.
The problem is, whenever I open a file in Jupyter Notebook, it will open a new tab and start to load, but it aways end up with 502 proxy error. I have no idea why it happens as there is no error in the output.log. It looks to me like the problem of apache server, but I don’t know how to find out the root cause and fix it.
Error Message:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server
Output Log
[I 22:30:14.310 NotebookApp] Serving notebooks from local directory: /data/home/xxxx
[I 22:30:14.311 NotebookApp] Jupyter Notebook 6.4.5 is running at:
[I 22:30:14.311 NotebookApp] http://c51-s001:54978/node/c51-s001/54978/
[I 22:30:14.311 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Discovered Jupyter Notebook server listening on port 54978!
TIMING - Wait ended at: Tue Apr 25 22:30:14 CST 2023
Generating connection YAML file...
[I 22:32:16.428 NotebookApp] 302 POST /node/c51-s001/54978/login (xx) 3.320000ms
[I 22:32:16.525 NotebookApp] 302 GET /node/c51-s001/54978/ (xx) 0.920000ms
What make me fell interesting is attempt to connect to 10.1.51.35:80, the ip is the host c51-s001, I don’t know why it try to connect to 80 port instead of 52418, which is the listening port of Jupyter notebook.
In my browser I found the request is sent to ws://localhost:8086/node/c51-s001/52418/api/kernels/415697fa-6116-4332-8f4d-266e5fc750ea/channels which seems correct to me. So I think it may be the issue of reverse proxy.
The timeout would be something that needs adjusted on the SLURM side, the proxy has no setting for that.
It is also strange to me that the IP is being used for the connection, I would have expected the compute cluster’s hostname there. That also makes me wonder about some DNS configuration.
Could you post the contents of the connection.yml when you submit?
One thing you could try is to check the status of the Slurm job running on that node to see if it’s still running or if it has timed out. You can do this by running the command squeue -u <username> -n <jobname> (replacing <username> and <jobname> with the appropriate values).
I don’t know what is happening, but sometimes it just works after restart Jupyter kernel. I don’t know where localhost:80 is comming from as there is no such settings.
And now I have another 503 issue when I try to change the file name in Jupyter notebook. Just my gussing, it seems like Apache didn’t allow for PATCH method. Is there a way to fix this via ood configuration file?
One of my collegue found a fun fact that if there is a path token in URL start with number, for example, in ws://localhost:8086/node/c51-s001/52418/api/kernels/415697fa-6116-4332-8f4d-266e5fc750ea/channels, as you can see 415697fa-6116-4332-8f4d-266e5fc750ea is start with number, then the proxy will have problem, otherwise it will be OK. I didn’t dive into the code to confirm this issue. But I also observe the same problem in my case. Maybe there are some regular expressions didn’t take this into account.