Out jupyter instance stopped connecting. It now keeps timing out, and cannot find the port.
This is my output.log:
Script starting...
Waiting for Jupyter Notebook server to open port 10570...
TIMING - Starting wait at: Thu Feb 29 11:47:55 CST 2024
TIMING - Starting main script at: Thu Feb 29 11:47:55 CST 2024
jupyter version '5.7.1' loaded
Please do not load this module. It is designed for Open OnDemand.
Currently Loaded Modulefiles:
1) jupyter/py-3.8
TIMING - Starting jupyter at: Thu Feb 29 11:47:56 CST 2024
+ jupyter notebook --config=/home/wabuala/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/51c67153-5f46-49d9-902c-3d061bc32af1/config.py
Timed out waiting for Jupyter Notebook server to open port 10570!
TIMING - Wait ended at: Thu Feb 29 11:48:56 CST 2024
Cleaning up...
/home/wabuala/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/51c67153-5f46-49d9-902c-3d061bc32af1/script.sh: line 28: 1569923 Terminated jupyter notebook --config="${CONFIG_FILE}"
Nothing has changed. It was working perfectly 2 hours ago. What could be going on here?
Issue is only affecting Jupyter. I can get RStudio, Shell, and Desktop to spawn.
It’s not clear to me what could be happening given Jupyter doesn’t even log anything. It should be printing something to stdout right after you issue the command, but I see nothing there.
I would get a desktop and try to replicate the issue interactively (or ssh to the compute node while the Jupyter job is active). You can use the command jupyter notebook --config=/home/wabuala/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/51c67153-5f46-49d9-902c-3d061bc32af1/config.py directly or try to run the shell scripts in that same directory as config.py.
You probably need to use localhost instead of the node name. I see some errors in the output, but I’d ask how long it took for it to start printing that stuff out.
I suspect something has changed. I’d advise you to think/ask your colleagues if anything has. Also note that Jupyter because it’s Python, may recognize the python packages you have installed in your own HOME under ~/.local/lib/<python version>.
I prematurely posted this. There was an issue on the storage side, slowing down the execution of the script. I increased the timeout and it’s working. Apologies for that, and thanks for your support!
