Jupyter App issues

Get Help
1

We’re running OOD 3.1.10 and using a Jupyter app (deployed using the example in the documentation I think). One user is able to consistently break Jupyter on one node, but others are fine. What I’m seeing by comparing the $HOME/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/<job_code> directories for each node’s jobs, is that the config.py file for the working node has a field for c.NotebookApp.password, which starts with “u’sha1:” and then has a short string, a second colon, and a longer string, followed by single quote. On the non-working node, the longer string are not present; the single quote follows the short string and second colon. When the user tries to open the Jupyter app after being allocated, on the non-working node it prompts him for a password, and the password from connection.yml does not work (I’m not sure how the hashing function works, so this may be expected). On the working node, he’s not prompted for a password.

To make things more interesting, when I test on the non-working node, it works fine for my (privileged) user.

What further troubleshooting can I do?

2

This was really annoying to troubleshoot. A labmate of our user with the issue was able to reproduce the error, and it turned out that they were each setting LD_LIBRARY_PATH in their .bash_profile scripts to a kerberos module they compiled for their team. That compiled version of libcrypto.so did not include support for EVP_md2 that the openssl call to hash the jupyter password needs.

This line was at the top of output.log, but I had missed it previously:

openssl: symbol lookup error: openssl: undefined symbol: EVP_md2, version OPENSSL_1_1_0

I added an override for LD_LIBRARY_PATH to point to the system libcrypto.so (/usr/lib64) and that fixed this.

But now I have a new issue that cropped up a few days later. The config.log file looks good when a job is created, but the nodes I have tested on are not spawning the actual network port so the job times out.

Script starting...
Waiting for Jupyter Notebook server to open port 57527...
TIMING - Starting wait at: Thu Mar 26 08:15:06 EDT 2026
TIMING - Starting main script at: Thu Mar 26 08:15:06 EDT 2026

And eventually:

Timed out waiting for Jupyter Notebook server to open port 57527!

Has anybody encountered this?