Ariel’s comment explains in detail the steps to debug the problem.
This problem occurs when connection file generated by one of the running interactive sessions is corrupt, due to non-printable ASCII characters, non valid UTF8. In this particular instance the locale on the compute node was set to ‘en_US’ instead of ‘en_US.UTF-8’. Since our code was using /dev/urandom
to generate the password used in the connection file, some non UTF-8 characters were getting into the file. We changed the code used to generate passwords, addressed in https://github.com/OSC/ood_core/issues/91, so hopefully in OnDemand 1.4 this problem will be solved.