Testing things outside of the user’s context used by OnDemand will give different results than as root if SELinux is enabled. On a compute node that’s having issues you’ll have to debug if SELinux is the culprit by doing something like this:
cat /var/log/audit/audit.log | audit2allow
It’s possible the processes are getting blocked by SELinux and a custom policy is needed or some boolean needs to be enabled.
If audit2allow produces too much output then you can truncate the audit log before running a test > /var/log/audit/audit.log.
I noticed on some nodes application fails to generate connection.yml file. on some it does and everything works correctly. What part of the application generate this file? and is there any dependency needed to generate this file?
The dashboard will generate all that data after everything is setup appropriately. It’s a book-keeping sort of thing.
You’re seeing those files because the batch connect application was setup correctly. Everything working out is the only real dependency for these files being generated.
Below is more detail about what generates the connection.yml, but I’m not sure that it will be helpful.
This is part of the Ruby class that is used to generate the job script that is submitted by sending the script contents into stdin of the submit command (whether that be qsub or bsub or sbatch etc.):
That is Ruby string interpolation, where conn_params is an array of environment variables. So host, port, etc. env vars are set to certain values and whenever create_yml is called in the job script the connection.yml file gets generated.
This is the end of the script:
All of the #{} tags are Ruby interpolation. So the run_script is the path to the main script that is executed, then the after_script is executed, then the connection file is created.
For vnc templates create_yml is also called after each successful connection, with a new password generated: