Hi folks, So a user had an issue with installing R packages as whoever set up the bioconductor containers really goofed and didn’t follow through with their work. It took me a few hours to figure out everything, then I made it right, and had the user log outlast Thursday. He hasn’t been able to log back in since. He gets this every time he tried to get into Xfce Remote Desktop. Since Friday I found horrific configuration of httpd, somebody had disbled mod_mpm_event and the server settings that goes with that. It was producing a ton of errors since about the beginning of March. I’m providing /var/log/httpd/error_log which shows you the mpm_node events. Showing you the /var/log/audit.log (partial). Also provided the /var/lib/ondemand-nginx/puns//access.log and error.log.
Falling asleep at my desk. I’ll check back in the morning an dsee what else I can provide.
Appreciate you guys,. 4.1 looks great and runs pretty damn fast now that I fixed the httpd server with some threading. lol.
Kenny @ MSU
var-log-error_log.txt (13.2 KB)
puns-access.txt (301.1 KB)
puns-error.txt (706.5 KB)
This has 2 parts, this is Part 1:
Ok after reading for a bit from your manual I found the rake test:jobs command. I can verify that user has a .secret_key_base text file in /var/lib/ondemand-nginx/sys/puns/
[root@tempest-web dashboard]# su q75r136 -c ‘source /opt/ood/ondemand/enable; bin/rake --trace test:jobs:tempest RAILS_ENV=production’
** Invoke test:jobs:tempest (first_time)
** Invoke environment (first_time)
** Execute environment
Rails Error: Unable to access log file. Please ensure that /var/www/ood/apps/sys/dashboard/log/production.log exists and is writable (i.e. make it writable for user and group: chmod 0664 /var/www/ood/apps/sys/dashboard/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
ArgumentError: Missing secret_key_base for ‘production’ environment, set this string with bin/rails credentials:edit (ArgumentError)
raise ArgumentError, "Missing `secret_key_base` for '#{Rails.env}' environment, set this string with `bin/rails credentials:edit`"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:661:in validate_secret_key_base' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:483:in secret_key_base’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:161:in key_generator' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/turbo-rails-2.0.7/lib/turbo/engine.rb:92:in block (2 levels) in class:Engine’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:94:in block in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:87:in with_execution_control’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:92:in execute_hook' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:78:in block in run_load_hooks’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:77:in each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/activesupport-7.1.6/lib/active_support/lazy_load_hooks.rb:77:in run_load_hooks’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application/finisher.rb:93:in block in <module:Finisher>' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/initializable.rb:32:in instance_exec’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/initializable.rb:32:in run' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/initializable.rb:61:in block in run_initializers’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:231:in block in tsort_each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:353:in block (2 levels) in each_strongly_connected_component’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:434:in each_strongly_connected_component_from' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:352:in block in each_strongly_connected_component’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:350:in each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:350:in call’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:350:in each_strongly_connected_component' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:229:in tsort_each’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/tsort-0.2.0/lib/tsort.rb:208:in tsort_each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/initializable.rb:60:in run_initializers’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:426:in initialize!' /var/www/ood/apps/sys/dashboard/config/environment.rb:7:in <top (required)>’
/usr/share/ruby/bundled_gems.rb:69:in require' /usr/share/ruby/bundled_gems.rb:69:in block (2 levels) in replace_require’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/zeitwerk-2.6.18/lib/zeitwerk/kernel.rb:34:in require' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:402:in require_environment!’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/railties-7.1.6/lib/rails/application.rb:591:in block in run_tasks_blocks' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:281:in block in execute’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:281:in each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:281:in execute’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:219:in block in invoke_with_call_chain' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:199:in synchronize’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:199:in invoke_with_call_chain' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:243:in block in invoke_prerequisites’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:241:in each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:241:in invoke_prerequisites’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:218:in block in invoke_with_call_chain' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:199:in synchronize’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:199:in invoke_with_call_chain' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/task.rb:188:in invoke’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:188:in invoke_task' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:138:in block (2 levels) in top_level’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:138:in each' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:138:in block in top_level’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:147:in run_with_threads' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:132:in top_level’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:83:in block in run' /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:214:in standard_exception_handling’
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.1.4-1/gems/rake-13.3.1/lib/rake/application.rb:80:in run' bin/rake:10:in ’
Tasks: TOP => test:jobs:tempest => environment
I don’t think that’s related. I’d guess there’s something in their ~/.bashrc that’s throwing this off. Specifically a conda environment or similar that’s supplied a different PATH with a different dbus-launch in that PATH.
Part 2: I took the advice from this statement:
ArgumentError: Missing secret_key_base for ‘production’ environment, set this string with bin/rails credentials:edit (ArgumentError)
from dashboard I ran bin/raise credentials:edit and I got this:
Adding config/master.key to store the encryption key: 3e1199237cea5c00a2f52969e3c8bd60
Save this in a password manager your team can access.
If you lose the key, no one, including you, can access anything encrypted with it.
create config/master.key
IMPORTANT: Don’t commit config/master.key. Add this to your ignore file:
/config/master.key
on_green
No $VISUAL or $EDITOR to open file in. Assign one like this:
VISUAL=“mate --wait” bin/rails credentials:edit
For editors that fork and exit immediately, it’s important to pass a wait flag;
otherwise, the file will be saved immediately with no chance to edit.
the comand: [root@tempest-web dashboard]# su q75r136 -c ‘source /opt/ood/ondemand/enable; bin/rake --trace -T test:jobs:tempest RAILS_ENV=production SUBMIT_ARGS=“-A group-christineforeman”’
rake test:jobs:tempest # Test the cluster: tempest
rake test:jobs:tempest-nextgen # Test the cluster: tempest-nextgen
so this fixed something? I did this for user=q75r136 ( our school netid format). Does he need to add anything to his login script(s)?
Kenny
Omg we just installed microconda because we couldn’t install qiime2 v2026.4 under the Mamba module we have. His .bash_profile is empty, so i called source .bashrc from there. I’ll comment that out… talk about going down a few rabbit holes 
K
Aaaand I blew OOD up. lol Oh boy… got a meeting. I’ll bb.
Kenny
Hi Jeff, After all I did to try and figure out what I thought was a complex issue, it turns out that we experienced some pretty significant delays in jobs starting throughout the cluster. When we finally realized that my customer’s job is running and he could get in fine, we determined delays up to 5 minutes for a job that starts in 3 seconds flat on a normal run.
At least I realized that we need to pay closer attention to how OOD runs under the hood. There’s a lot to this system. 
Always appreciate OOD’s time is assisting. I’ll close this.
Kenny
