Dashboard/Proxy Time Out

Get Help
1

Hi,

We’ve just launched a new Open OnDemand portal to our users but getting some complaints from users receiving an error

looking at the ondemand-nginx logs we see something is timing out

App 1788706 output: /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1/gems/activesupport-7.0.8.5/lib/active_support/core_ext/object.rb:13: warning: bigdecimal was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
App 1788706 output: You can add bigdecimal to your Gemfile or gemspec to silence this warning.
App 1788706 output: /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1/gems/activesupport-7.0.8.5/lib/active_support/notifications.rb:4: warning: mutex_m was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
App 1788706 output: You can add mutex_m to your Gemfile or gemspec to silence this warning.
[ E 2025-08-13 14:09:16.8008 1788657/T18 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/ood/apps/sys/dashboard: A timeout occurred while spawning an application process.
  Error ID: 7ad23d48
  Error details saved to: /tmp/passenger-error-pGwivk.html

[ E 2025-08-13 14:09:16.8045 1788657/T9 age/Cor/Con/CheckoutSession.cpp:284 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is 7ad23d48. Please see earlier logs for details about the error

Watching the processes on an attempt to login we see the process stuck doing IO is

ruby /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb

I’ve managed to strace the process and there are no obvious error, it just eventually receives a SIGKILL

openat(AT_FDCWD, "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1/gems/ood_appkit-2.1.6/lib/browser/platform/windows_mobile.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1/gems/redcarpet-3.6.0/lib/browser/platform/windows_mobile.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1/extensions/x86_64-linux/3.3.0/redcarpet-3.6.0/browser/platform/windows_mobile.rb", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
+++ killed by SIGKILL +++

I do notice users home directories, which are on network storage, are also being searched. The timing from the strace output for the open syscalls are significantly longer when searching these than the local gems directories.

GEM_PATH seems to be set without the users home directory (trailing “:” included)

GEM_PATH=/opt/ood/ondemand/root/usr/share/gems/3.3:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1:

But when I look in the HTML error file fro passenger I see the users home directory included

RUBY_VERSION = 3.3.8
RUBY_PLATFORM = x86_64-linux
RUBY_ENGINE = ruby
RubyGems version = 3.5.22
RubyGems paths = ["/opt/ood/ondemand/root/usr/share/gems/3.3", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1", "/home/user/.local/share/gem/ruby", "/usr/share/gems", "/usr/local/share/gems"]

Is there a way to exclude this from being searched. Most users don’t have that directory and if they do I don’t think we would want to load any gems from that location anyway.

2

You could remove some ondemand-gem packages if you have a lot of them to reduce the gems being loaded.

That said - I’m not 100% sure if this will work, but maybe you can set GEM_HOME=/dev/null in the PUN’s environment?

3

Hi Jeff, thanks for the quick reply. I’ve tried setting GEM_HOME=/dev/null but that doesn’t seem to make a difference.

From what I can see from looking at /proc/<pid>/environ the GEM_HOME and GEM_PATH env variables are set correctly for the process.

The trailing colon on GEM_PATH does seem to have some effect, at least when running ruby from the command line

# echo $GEM_PATH
/opt/ood/ondemand/root/usr/share/gems/3.3:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1:

# ruby -r rubygems -e 'p Gem.path'
["/opt/ood/ondemand/root/usr/share/gems/3.3", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1", "/home/user/.local/share/gem/ruby", "/usr/share/gems", "/usr/local/share/gems"]

vs when it is exported without

# echo $GEM_PATH
/opt/ood/ondemand/root/usr/share/gems/3.3:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1:/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1

# ruby -r rubygems -e 'p Gem.path'
["/opt/ood/ondemand/root/usr/share/gems/3.3", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.6-1", "/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.3-1"]

We still see lots of hits in the strace output for home directories though

# grep home /tmp/nginx_strace_1755173216.out | wc -l
2845

Looking closer at that strace output though, I notice it only for 3 gems/ruby libraries

# grep home /tmp/nginx_strace_1755173216.out | grep openat | awk '{ print $3 }' | awk -F'/' '{ print $10 }' | sort | uniq -c
    651 base64-0.2.0
      1 cache",
    693 net-protocol-0.2.2
    913 ruby2_keywords-0.0.5

I don’t have any gems installed in my home directory so not sure why it would be looking for those specifically in my home directory. it doesn’t seem to look for them elsewhere. if we filter all the hits for the home directory out we end up with just

# grep base64-0.2.0 /tmp/nginx_strace_1755173216.out | grep -v home
14940 stat("/usr/share/gems/specifications/default/base64-0.2.0.gemspec", {st_mode=S_IFREG|0644, st_size=1030, ...}) = 0 <0.000032>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000032>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000029>
14940 stat("/usr/share/gems/specifications/default/base64-0.2.0.gemspec", {st_mode=S_IFREG|0644, st_size=1030, ...}) = 0 <0.000026>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000025>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000025>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000029>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000029>
14940 stat("/usr/share/gems/specifications/default/base64-0.2.0.gemspec", {st_mode=S_IFREG|0644, st_size=1030, ...}) = 0 <0.000032>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000032>
14940 openat(AT_FDCWD, "/usr/share/gems/specifications/default/base64-0.2.0.gemspec", O_RDONLY|O_CLOEXEC) = 6 <0.000031>

I’m not familar with Ruby so trying to learn as I go.

In any case, our network storage seems much happier today and actually loading the dashboard hasn’t been an issue. Is there a way to extend the passenger timeout value to weather those times it’s not as responsive as it should be?

4

Yea there’s this Passenger configuration. I’ll look more into

Also maybe in that same /etc/ood/profile you can somehow remove HOME from the GEM_PATH. I’m sure there’s some sed expression to do that, though I don’t know it off hand.

5

Thank you again for the quick reply. I should have mentioned I did try messing around with GEM_PATH in /opt/ood/ondemand/enable where it looks like it’s built up. Modifying the loop to exclude the trailing colon doesn’t seem to have an effect, though it looks like it’s getting passed through to the process.

In the HTML error file from passenger, under the subprocess tab I see that the path for those libraries is set explicitly to the home location under load_path, while all the others are /opt/ood/ondemand/.. or /usr

/home/user/.local/share/gem/ruby/gems/ruby2_keywords-0.0.5/lib
/home/user/.local/share/gem/ruby/gems/net-protocol-0.2.2/lib
/home/user/.local/share/gem/ruby/gems/base64-0.2.0/lib