I have installed OOD 3.0.3 on two RHEL9 systems that are almost identical except that each connects to a different Slurm cluster. On one system it works and on the other, when trying to load the homepage, authentication succeeds and the page (partially) loads with the expected menu at the top, followed by the error:
Internal Server Error
Please try again in a few moments. If you continue to see this error, please contact support.
Details:
#<Errno::EACCES: Permission denied @ dir_s_mkdir - /var/www/ood/apps/sys/dashboard/tmp/cache>
(will include stacktrace below)
From what I’ve read, it seems that this error indicates that it’s trying to create a cache directory at an incorrect location. I think some configuration problem on my system is causing this. I have checked:
# grep cache_store /var/www/ood/apps/sys/dashboard/config/environments/production.rb
config.cache_store = :memory_store
/etc/ood/config/apps/dashboard/env sets TMPDIR and the path it’s set to is present with world read and write access.
/etc/ood/config/apps/dashboard/initializers/ood.rb sets some paths like:
paths << FavoritePath.new("/some")
paths << FavoritePath.new("/path")
and all these paths are present and accessible.
The same package versions are installed on the working and broken systems (including ruby 3.0.4, nodejs 16.20.2). I have also rebuilt both of these systems, ensuring that the same build process was used for each.
What next steps I can take to debug this issue?
Stacktrace:
#<Errno::EACCES: Permission denied @ dir_s_mkdir - /var/www/ood/apps/sys/dashboard/tmp/cache>
/usr/share/ruby/fileutils.rb:253:in `mkdir'
/usr/share/ruby/fileutils.rb:253:in `fu_mkdir'
/usr/share/ruby/fileutils.rb:231:in `block (2 levels) in mkdir_p'
/usr/share/ruby/fileutils.rb:229:in `reverse_each'
/usr/share/ruby/fileutils.rb:229:in `block in mkdir_p'
/usr/share/ruby/fileutils.rb:211:in `each'
/usr/share/ruby/fileutils.rb:211:in `mkdir_p'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache/file_store.rb:89:in `set'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache.rb:227:in `set'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache.rb:93:in `fetch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:71:in `file_digest'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/unloaded_asset.rb:105:in `dependency_history_key'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/loader.rb:324:in `fetch_asset_from_dependency_cache'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/loader.rb:43:in `load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cached_environment.rb:44:in `block in load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:207:in `block in fetch_or_store'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:187:in `fetch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:206:in `fetch_or_store'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cached_environment.rb:44:in `load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:81:in `find_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:119:in `[]'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:358:in `find_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:337:in `digest_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:326:in `asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:103:in `block in resolve_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:243:in `block in resolve_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `detect'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `resolve_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:102:in `resolve_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:80:in `compute_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_url_helper.rb:203:in `asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_url_helper.rb:321:in `javascript_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:98:in `block in javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:97:in `map'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:97:in `javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:157:in `block in javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:155:in `map'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:155:in `javascript_include_tag'
/var/www/ood/apps/sys/dashboard/app/views/dashboard/index.html.erb:2:in `block in _app_views_dashboard_index_html_erb__2669434056597945246_63160'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `capture'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:159:in `content_for'
/var/www/ood/apps/sys/dashboard/app/views/dashboard/index.html.erb:1:in `_app_views_dashboard_index_html_erb__2669434056597945246_63160'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `public_send'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `_run'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:154:in `block in render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:345:in `instrument_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:152:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:71:in `block in render_with_layout'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:70:in `render_with_layout'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:11:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:117:in `block in _render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:273:in `in_rendering_context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:116:in `_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/streaming.rb:218:in `_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:103:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/rendering.rb:25:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:36:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
/usr/share/ruby/benchmark.rb:308:in `realtime'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:45:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
<internal:kernel>:90:in `tap'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/lograge-0.12.0/lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'