OOD 3.0.3 Permission denied @ dir_s_mkdir

I have installed OOD 3.0.3 on two RHEL9 systems that are almost identical except that each connects to a different Slurm cluster. On one system it works and on the other, when trying to load the homepage, authentication succeeds and the page (partially) loads with the expected menu at the top, followed by the error:

Internal Server Error
Please try again in a few moments. If you continue to see this error, please contact support.
Details:
#<Errno::EACCES: Permission denied @ dir_s_mkdir - /var/www/ood/apps/sys/dashboard/tmp/cache>

(will include stacktrace below)

From what I’ve read, it seems that this error indicates that it’s trying to create a cache directory at an incorrect location. I think some configuration problem on my system is causing this. I have checked:

# grep cache_store /var/www/ood/apps/sys/dashboard/config/environments/production.rb 
  config.cache_store = :memory_store

/etc/ood/config/apps/dashboard/env sets TMPDIR and the path it’s set to is present with world read and write access.

/etc/ood/config/apps/dashboard/initializers/ood.rb sets some paths like:

    paths << FavoritePath.new("/some")
    paths << FavoritePath.new("/path")

and all these paths are present and accessible.

The same package versions are installed on the working and broken systems (including ruby 3.0.4, nodejs 16.20.2). I have also rebuilt both of these systems, ensuring that the same build process was used for each.

What next steps I can take to debug this issue?

Stacktrace:

#<Errno::EACCES: Permission denied @ dir_s_mkdir - /var/www/ood/apps/sys/dashboard/tmp/cache>

/usr/share/ruby/fileutils.rb:253:in `mkdir'
/usr/share/ruby/fileutils.rb:253:in `fu_mkdir'
/usr/share/ruby/fileutils.rb:231:in `block (2 levels) in mkdir_p'
/usr/share/ruby/fileutils.rb:229:in `reverse_each'
/usr/share/ruby/fileutils.rb:229:in `block in mkdir_p'
/usr/share/ruby/fileutils.rb:211:in `each'
/usr/share/ruby/fileutils.rb:211:in `mkdir_p'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache/file_store.rb:89:in `set'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache.rb:227:in `set'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cache.rb:93:in `fetch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:71:in `file_digest'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/unloaded_asset.rb:105:in `dependency_history_key'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/loader.rb:324:in `fetch_asset_from_dependency_cache'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/loader.rb:43:in `load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cached_environment.rb:44:in `block in load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:207:in `block in fetch_or_store'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:187:in `fetch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/map.rb:206:in `fetch_or_store'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/cached_environment.rb:44:in `load'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:81:in `find_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-4.2.0/lib/sprockets/base.rb:119:in `[]'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:358:in `find_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:337:in `digest_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:326:in `asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:103:in `block in resolve_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:243:in `block in resolve_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `detect'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:242:in `resolve_asset'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:102:in `resolve_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:80:in `compute_asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_url_helper.rb:203:in `asset_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_url_helper.rb:321:in `javascript_path'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:98:in `block in javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:97:in `map'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/asset_tag_helper.rb:97:in `javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:157:in `block in javascript_include_tag'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:155:in `map'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:155:in `javascript_include_tag'
/var/www/ood/apps/sys/dashboard/app/views/dashboard/index.html.erb:2:in `block in _app_views_dashboard_index_html_erb__2669434056597945246_63160'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:45:in `capture'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/helpers/capture_helper.rb:159:in `content_for'
/var/www/ood/apps/sys/dashboard/app/views/dashboard/index.html.erb:1:in `_app_views_dashboard_index_html_erb__2669434056597945246_63160'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `public_send'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:247:in `_run'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:154:in `block in render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:345:in `instrument_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/template.rb:152:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:61:in `block (2 levels) in render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:56:in `block in render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:71:in `block in render_with_layout'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:205:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:70:in `render_with_layout'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:55:in `render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/template_renderer.rb:11:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:117:in `block in _render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/base.rb:273:in `in_rendering_context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:116:in `_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/streaming.rb:218:in `_render_template'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:103:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/rendering.rb:25:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:36:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
/usr/share/ruby/benchmark.rb:308:in `realtime'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:45:in `render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
<internal:kernel>:90:in `tap'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/lograge-0.12.0/lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

Hey and sorry for the issues.

For the logging side of this we have a page around what logs will show you which information here: Debugging and Monitoring — Open OnDemand 3.0.3 documentation

Do you see this file being created on the other system? I’d agree there’s something on the one system that is writing this file for some reason, but I’m curious to know if it happens on the other. It shouldn’t, but that can at least clue in on if there’s something set in one and not the other.

I’m wondering looking at this more if somehow Sprockets thinks it needs this cache to write to.

This has me wondering if somehow the change in that env is causing this. What is that set to cuurently?

Update: We found that there were a lot of differences in /var/www/ood/apps/sys/dashboard (outside of locations with cache in their names) between the working and broken cases, and copying the working version of the app to the other system seems to fix the problem. Still don’t know why the package install didn’t update that app as needed. It was done by:

sudo dnf install https://yum.osc.edu/ondemand/3.0/ondemand-release-web-3.0-1.noarch.rpm
sudo dnf install ondemand

and I’ve confirmed that the path /var/www/ood/apps/sys/dashboard is writable by root.