After a Reboot and a number of changes to our slurm cluster all users are getting the following error logging int our OOD(4.0.8).
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server
The OIDC login flow works and immediately after redirecting to our OOD server they get the error I can see the nginx: worker process starting as the user in top.
/var/log/ondemand-nginx/error.log
2025/12/22 13:04:16 [notice] 36037#0: signal process started
2025/12/22 14:00:01 [notice] 54866#0: signal process started
2025/12/22 14:00:01 [notice] 54877#0: signal process started
/var/log/ondemand-nginx/$USER/error.log
[ N 2025-12-22 14:00:01.4678 36061/T9 age/Cor/CoreMain.cpp:671 ]: Signal received. Gracefully shutting down… (send signal 2 more time(s) to force shutdown)
[ N 2025-12-22 14:00:01.4678 36061/T1 age/Cor/CoreMain.cpp:1246 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected…
[ N 2025-12-22 14:00:01.4678 36061/T9 Ser/Server.h:901 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/T9 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Td Ser/Server.h:901 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Td Ser/Server.h:558 ]: [ServerThr.3] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tf Ser/Server.h:901 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Tf Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tj Ser/Server.h:901 ]: [ServerThr.6] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Th Ser/Server.h:901 ]: [ServerThr.5] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Tb Ser/Server.h:901 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Th Ser/Server.h:558 ]: [ServerThr.5] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tr Ser/Server.h:901 ]: [ServerThr.10] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Tl Ser/Server.h:901 ]: [ServerThr.7] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Tb Ser/Server.h:558 ]: [ServerThr.2] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tr Ser/Server.h:558 ]: [ServerThr.10] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tn Ser/Server.h:901 ]: [ServerThr.8] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4679 36061/Tn Ser/Server.h:558 ]: [ServerThr.8] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tv Ser/Server.h:901 ]: [ServerThr.12] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/Tv Ser/Server.h:558 ]: [ServerThr.12] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tt Ser/Server.h:901 ]: [ServerThr.11] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/Tp Ser/Server.h:901 ]: [ServerThr.9] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/T11 Ser/Server.h:901 ]: [ServerThr.15] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/Tt Ser/Server.h:558 ]: [ServerThr.11] Shutdown finished
[ N 2025-12-22 14:00:01.4680 36061/Tp Ser/Server.h:558 ]: [ServerThr.9] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tx Ser/Server.h:901 ]: [ServerThr.13] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/T11 Ser/Server.h:558 ]: [ServerThr.15] Shutdown finished
[ N 2025-12-22 14:00:01.4680 36061/Tz Ser/Server.h:901 ]: [ServerThr.14] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/Tx Ser/Server.h:558 ]: [ServerThr.13] Shutdown finished
[ N 2025-12-22 14:00:01.4680 36061/Tz Ser/Server.h:558 ]: [ServerThr.14] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tj Ser/Server.h:558 ]: [ServerThr.6] Shutdown finished
[ N 2025-12-22 14:00:01.4679 36061/Tl Ser/Server.h:558 ]: [ServerThr.7] Shutdown finished
[ N 2025-12-22 14:00:01.4680 36061/T15 Ser/Server.h:901 ]: [ApiServer] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4680 36061/T15 Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2025-12-22 14:00:01.4680 36061/T13 Ser/Server.h:901 ]: [ServerThr.16] Freed 0 spare client objects
[ N 2025-12-22 14:00:01.4681 36061/T13 Ser/Server.h:558 ]: [ServerThr.16] Shutdown finished
[ N 2025-12-22 14:00:01.6431 36061/T1 age/Cor/CoreMain.cpp:1326 ]: Passenger core shutdown finished
[ N 2025-12-22 15:28:55.1544 4730/T1 age/Wat/WatchdogMain.cpp:1370 ]: Starting Passenger watchdog…
[ N 2025-12-22 15:28:55.5140 4737/T1 age/Cor/CoreMain.cpp:1341 ]: Starting Passenger core…
[ N 2025-12-22 15:28:55.5142 4737/T1 age/Cor/CoreMain.cpp:257 ]: Passenger core running in multi-application mode.
[ N 2025-12-22 15:28:55.5620 4737/T1 age/Cor/CoreMain.cpp:1016 ]: Passenger core online, PID 4737
App 4796 output: /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.8-1/gems/activesupport-7.0.8.5/lib/active_support/core_ext/object.rb:13: warning: bigdecimal was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
App 4796 output: You can add bigdecimal to your Gemfile or gemspec to silence this warning.
App 4796 output: /opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.8-1/gems/activesupport-7.0.8.5/lib/active_support/notifications.rb:4: warning: mutex_m was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.4.0.
App 4796 output: You can add mutex_m to your Gemfile or gemspec to silence this warning.
App 4796 output: [2025-12-22 15:30:07 -0700 ] WARN “Error opening MOTD at \nException: bad URI(is not URI?): nil”
App 4796 output: [2025-12-22 15:30:07 -0700 ] INFO “method=GET path=/pun/sys/dashboard/ format=html controller=DashboardController action=index status=200 allocations=122419 duration=67928.66 view=141.73”
[ N 2025-12-22 15:33:59.8321 4737/T5 age/Cor/CoreMain.cpp:1147 ]: Checking whether to disconnect long-running connections for process 4796, application /var/www/ood/apps/sys/dashboard (production)
Thanks for any help.