NGINX reverse proxy error after OOD upgrade

Get Help
1

Hello!

Recently, I upgraded an instance of Open OnDemand from version 3.0.1 to 3.1.1. The upgrade was successfully completed, and it’s possible to use OOD without issues. However, I use NGINX as a reverse proxy to access OOD externally, and it has been presenting the following errors when I try to create an interactive session (using OOD locally works without problems):

App 915075 output: [2024-02-28 16:37:56 -0300 ]  WARN "HTTP Origin header (https://url) didn't match request.base_url (https://IP)"
App 915075 output: [2024-02-28 16:37:56 -0300 ]  INFO "method=POST path=/pun/sys/dashboard/batch_connect/sys/rstudio/session_contexts format=html controller=BatchConnect::SessionContextsController action=create status=422 error='ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken' allocations=132 duration=0.99 view=0.00"
App 915075 output: [2024-02-28 16:37:56 -0300 ] FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):\n  \nactionpack (6.1.7.6) lib/action_controller/metal/request_forgery_protection.rb:211:in `handle_unverified_request'\nactionpack (6.1.7.6) lib/action_controller/metal/request_forgery_protection.rb:243:in `handle_unverified_request'\nactionpack (6.1.7.6) lib/action_controller/metal/request_forgery_protection.rb:238:in `verify_authenticity_token'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:427:in `block in make_lambda'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'\nactionpack (6.1.7.6) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:199:in `block in halting'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:512:in `block in invoke_before'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:512:in `each'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:512:in `invoke_before'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:105:in `run_callbacks'\nactionpack (6.1.7.6) lib/abstract_controller/callbacks.rb:41:in `process_action'\nactionpack (6.1.7.6) lib/action_controller/metal/rescue.rb:22:in `process_action'\nactionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\nactivesupport (6.1.7.6) lib/active_support/notifications.rb:203:in `block in instrument'\nactivesupport (6.1.7.6) lib/active_support/notifications/instrumenter.rb:24:in `instrument'\nactivesupport (6.1.7.6) lib/active_support/notifications.rb:203:in `instrument'\nactionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:33:in `process_action'\nactionpack (6.1.7.6) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'\nactionpack (6.1.7.6) lib/abstract_controller/base.rb:165:in `process'\nactionview (6.1.7.6) lib/action_view/rendering.rb:39:in `process'\nactionpack (6.1.7.6) lib/action_controller/metal.rb:190:in `dispatch'\nactionpack (6.1.7.6) lib/action_controller/metal.rb:254:in `dispatch'\nactionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'\nactionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:33:in `serve'\nactionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:50:in `block in serve'\nactionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in `each'\nactionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in `serve'\nactionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:842:in `call'\nrack (2.2.8) lib/rack/tempfile_reaper.rb:15:in `call'\nrack (2.2.8) lib/rack/etag.rb:27:in `call'\nrack (2.2.8) lib/rack/conditional_get.rb:40:in `call'\nrack (2.2.8) lib/rack/head.rb:12:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/http/permissions_policy.rb:22:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/http/content_security_policy.rb:19:in `call'\nrack (2.2.8) lib/rack/session/abstract/id.rb:266:in `context'\nrack (2.2.8) lib/rack/session/abstract/id.rb:260:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/cookies.rb:697:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'\nactivesupport (6.1.7.6) lib/active_support/callbacks.rb:98:in `run_callbacks'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\nlograge (0.14.0) lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'\nrailties (6.1.7.6) lib/rails/rack/logger.rb:26:in `block in call'\nactivesupport (6.1.7.6) lib/active_support/tagged_logging.rb:99:in `block in tagged'\nactivesupport (6.1.7.6) lib/active_support/tagged_logging.rb:37:in `tagged'\nactivesupport (6.1.7.6) lib/active_support/tagged_logging.rb:99:in `tagged'\nrailties (6.1.7.6) lib/rails/rack/logger.rb:26:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'\nrequest_store (1.5.1) lib/request_store/middleware.rb:19:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/request_id.rb:26:in `call'\nrack (2.2.8) lib/rack/method_override.rb:24:in `call'\nrack (2.2.8) lib/rack/runtime.rb:22:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/executor.rb:14:in `call'\nrack (2.2.8) lib/rack/sendfile.rb:110:in `call'\nactionpack (6.1.7.6) lib/action_dispatch/middleware/host_authorization.rb:148:in `call'\nrailties (6.1.7.6) lib/rails/engine.rb:539:in `call'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'"                             App 915075 output: [2024-02-28 16:37:59 -0300 ]  INFO "method=GET path=/pun/sys/dashboard/batch_connect/sessions.js format=js controller=BatchConnect::SessionsController action=index status=200 allocations=5756 duration=29.20 view=0.76"

Could you assist me with tips or suggestions for this issue? Currently, I have tested the following solutions: ruby on rails - Why is my HTTP Origin header not matching request.base_url and how to fix? - Stack Overflow and ruby on rails - ActionController::InvalidAuthenticityToken - Stack Overflow, but without success.

Thanks,
Nícolas

2

What does your nginx config look like?

3

Hi Andreas!

Here’s the Nginx config file:
nginx.txt (2.1 KB)

Just a quick note, before the OOD update, it was possible to create interactive sessions without any issues via this external access.

4

It looks to me like your ssl certificates and ondemand https config aren’t set up so that the origin headers from the browser hitting the nginx are correct from the perspective of the ondemand httpd. I think what you want is that the ssl certificates for ondemand and “servername” in ood_portal.yml match the external URL of your nginx, f.ex by using SAN (Subject Alternative Name) when you create the certificates.

Not sure how your setup used to work before the upgrade though. Did you create new certificates and forgot the SAN fields maybe? In any case, it should work if you set it up as described above.

5

Hi Andreas!

Thank you for your response!

Actually, the SSL certificates are properly configured according to the server name, it was just modified for sending purposes.

6

This has to do with how you’re forwarding requests. OOD makes sure that the servername, any server_aliases and proxy_server get put on this list.

So, for example our production server is ondemand.osc.edu. This is the servername but also and more importantly, it’s the host we use in the browsers URL.

It seems that the hostname you’re using in the browser doesn’t match any host we’ve automatically added to the allowlist.

Also thanks @buzh for chiming in!

7

Hi Jeff!

Thank you very much for the response!

8

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.