I get "InvalidAuthenticityToken" on performing most actions

[georgets (George Tsouloupas)]

Hi,
We are trying out On-demand.

• I’ve installed on-demand on a test cluster.
• I have set up LDAP authentication and set up the (local) slurm cluster (OOD runs on the cluster’s headnode)
• I can log in as “george” and see existing jobs that I have submitted from the command-line

The issue: Most operations (e.g. delete a job) return a 422 error and the logs mention
“Can’t verify CSRF token authenticity.”
and ActionController::InvalidAuthenticityToken.
(see log output below)

Any pointers? Is there a way to get more detailed logging?

Many thanks!
George.

Logged in as “george” and looking under /var/log/ondemand-nginx/george/error.log I find the following:

pp 8548 output: [2021-06-08 23:12:51 +0300 ] WARN “Can’t verify CSRF token authenticity.”
App 8548 output: [2021-06-08 23:12:51 +0300 ] INFO “method=DELETE path=/pun/sys/dashboard/activejobs format=html controller=ActiveJobsController action=delete_job status=422 error=‘ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticit
yToken’ duration=0.97 view=0.00”
App 8548 output: [2021-06-08 23:12:51 +0300 ] FATAL “”
App 8548 output: [2021-06-08 23:12:51 +0300 ] FATAL “ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):”
App 8548 output: [2021-06-08 23:12:51 +0300 ] FATAL “”
App 8548 output: [2021-06-08 23:12:51 +0300 ] FATAL “actionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:215:in handle_unverified_request'\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:247:in handle_unve
rified_request’\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:242:in verify_authenticity_token'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:426:in block in make_lambda’\nactivesupport (5.2.6) lib/active_support/callba
cks.rb:198:in block (2 levels) in halting'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:34:in block (2 levels) in module:Callbacks’\nactivesupport (5.2.6) lib/active_support/callbacks.rb:199:in block in halting'\nactivesupport (5.2.6) lib/active _support/callbacks.rb:513:in block in invoke_before’\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in each'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in invoke_before’\nactivesupport (5.2.6) lib/active_support/callbacks.rb:131:in
run_callbacks'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:41:in process_action’\nactionpack (5.2.6) lib/action_controller/metal/rescue.rb:22:in process_action'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:34:in block in p
rocess_action’\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in block in instrument'\nactivesupport (5.2.6) lib/active_support/notifications/instrumenter.rb:23:in instrument’\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in i nstrument'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:32:in process_action’\nactionpack (5.2.6) lib/action_controller/metal/params_wrapper.rb:256:in process_action'\nactionpack (5.2.6) lib/abstract_controller/base.rb:134:in process’\na
ctionview (5.2.6) lib/action_view/rendering.rb:32:in process'\nactionpack (5.2.6) lib/action_controller/metal/live.rb:255:in block (2 levels) in process’\nactivesupport (5.2.6) lib/active_support/dependencies/interlock.rb:42:in block in running'\nactivesuppor t (5.2.6) lib/active_support/concurrency/share_lock.rb:162:in sharing’\nactivesupport (5.2.6) lib/active_support/dependencies/interlock.rb:41:in running'\nactionpack (5.2.6) lib/action_controller/metal/live.rb:247:in block in process’\nactionpack (5.2.6) lib/
action_controller/metal/live.rb:291:in `block in new_controller_thread’”

[jeff.ohrstrom]

Hi and welcome! Are you using a proxy of some kind? The issue is the token is generated with scheme://host:port (and maybe path?) on the server side. So if you connect through proxy (a different host) it may be throwing this error.

[georgets (George Tsouloupas)]

Thanks for the reply!
No proxy, but we do have to go through a 1-1 NAT for internet access though. To your knowledge, is there a requirement for reverse-DNS to work? (i.e. hostname->IP and IP->hostname to be consistent).

I will keep at this. This is the kind of thing we have been looking for for a long time!

Thanks again,
g.

[MohsenQazi (Mohsen Ghazi)]

I have a similar situation; I can’t create a job.

And the log is same as @georgets

App 26991 output: [2021-06-16 12:44:42 +0430 ]  WARN "Can't verify CSRF token authenticity."
App 26991 output: [2021-06-16 12:44:42 +0430 ]  INFO "method=POST path=/pun/sys/myjobs/create_from_path format=html controller=WorkflowsController action=create_from_path status=422 error='ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken' duration=0.46 view=0.00 db=0.00"
App 26991 output: [2021-06-16 12:44:42 +0430 ] FATAL ""
App 26991 output: [2021-06-16 12:44:42 +0430 ] FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):"
App 26991 output: [2021-06-16 12:44:42 +0430 ] FATAL ""
App 26991 output: [2021-06-16 12:44:42 +0430 ] FATAL "actionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:215:in `handle_unverified_request'\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:247:in `handle_unverified_request'\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:242:in `verify_authenticity_token'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:426:in `block in make_lambda'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:199:in `block in halting'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `block in invoke_before'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `each'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `invoke_before'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:131:in `run_callbacks'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:41:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/rescue.rb:22:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in `block in instrument'\nactivesupport (5.2.6) lib/active_support/notifications/instrumenter.rb:23:in `instrument'\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in `instrument'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:32:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\nactiverecord (5.2.6) lib/active_record/railties/controller_runtime.rb:24:in `process_action'\nactionpack (5.2.6) lib/abstract_controller/base.rb:134:in `process'\nactionview (5.2.6) lib/action_view/rendering.rb:32:in `process'\nactionpack (5.2.6) lib/action_controller/metal.rb:191:in `dispatch'\nactionpack (5.2.6) lib/action_controller/metal.rb:252:in `dispatch'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:34:in `serve'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:52:in `block in serve'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:35:in `each'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:35:in `serve'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:840:in `call'\nrack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'\nrack (2.2.3) lib/rack/etag.rb:27:in `call'\nrack (2.2.3) lib/rack/conditional_get.rb:40:in `call'\nrack (2.2.3) lib/rack/head.rb:12:in `call'\nactionpack (5.2.6) lib/action_dispatch/http/content_security_policy.rb:18:in `call'\nrack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'\nrack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/cookies.rb:670:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:98:in `run_callbacks'\nactionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\nlograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'\nrailties (5.2.6) lib/rails/rack/logger.rb:26:in `block in call'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `block in tagged'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:28:in `tagged'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `tagged'\nrailties (5.2.6) lib/rails/rack/logger.rb:26:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'\nrequest_store (1.5.0) lib/request_store/middleware.rb:19:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/request_id.rb:27:in `call'\nrack (2.2.3) lib/rack/method_override.rb:24:in `call'\nrack (2.2.3) lib/rack/runtime.rb:22:in `call'\nactivesupport (5.2.6) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/executor.rb:14:in `call'\nrack (2.2.3) lib/rack/sendfile.rb:110:in `call'\nrailties (5.2.6) lib/rails/engine.rb:524:in `call'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'"

“Create From Default Template” option has the same result, too.

[jeff.ohrstrom]

Hi I just updated a similar issue on our Github. Please see it for steps to remediate.

HTTP 422 Error / InvalidAuthenticityToken

[MohsenQazi (Mohsen Ghazi)]

Hello @jeff.ohrstrom ;
Unfortunately, My problem is still there!

I tested both of your solutions for /var/www/ood/apps/sys/dashboard/config/initializers/session_store.rb ; Stop and start the httpd24-httpd; Login to OOD but create a job from template, for instance, shows up the page I uploaded in my previous reply.

[jeff.ohrstrom]

Sorry! That ticket edited the dashboard app for the same issue. I didn’t realize you’re looking at the job composer (myjobs)

Edit the file in the same way as the ticket, only you’ll note it’s in the directory for myjobs app.

/var/www/ood/apps/sys/myjobs/config/initializers/session_store.rb

[jeff.ohrstrom]

Did that work for you @MohsenQazi ?

[georgets (George Tsouloupas)]

By the way, my issue was resolved after setting up https…
(on 2.10)

g.

[jeff.ohrstrom]

Yep that works too. We force session cookies to use SSL in 2.0.x, so if you don’t have SSL you either need to disable that flag or use local storage for session data. Or just setup SSL.

If you do end up modifying files in /var/ww/ood they’ll be overwritten by updates. There’s a ticket to make this session store configurable that’ll be out in 2.0.14 or so.