JupyterLab disconnections when using OOD Apache's ProxyPass but working fine when connecting directly to the node

[danielng]

Hello!

I bring a kind of a strange issue that my users are facing when using JupyterLab in OOD v2.0.31.

They are reporting that when they have a Jupyter job active for long hours (more than 8h and doesn’t need to be actually processing anything, just the notebook alive), they find that around 8h after the job starts, if they are actively working on the notebook (typing code, getting results, etc) their web client gets disconnected from the Jupyter server getting the error I show in the following screenshot:

Trying to debug the issue, since nothing seemed to be found on the logs and the error is quite time consuming to replicate, I created a secondary “Connect to Jupyter” button in OOD that instead of using the apache ProxyPass url (https://ondemand.[SERVER]/node/[NODE_IP]/[PORT]/lab?) connects directly to the JupyterLab port on the running compute node (yes, clients do have network access to compute nodes) using an URL like “http://[NODE_IP]:[PORT]/node/[NODE_IP]/[PORT]/lab?”

Using this new URL all the errors and issues went away, so my conclusion is that the issue should be related to the Apache’s ProxyPass configuration, any ideas of which kind of options could be changed?

Just for refreshing memories, here is the correspondent Apache config for the ProxyPass:

# Reverse proxy traffic to backend webserver through IP sockets:
  #
  #     https://ondemand.gritstone.com:443/node/HOST/PORT/index.html
  #     #=> http://HOST:PORT/node/HOST/PORT/index.html
  #
  <LocationMatch "^/node/(?<host>[^/]+)/(?<port>\d+)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^[^/]+//[^/]+" ""

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

Thanks.

[travert (Travis Ravert)]

Thanks for the question. What authentication are you using?

The timing seems to line pretty well with the oidc_session_max_duration setting:
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html?highlight=oidc_session_max_duration

Which looks to just use the default from the auth_openidc:

OpenIDC/mod_auth_openidc/blob/a6c2d668e137c5efed5aef85ca6b09c3bbae9f90/auth_openidc.conf#L590

[danielng]

I really believe you got it!

We do use OpenID for authentication and our tokens last more than 8h, so it makes sense that the user is not getting a new token but Apache is considering that one “expired” due to the 8h default config, I’m going to test it out and report back.