Websocket not working behind web proxy

louistw · July 3, 2024, 8:24pm

Hi,

Our site is trying to setup two OOD running as proxy backend and routing the user based on their system group.

The setting has three VM running CentOS 7, one as reverse proxy server and other two as OOD instance-1, and OOD instance-2. Both of the OOD instances are running OOD version 1.6.25-1.

The reverse proxy server to take the initial request and do authentication via shibboleth and then set the request header Proxy-User with the REMOTE_USER value and proxy the request to OOD.

With the current configuration(attached below), we are able to route users to different backends. However, the Interactive App(HPC Desktop) will throw 422 error when we hit the launch button. and the Shell App shows Failed to establish a websocket connection.

Could someone point me a direction here?

In both OOD instances, we have auth and user_env settings in the ood_portal.yaml as follow:

auth:
  - "RewriteCond %{IS_SUBREQ} ^false$"
  - "RewriteCond %{HTTP:Proxy-user} '([^!]+?)(@example.com)?$'"
  - "RewriteRule . - [E=PROXY_USER:%1]"

user_env: PROXY_USER

front end apache config:

<VirtualHost *:80>
  ServerName example.com
  RewriteEngine On
  RewriteRule ^(.*) https://example.com:443$1 [R=301,NE,L]
</VirtualHost>
<VirtualHost *:443>
  ServerName example.com

  RewriteEngine On
  RewriteCond %{HTTP_HOST} !^(example.com(:443)?)?$ [NC]
  RewriteRule ^(.*) https://example.com:443$1 [R=301,NE,L]

  # The script used to determine which backend server the user should go to
  # Based on system group
  RewriteEngine On
  RewriteMap grp "prg:/var/www/ood/register/rewrite_map.py" root:root

  # Support maintenance page during outages of OnDemand
  RewriteEngine On
  RewriteCond /var/www/ood/public/maintenance/index.html -f
  RewriteCond /etc/ood/maintenance.enable -f
  RewriteCond %{REQUEST_URI} !/public/maintenance/.*$
  RewriteRule ^.*$ /public/maintenance/index.html [R=302,L]

  Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"

  SSLEngine On
  SSLCertificateFile "/etc/pki/tls/certs/localhost.crt"
  SSLCertificateKeyFile "/etc/pki/tls/private/localhost.key"
  SSLProtocol -all +TLSv1.2
  SSLHonorCipherOrder On
  SSLCipherSuite  DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256
  ServerSignature Off
  TraceEnable Off

  LogLevel alert proxy:trace8
  LogLevel alert rewrite:trace8

  <Location "/">
    AuthType shibboleth
    ShibRequestSetting requireSession 1
    Require valid-user
    ShibUseHeaders On
    RewriteCond %{IS_SUBREQ} ^false$
    RewriteCond %{HTTP:REMOTE_USER} '([^!]+?)(@example\.com)?$'
    RewriteRule . - [E=PROXY_USER:%1]

    RequestHeader set Proxy-User %{PROXY_USER}e

    ProxyPreserveHost On
    RewriteCond %{HTTP:Connection} !upgrade [NC]
    RewriteCond %{HTTP:Upgrade} !websocket [NC]
    RewriteCond "%{ENV:PROXY_USER}" ^(.+)$
    RewriteCond "%{REQUEST_URI}" !index.html$
    RewriteCond "${grp:%1|login001.cm.cluster}" ^(.+)$
    RewriteRule .* "http://%1%{REQUEST_URI}" [P]

    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteCond "%{ENV:PROXY_USER}" ^(.+)$
    RewriteCond "${grp:%1|login001.cm.cluster}" ^(.+)$
    RewriteRule .* "ws://%1%{REQUEST_URI}" [P,L]

  </Location>

  ProxyPassReverse / "http://login001.cm.cluster/"
  ProxyPassReverse / "http://login002.cm.cluster/"
</VirtualHost>

jeff.ohrstrom · July 3, 2024, 8:34pm

I’m not entirely sure, but I would first track down who’s returning the 422. With so many proxies in the chain - I’m curious as to who/what is returning it.

As i understand it, the request chain looks like this. I’d want to confirm first who/which one is returning 422.

apache -> ood apache -> websockify server

jeff.ohrstrom · July 3, 2024, 8:35pm

Also - hi and welcome!

louistw · July 3, 2024, 9:09pm

Hi Jeff,

Thanks for the quick response.

After a few changes, the 422 error is gone, I am not sure which change fixes it. But I can now launch HPC Desktop and use the Desktop without issue.

The websocket issue with the Shell App still exists though. I turned on log for the Shell App in nginx config, and found 401 error from the Shell App.

I attached the log here:

2024/07/03 15:56:34 [debug] 20272#0: *28 http cl:-1 max:10737418240
2024/07/03 15:56:34 [debug] 20272#0: *28 rewrite phase: 3
2024/07/03 15:56:34 [debug] 20272#0: *28 post rewrite phase: 4
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 5
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 6
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 7
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 8
2024/07/03 15:56:34 [debug] 20272#0: *28 access phase: 9
2024/07/03 15:56:34 [debug] 20272#0: *28 access phase: 10
2024/07/03 15:56:34 [debug] 20272#0: *28 post access phase: 11
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 12
2024/07/03 15:56:34 [debug] 20272#0: *28 generic phase: 13
2024/07/03 15:56:34 [debug] 20272#0: *28 http script copy: "/var/www/ood/apps/sys/shell/public"
2024/07/03 15:56:34 [debug] 20272#0: *28 http script capture: "/ssh/login002"
2024/07/03 15:56:34 [debug] 20272#0: *28 add cleanup: 00005555559C5A70
2024/07/03 15:56:34 [debug] 20272#0: *28 http init upstream, client timer: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 epoll add event: fd:3 op:3 ev:80002005
2024/07/03 15:56:34 [debug] 20272#0: *28 malloc: 00005555559B9700:8814
2024/07/03 15:56:34 [debug] 20272#0: *28 http script copy: "X-Sendfile-Type: X-Accel-Redirect
"
2024/07/03 15:56:34 [debug] 20272#0: *28 http script copy: "X-Accel-Mapping: /=/sendfile
"
2024/07/03 15:56:34 [debug] 20272#0: *28 http cleanup add: 00005555559C6118
2024/07/03 15:56:34 [debug] 20272#0: *28 get rr peer, try: 1
2024/07/03 15:56:34 [debug] 20272#0: *28 stream socket 9
2024/07/03 15:56:34 [debug] 20272#0: *28 epoll add connection: fd:9 ev:80002005
2024/07/03 15:56:34 [debug] 20272#0: *28 connect to unix:/tmp/passenger.UsVwNlF/agents.s/core, fd:9 #29
2024/07/03 15:56:34 [debug] 20272#0: *28 connected
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream connect: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 posix_memalign: 0000555555ADB240:128 @16
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream send request
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream send request body
2024/07/03 15:56:34 [debug] 20272#0: *28 chain writer buf fl:1 s:8814
2024/07/03 15:56:34 [debug] 20272#0: *28 chain writer in: 00005555559C6158
2024/07/03 15:56:34 [debug] 20272#0: *28 writev: 8814 of 8814
2024/07/03 15:56:34 [debug] 20272#0: *28 chain writer out: 0000000000000000
2024/07/03 15:56:34 [debug] 20272#0: *28 event timer add: 9: 12000000:96554067
2024/07/03 15:56:34 [debug] 20272#0: *28 http finalize request: -4, "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc" a:1, c:2
2024/07/03 15:56:34 [debug] 20272#0: *28 http request count:2 blk:0
2024/07/03 15:56:34 [debug] 20272#0: *28 http run request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream check client, write event:1, "/pun/sys/shell/ssh/login002"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream dummy handler
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream dummy handler
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream process header
2024/07/03 15:56:34 [debug] 20272#0: *28 malloc: 0000555555A74F30:16384
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: eof:0, avail:1
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: fd:9 253 of 16384
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi status 401 "401 Unauthorized"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "Status: 401 Unauthorized"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "X-OOD-Failure-Reason: invalid origin"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "Content-Encoding: UTF-8"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "Content-Type: text/html; charset=UTF-8"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "Date: Wed, 03 Jul 2024 20:56:34 GMT"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "Connection: close"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header: "X-Powered-By: Phusion Passenger 5.3.7"
2024/07/03 15:56:34 [debug] 20272#0: *28 http scgi header done
2024/07/03 15:56:34 [debug] 20272#0: *28 xslt filter header
2024/07/03 15:56:34 [debug] 20272#0: *28 HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 401 Unauthorized
X-OOD-Failure-Reason: invalid origin
Content-Encoding: UTF-8
Date: Wed, 03 Jul 2024 20:56:34 GMT
X-Powered-By: Phusion Passenger 5.3.7
Server: nginx/1.14.0 + Phusion Passenger 5.3.7

2024/07/03 15:56:34 [debug] 20272#0: *28 write new buf t:1 f:0 00005555559C64F0, pos 00005555559C64F0, size: 334 file: 0, size: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter: l:0 f:0 s:334
2024/07/03 15:56:34 [debug] 20272#0: *28 http output filter "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http copy filter: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 image filter
2024/07/03 15:56:34 [debug] 20272#0: *28 xslt filter body
2024/07/03 15:56:34 [debug] 20272#0: *28 http postpone filter "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc" 00007FFFFFFFDE20
2024/07/03 15:56:34 [debug] 20272#0: *28 http chunk: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 write old buf t:1 f:0 00005555559C64F0, pos 00005555559C64F0, size: 334 file: 0, size: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 write new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter: l:0 f:1 s:334
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter limit 0
2024/07/03 15:56:34 [debug] 20272#0: *28 writev: 334 of 334
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter 0000000000000000
2024/07/03 15:56:34 [debug] 20272#0: *28 http copy filter: 0 "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream dummy handler
2024/07/03 15:56:34 [debug] 20272#0: *28 http run request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream process non buffered downstream
2024/07/03 15:56:34 [debug] 20272#0: *28 event timer: 9, old: 96554067, new: 96554072
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream request: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http upstream process non buffered upstream
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: eof:1, avail:1
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: fd:9 0 of 16384
2024/07/03 15:56:34 [debug] 20272#0: *28 finalize http upstream request: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 finalize Passenger request
2024/07/03 15:56:34 [debug] 20272#0: *28 free rr peer 1 0
2024/07/03 15:56:34 [debug] 20272#0: *28 close http upstream connection: 9
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555ADB240, unused: 48
2024/07/03 15:56:34 [debug] 20272#0: *28 event timer del: 9: 96554067
2024/07/03 15:56:34 [debug] 20272#0: *28 reusable connection: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 http output filter "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http copy filter: "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 image filter
2024/07/03 15:56:34 [debug] 20272#0: *28 xslt filter body
2024/07/03 15:56:34 [debug] 20272#0: *28 http postpone filter "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc" 00007FFFFFFFDE20
2024/07/03 15:56:34 [debug] 20272#0: *28 http chunk: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 write new buf t:0 f:0 0000000000000000, pos 00005555556F4E16, size: 5 file: 0, size: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter: l:1 f:0 s:5
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter limit 0
2024/07/03 15:56:34 [debug] 20272#0: *28 writev: 5 of 5
2024/07/03 15:56:34 [debug] 20272#0: *28 http write filter 0000000000000000
2024/07/03 15:56:34 [debug] 20272#0: *28 http copy filter: 0 "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc"
2024/07/03 15:56:34 [debug] 20272#0: *28 http finalize request: 0, "/pun/sys/shell/ssh/login002?csrf=wbFClBpv-pc0Nwx6Guk0yVZbEd9GjiBoT_Pc" a:1, c:1
2024/07/03 15:56:34 [debug] 20272#0: *28 set http keepalive handler
2024/07/03 15:56:34 [debug] 20272#0: *28 http close request
2024/07/03 15:56:34 [debug] 20272#0: *28 http log handler
2024/07/03 15:56:34 [debug] 20272#0: *28 run cleanup: 00005555559C5A70
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555A74F30
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 00005555559B9700
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555AD7E20, unused: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555ADCDB0, unused: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 00005555559C5940, unused: 61
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 00005555559B4AE0
2024/07/03 15:56:34 [debug] 20272#0: *28 hc free: 0000000000000000
2024/07/03 15:56:34 [debug] 20272#0: *28 hc busy: 0000555555ADF638 1
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555AD9030
2024/07/03 15:56:34 [debug] 20272#0: *28 reusable connection: 1
2024/07/03 15:56:34 [debug] 20272#0: *28 event timer add: 3: 75000:84629072
2024/07/03 15:56:34 [debug] 20272#0: *28 http empty handler
2024/07/03 15:56:34 [debug] 20272#0: *28 http keepalive handler
2024/07/03 15:56:34 [debug] 20272#0: *28 malloc: 00005555559B4AE0:1024
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: eof:1, avail:1
2024/07/03 15:56:34 [debug] 20272#0: *28 recv: fd:3 0 of 1024
2024/07/03 15:56:34 [info] 20272#0: *28 client unix: closed keepalive connection
2024/07/03 15:56:34 [debug] 20272#0: *28 close http connection: 3
2024/07/03 15:56:34 [debug] 20272#0: *28 event timer del: 3: 84629072
2024/07/03 15:56:34 [debug] 20272#0: *28 reusable connection: 0
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 00005555559B4AE0
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000000000000000
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555ADF450, unused: 8
2024/07/03 15:56:34 [debug] 20272#0: *28 free: 0000555555A6EA40, unused: 400

jeff.ohrstrom · July 5, 2024, 2:13pm

Looks like you need to tweak the shell’s origin settings.

https://osc.github.io/ood-documentation/latest/customizations.html?highlight=origin#fix-unauthorized-websocket-connection-in-shell-app

louistw · July 19, 2024, 1:35am

Thank you, that was it!

I add the line OOD_SHELL_ORIGIN_CHECK=https://<proxy-server> in /etc/ood/config/apps/shell/env and now the Shell app works!

system · January 15, 2025, 1:36am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
OOD app that uses websockets Get Help	9	869	May 26, 2022
Port configuration for OOD behind a web proxy? Get Help ondemand2	3	699	July 9, 2022
Putting an entire OOD setup behind a web proxy? Get Help	11	4272	May 26, 2022
Apache confusing user sessions? Get Help	7	324	January 2, 2024
Everything but ood server's reverse proxying working for remote desktop Get Help	6	1128	January 11, 2023

Websocket not working behind web proxy

Related topics