# Failed to establish a websocket connection - USC

**URL:** <https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163>\
**Category:** Get Help\
**Created:** [October 13, 2020, 9:54pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163 "2020-10-13T21:54:02Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkhongusc](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jkhongusc/32/2098_2.png) [@jkhongusc](https://discourse.openondemand.org/u/jkhongusc)\
**Post date:** [October 13, 2020, 9:54pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/1 "2020-10-13T21:54:02Z")

</div>

New installation of Open OnDemand for a first time admin. I did a basic OOD configuration and was able to log into the OOD dashboard. When I tried to access the web shell for our login node, I get an error page. Note the url looks correct - [https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu:](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu:)  
`Failed to establish a websocket connection. Be sure you are using a browser that supports websocket connections.`

I have read the other topic on here but they didnt help. I’ve double checked configurations and logs… nothing looks wrong. Any hints on how to debug further?

Detailed information about my setup:

- Shared server, ports 80, 8080, 443 were already used. I had to configure 9443 & fronted by a load balancer.

- Use Shibboleth SSO for authentication

- Note that our log node should prompt for 2FA. This might be an issue, but I thought I would see error alluding to that. Like so:  
$ ssh jkhong@discovery.usc.edu  
Duo two-factor login for jkhong

Logs when I access [https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu:](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu:)  
**/opt/rh/httpd24/root/etc/httpd/logs/access\_log**  
128.125.253.76 - jkhong [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/ssh/discovery.usc.edu HTTP/1.1” 200 1330 “[https://carcaccount.usc.edu/pun/sys/dashboard](https://carcaccount.usc.edu/pun/sys/dashboard)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”  
128.125.253.76 - jkhong [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/stylesheets/style.css HTTP/1.1” 200 333 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”  
128.125.253.76 - jkhong [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/javascripts/ood\_shell.1.js HTTP/1.1” 200 3168 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”  
128.125.253.76 - jkhong [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/javascripts/hterm\_all\_1.85.js HTTP/1.1” 200 656388 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”  
128.125.253.76 - - [13/Oct/2020:14:43:37 -0700] “GET /favicon.ico HTTP/1.1” 404 196 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36”

**/opt/rh/httpd24/root/etc/httpd/logs/error\_log**  
[Tue Oct 13 14:43:36.690213 2020] [lua:info] [pid 31270] [client 128.125.253.76:9515] req\_protocol=“HTTP/1.1” req\_handler=“proxy-server” req\_method=“GET” req\_accept=“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,_/_;q=0.8,application/signed-exchange;v=b3;q=0.9” req\_user\_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” res\_content\_length=“1330” req\_content\_type="" res\_content\_encoding="" req\_status=“200” req\_origin="" time\_user\_map=“125.619” local\_user=“jkhong” req\_referer=“[https://carcaccount.usc.edu/pun/sys/dashboard](https://carcaccount.usc.edu/pun/sys/dashboard)” res\_content\_language="" req\_port=“443” log\_time=“2020-10-13T21:43:36.690042Z” req\_server\_name=“carcaccount.usc.edu” log\_hook=“ood” req\_accept\_charset="" req\_hostname=“carcaccount.usc.edu” res\_content\_location="" res\_content\_disp="" req\_is\_websocket=“false” remote\_user=“jkhong” res\_location="" req\_user\_ip=“128.125.253.76” req\_is\_https=“true” req\_filename=“proxy:[http://localhost/pun/sys/shell/ssh/discovery.usc.edu](http://localhost/pun/sys/shell/ssh/discovery.usc.edu)” req\_uri="/pun/sys/shell/ssh/discovery.usc.edu" time\_proxy=“4.925” res\_content\_type=“text/html; charset=utf-8” req\_accept\_language=“en-us,en;q=0.9,lb;q=0.8” req\_cache\_control=“no-cache” req\_accept\_encoding=“gzip, deflate, br”, referer: [https://carcaccount.usc.edu/pun/sys/dashboard](https://carcaccount.usc.edu/pun/sys/dashboard)  
[Tue Oct 13 14:43:36.888310 2020] [lua:info] [pid 31270] [client 128.125.253.76:9515] req\_protocol=“HTTP/1.1” req\_handler=“proxy-server” req\_method=“GET” req\_accept=“text/css,_/_;q=0.1” req\_user\_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” res\_content\_length=“333” req\_content\_type="" res\_content\_encoding="" req\_status=“200” req\_origin="" time\_user\_map=“127.045” local\_user=“jkhong” req\_referer=“[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” res\_content\_language="" req\_port=“443” log\_time=“2020-10-13T21:43:36.888164Z” req\_server\_name=“carcaccount.usc.edu” log\_hook=“ood” req\_accept\_charset="" req\_hostname=“carcaccount.usc.edu” res\_content\_location="" res\_content\_disp="" req\_is\_websocket=“false” remote\_user=“jkhong” res\_location="" req\_user\_ip=“128.125.253.76” req\_is\_https=“true” req\_filename=“proxy:[http://localhost/pun/sys/shell/stylesheets/style.css](http://localhost/pun/sys/shell/stylesheets/style.css)” req\_uri="/pun/sys/shell/stylesheets/style.css" time\_proxy=“1.233” res\_content\_type=“text/css” req\_accept\_language=“en-us,en;q=0.9,lb;q=0.8” req\_cache\_control=“no-cache” req\_accept\_encoding=“gzip, deflate, br”, referer: [https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)  
[Tue Oct 13 14:43:36.902130 2020] [lua:info] [pid 31377] [client 128.125.253.76:43093] req\_protocol=“HTTP/1.1” req\_handler=“proxy-server” req\_method=“GET” req\_accept="_/_" req\_user\_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” res\_content\_length=“3168” req\_content\_type="" res\_content\_encoding="" req\_status=“200” req\_origin="" time\_user\_map=“127.557” local\_user=“jkhong” req\_referer=“[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” res\_content\_language="" req\_port=“443” log\_time=“2020-10-13T21:43:36.901986Z” req\_server\_name=“carcaccount.usc.edu” log\_hook=“ood” req\_accept\_charset="" req\_hostname=“carcaccount.usc.edu” res\_content\_location="" res\_content\_disp="" req\_is\_websocket=“false” remote\_user=“jkhong” res\_location="" req\_user\_ip=“128.125.253.76” req\_is\_https=“true” req\_filename=“proxy:[http://localhost/pun/sys/shell/javascripts/ood\_shell.1.js](http://localhost/pun/sys/shell/javascripts/ood_shell.1.js)” req\_uri="/pun/sys/shell/javascripts/ood\_shell.1.js" time\_proxy=“1.253” res\_content\_type=“application/javascript” req\_accept\_language=“en-us,en;q=0.9,lb;q=0.8” req\_cache\_control=“no-cache” req\_accept\_encoding=“gzip, deflate, br”, referer: [https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)  
[Tue Oct 13 14:43:36.968652 2020] [lua:info] [pid 31226] [client 128.125.253.76:29334] req\_protocol=“HTTP/1.1” req\_handler=“proxy-server” req\_method=“GET” req\_accept="_/_" req\_user\_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” res\_content\_length=“656388” req\_content\_type="" res\_content\_encoding="" req\_status=“200” req\_origin="" time\_user\_map=“128.229” local\_user=“jkhong” req\_referer=“[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” res\_content\_language="" req\_port=“443” log\_time=“2020-10-13T21:43:36.968492Z” req\_server\_name=“carcaccount.usc.edu” log\_hook=“ood” req\_accept\_charset="" req\_hostname=“carcaccount.usc.edu” res\_content\_location="" res\_content\_disp="" req\_is\_websocket=“false” remote\_user=“jkhong” res\_location="" req\_user\_ip=“128.125.253.76” req\_is\_https=“true” req\_filename=“proxy:[http://localhost/pun/sys/shell/javascripts/hterm\_all\_1.85.js](http://localhost/pun/sys/shell/javascripts/hterm_all_1.85.js)” req\_uri="/pun/sys/shell/javascripts/hterm\_all\_1.85.js" time\_proxy=“68.766” res\_content\_type=“application/javascript” req\_accept\_language=“en-us,en;q=0.9,lb;q=0.8” req\_cache\_control=“no-cache” req\_accept\_encoding=“gzip, deflate, br”, referer: [https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)

**/var/log/ondemand-nginx/jkhong/access.log**  
unix: - - [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/ssh/discovery.usc.edu HTTP/1.1” 200 1330 “[https://carcaccount.usc.edu/pun/sys/dashboard](https://carcaccount.usc.edu/pun/sys/dashboard)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” “10.2.203.102, 128.125.253.76”  
unix: - - [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/stylesheets/style.css HTTP/1.1” 200 333 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” “10.2.203.102, 128.125.253.76”  
unix: - - [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/javascripts/ood\_shell.1.js HTTP/1.1” 200 3168 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” “10.2.203.102, 128.125.253.76”  
unix: - - [13/Oct/2020:14:43:36 -0700] “GET /pun/sys/shell/javascripts/hterm\_all\_1.85.js HTTP/1.1” 200 656388 “[https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu](https://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu)” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36” “10.2.203.102, 128.125.253.76”

**/var/log/ondemand-nginx/jkhong/error.log**  
App 28598 output: Listening on 3000

Browser web developer console error:  
`ood_shell.1.js:12 WebSocket connection to 'wss://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu?csrf=REKBEv2L-N8xwpyH7bOQhktqwb52YtDYK6yc' failed: Connection closed before receiving a handshake response`

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 15, 2020, 1:49pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/2 "2020-10-15T13:49:20Z")

</div>

Hi yea, that message isn’t totally helpful. I’d suggest making sure `discovery.usc.edu` is a part of the allowlist. [You can see docs for setting the allowlist here](https://osc.github.io/ood-documentation/master/customization.html?highlight=allowlist#set-ssh-allowlist).

More than that, I think in the developer tools - in the network tab seeing what’s there may provide some insight. What the response code and the response headers are for that upgrade request.

In this case, i get 401 unauthorized and there’s an `X-OOD-Failure-Reason` header saying the host is not in the allowlist.  
 ![image](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/97e917bc8cef6b504e417112bb2b27b3134c0952.png)

---

<div class="post-metadata">

**Author:** ![jkhongusc](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jkhongusc/32/2098_2.png) [@jkhongusc](https://discourse.openondemand.org/u/jkhongusc)\
**Post date:** [October 15, 2020, 5:07pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/3 "2020-10-15T17:07:46Z")

</div>

Jeff -  
Thanks for your response. I did a lot of research into this issue. I did set OOD\_SSHHOST\_ALLOWLIST just in case. I printed it out in app.js just to verify, from errors.log:  
`App 28680 output: host_allowlist: discovery.usc.edu:discovery1.usc.edu:discovery2.usc.edu:*.usc.edu`

WRT to the request - wss://carcaccount.usc.edu/pun/sys/shell/ssh/discovery.usc.edu?csrf= - in web developer the request is sent, but no response is received (connection is closed).

 ![ood_shell_connectionclosed](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/0dcd37fa37b3120f81752a32abd6d6ef94be8773.png)

At this point I suspect the shell code is not working on my server. I found the nodejs code - /var/www/ood/apps/sys/shell/apps.js - and added some log messages in there. The logs are sent to - /var/log/ondemand-nginx/jkhong/error.log - The only shell message I see is the “Listening on 3000”. I expect to see “Connection established” but never do. I also never see the upgrade event triggered - server.on(‘upgrade’,… - which should initiate the connection. This is as far as I have gotten in my debugging.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 15, 2020, 5:26pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/4 "2020-10-15T17:26:23Z")

</div>

OK gotcha, sorry I jumped right to something before I read your initial message fully.

You’re behind a load balancer. You may need to setup rules for `wss` and `ws` protocol. This is my next guess as to what is going on, your LB may be disconnecting the request if it’s working on L7 protocols. Your origin logs don’t show any `request_is_websocket="true"` entries, and at some point it should.

---

<div class="post-metadata">

**Author:** ![jkhongusc](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jkhongusc/32/2098_2.png) [@jkhongusc](https://discourse.openondemand.org/u/jkhongusc)\
**Post date:** [October 16, 2020, 1:10am UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/5 "2020-10-16T01:10:41Z")

</div>

It was our load balancer (Citrix Netscaler). It was not configured to allow websockets (ws and wss).

This is what I did to debug. I looked at developer tools request headers when it made the wss request. I then simulated the request using curl on the command-line. Here is an example curl wss request with it rejecting the wss (openssl errno 104):

```
 curl -vk -i -N -H "Host: carcaccount.usc.edu" -H "Origin: https://carcaccount.usc.edu" -H "Connection: Upgrade" -H "Upgrade: websocket" 'https://carcaccount.usc.edu/pun/sys/shell/discovery.usc.edu?csrf=Y6GDrzuL-dv6WrczQXBsax7mwOCasZ1Yp8e8'
> GET /pun/sys/shell/ssh/discovery.usc.edu?csrf=Y6GDrzuL-dv6WrczQXBsax7mwOCasZ1Yp8e8 HTTP/1.1
> Host: carcaccount.usc.edu
> User-Agent: curl/7.66.0
> Accept: */*
> Connection: Upgrade
> Upgrade: websocket
> Origin: https://carcaccount.usc.edu
>
* STATE: DO => DO_DONE handle 0x800083108; line 1756 (connection #0)
* STATE: DO_DONE => PERFORM handle 0x800083108; line 1877 (connection #0)
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* Marked for [closure]: Transfer returned error
* multi_done
* Closing connection 0
* The cache now contains 0 members
* Expire cleared (transfer 0x800083108)
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

```

I noticed the wss request never made it to apache httpd. At that point I did suspect the LB configuration. I found out that I had to make special configuration changes on our Citrix Netscaler LB to support ws and wss protocols. After that the wss curl request worked! And of course OOD shell worked too!

@jeff.ohrstrom - Thank you for your assistance and guess as the LB as the problem!

---

<div class="post-metadata">

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\
**Post date:** [May 26, 2022, 3:43pm UTC](https://discourse.openondemand.org/t/failed-to-establish-a-websocket-connection-usc/1163/6 "2022-05-26T15:43:03Z")

</div>