Code server does not work

I installed the code server from GitHub/OSC to OOD. But, it does not work.

When launching the code server from OOD, I got a 404 Not Found page. This is because the code server login site redirects a broken URL with duplicated server/port in it.

The broken URL is http://DOMAIN/rnode/HOST/PORT/rnode/HOST/PORT/login

But the right URL must be http://DOMAIN/rnode/HOST/PORT/login

When I go to the right URL, the code server works. This issue may be the same as #882.

The Pull Request made in #882 seems to have been merged into the OOD master branch. But this error continues in my environment.

I use OOD v2.0.23 and tried the code server some versions (v3.3.1, v3.4.1, and v3.9.3, v3.10.2). The reason I’ve tried multiple versions is that this issue may depend on the version of the code server. But all versions got the same error.

Best,

Hi Masahiro.

Thanks for the post. I wanted to acknowledge that we did see your post. We will need to look into this a bit deeper.

Thanks,
-gerald

I’m also facing this issue.

Related information:

I understand that code-server expects to be served under the root directory starting with v3.

… which is what /rnode urls are for…

Looking into the code-server logs

[2022-04-08T08:28:14.555Z] debug redirecting from /? to ./?folder=/data/gpfs-1/users/holtgrem_c

Which is problematic as this apparently causes the bad behaviour. What happens if we directly put ?folder=...?

This!

When looking into the web developer network introspection we see that the server tries to access

  • wss://HOST/rnode/hpc-cpu-45.cubi.bihealth.org/33778/?type=Management&reconnectionToken=ea1bcd38-cb14-4568-89ae-63e44e9e292d&reconnection=false&skipWebSocketFrames=false

When accessing this with https instead of wss then I see that it gets redirected to the double-nesting URL as previously. So… maybe the stripping of the prefix of rnode is not as complete as one would hope?

Digging a bit deeper, I enabled LogLevel debug in Apache. This is the relevant part of the logs

[Fri Apr 08 10:44:48.576690 2022] [proxy_ajp:debug] [pid 274995:tid 140177701279488] mod_proxy_ajp.c(743): [client 172.16.128.67:34498] AH00894: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576697 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1021): [client 172.16.128.67:34498] AH01076: url: ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false proxyname: (null) proxyport: 0
[Fri Apr 08 10:44:48.576704 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1024): [client 172.16.128.67:34498] AH01077: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576714 2022] [proxy_http:debug] [pid 274995:tid 140177701279488] mod_proxy_http.c(1970): [client 172.16.128.67:34498] AH01113: HTTP: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576722 2022] [proxy_scgi:debug] [pid 274995:tid 140177701279488] mod_proxy_scgi.c(538): [client 172.16.128.67:34498] AH00865: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576729 2022] [:debug] [pid 274995:tid 140177701279488] mod_proxy_uwsgi.c(466): [client 172.16.128.67:34498] declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576738 2022] [proxy_wstunnel:debug] [pid 274995:tid 140177701279488] mod_proxy_wstunnel.c(327): [client 172.16.128.67:34498] AH02451: serving URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576747 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2339): AH00942: WS: has acquired connection for (*)
[Fri Apr 08 10:44:48.576757 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2394): [client 172.16.128.67:34498] AH00944: connecting ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778
[Fri Apr 08 10:44:48.576968 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2620): [client 172.16.128.67:34498] AH00947: connected /?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778

So it appears that this again redirects to ./ which then causes problem.

I can confirm that v3.12.0 works with login disabled but has a similar problem when enabling password. Even when going to the /login URL, it tries to redirect to the twice-nested URL.

Looking into this a bit further. The logs of the code-server say

[2022-04-08T11:02:32.341Z] debug redirecting from /? to ./login
[2022-04-08T11:03:18.773Z] debug redirecting from /? to ./login
[2022-04-08T11:03:26.496Z] debug redirecting from /? to ./login

but the corresponding apache debug logs say

[Fri Apr 08 13:03:26.500810 2022] [lua:info] [pid 277115:tid 139633463645952] [client 172.16.128.67:36872] req_accept_charset="" req_is_https="true" res_content_encoding="" req_protocol="HTTP/1.1" req_filename="proxy:http://hpc-cpu-168.cubi.bihealth.org:47283/?" req_is_websocket="false" req_server_name="hpc-portal.cubi.bihealth.org" res_content_location="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" log_time="2022-04-08T11:03:26.500679.0Z" time_proxy="4.034" local_user="holtgrem_c" remote_user="charite\\holtgrem" req_uri="/rnode/hpc-cpu-168.cubi.bihealth.org/47283/" req_origin="" res_content_disp="" log_hook="ood" req_user_agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0" res_content_language="" req_handler="proxy-server" req_port="443" time_user_map="8.11" res_content_length="58" log_id="YlAWfn0UX88KqlhUOfCmFwAAAEk" res_content_type="text/html; charset=utf-8" req_user_ip="172.16.128.67" req_cache_control="" res_location="./rnode/hpc-cpu-168.cubi.bihealth.org/47283/login" req_referer="https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions" req_content_type="" req_accept_language="en-us,en;q=0.5" req_status="302" req_accept_encoding="gzip, deflate, br" req_method="GET" req_hostname="hpc-portal.cubi.bihealth.org", referer: https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions

Which points to that something here is rewritten that should rather not be.

I found the following in /etc/httpd/conf.d/ood-portal.conf:

  # Reverse "relative" proxy traffic to backend webserver through IP sockets:
  #
  #     https://localhost:443/rnode/HOST/PORT/index.html
  #     #=> http://HOST:PORT/index.html
  #
  <LocationMatch "^/rnode/(?<host>hpc-...-[\d]+.cubi.bihealth.org)/(?<port>\d+)(?<uri>/.*|)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^([^/]+//[^/]+)|(?=/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

By the comments I understand that ProxyPassReverse is not used for some reason but rather a custom regex is used.

edit2: I added the following line:

Header edit Location "rnode" "rnodex"

and now I’m getting redirects to

/rnode/hpc-cpu-168.cubi.bihealth.org/47283/rnodex/hpc-cpu-168.cubi.bihealth.org/47283/login

So I assume that we’re adding an unnecessary /rnode prefix

edit2: This looks similar to the following

So, what does the regex “^([^/]+//[^/]+)|(?=/)” mean? My interpretation is that if the path does not start with a slash / but contains a double-slash // and is followed by one or more non-slash characters then then everything up to but excluding the next slash / is matched and replaced with /rnode/.... If this is not the case but the string contains a slash / then we prepend /node....

What happens in the case of redirecting to ./login (If my assumptions above are correct)? Well, we would prepend /rnode/... and we end up with adding too much.

So what about using this here:

Header edit Location "^([^/]+//[^/]+)|(?=(?<!\.)/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

This makes it match an string containing a slash that is not ./… or so. At least the nested redirect does not happen BUT I get to see the login form and cannot get behidn it.

The code server logs now say

[2022-04-08T12:08:48.873Z] debug got cookie doman {"host":"hpc-cpu-168.cubi.bihealth.org"}

edit3: well that was not too helpful at all, what is now happening with the v4.2 version of code-server is it gets redirected to

https://hpc-portal.cubi.bihealth.org/rnode/hpc-cpu-219.cubi.bihealth.org/36929/?folder=/rnode/hpc-cpu-219.cubi.bihealth.org/36929/data/gpfs-1/users/holtgrem_c

phew… I have no clue any more.