Code server does not work

I installed the code server from GitHub/OSC to OOD. But, it does not work.

When launching the code server from OOD, I got a 404 Not Found page. This is because the code server login site redirects a broken URL with duplicated server/port in it.

The broken URL is http://DOMAIN/rnode/HOST/PORT/rnode/HOST/PORT/login

But the right URL must be http://DOMAIN/rnode/HOST/PORT/login

When I go to the right URL, the code server works. This issue may be the same as #882.

The Pull Request made in #882 seems to have been merged into the OOD master branch. But this error continues in my environment.

I use OOD v2.0.23 and tried the code server some versions (v3.3.1, v3.4.1, and v3.9.3, v3.10.2). The reason I’ve tried multiple versions is that this issue may depend on the version of the code server. But all versions got the same error.

Best,

Hi Masahiro.

Thanks for the post. I wanted to acknowledge that we did see your post. We will need to look into this a bit deeper.

Thanks,
-gerald

I’m also facing this issue.

Related information:

I understand that code-server expects to be served under the root directory starting with v3.

… which is what /rnode urls are for…

Looking into the code-server logs

[2022-04-08T08:28:14.555Z] debug redirecting from /? to ./?folder=/data/gpfs-1/users/holtgrem_c

Which is problematic as this apparently causes the bad behaviour. What happens if we directly put ?folder=...?

This!

When looking into the web developer network introspection we see that the server tries to access

  • wss://HOST/rnode/hpc-cpu-45.cubi.bihealth.org/33778/?type=Management&reconnectionToken=ea1bcd38-cb14-4568-89ae-63e44e9e292d&reconnection=false&skipWebSocketFrames=false

When accessing this with https instead of wss then I see that it gets redirected to the double-nesting URL as previously. So… maybe the stripping of the prefix of rnode is not as complete as one would hope?

Digging a bit deeper, I enabled LogLevel debug in Apache. This is the relevant part of the logs

[Fri Apr 08 10:44:48.576690 2022] [proxy_ajp:debug] [pid 274995:tid 140177701279488] mod_proxy_ajp.c(743): [client 172.16.128.67:34498] AH00894: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576697 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1021): [client 172.16.128.67:34498] AH01076: url: ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false proxyname: (null) proxyport: 0
[Fri Apr 08 10:44:48.576704 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1024): [client 172.16.128.67:34498] AH01077: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576714 2022] [proxy_http:debug] [pid 274995:tid 140177701279488] mod_proxy_http.c(1970): [client 172.16.128.67:34498] AH01113: HTTP: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576722 2022] [proxy_scgi:debug] [pid 274995:tid 140177701279488] mod_proxy_scgi.c(538): [client 172.16.128.67:34498] AH00865: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576729 2022] [:debug] [pid 274995:tid 140177701279488] mod_proxy_uwsgi.c(466): [client 172.16.128.67:34498] declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576738 2022] [proxy_wstunnel:debug] [pid 274995:tid 140177701279488] mod_proxy_wstunnel.c(327): [client 172.16.128.67:34498] AH02451: serving URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576747 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2339): AH00942: WS: has acquired connection for (*)
[Fri Apr 08 10:44:48.576757 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2394): [client 172.16.128.67:34498] AH00944: connecting ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778
[Fri Apr 08 10:44:48.576968 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2620): [client 172.16.128.67:34498] AH00947: connected /?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778

So it appears that this again redirects to ./ which then causes problem.

I can confirm that v3.12.0 works with login disabled but has a similar problem when enabling password. Even when going to the /login URL, it tries to redirect to the twice-nested URL.

Looking into this a bit further. The logs of the code-server say

[2022-04-08T11:02:32.341Z] debug redirecting from /? to ./login
[2022-04-08T11:03:18.773Z] debug redirecting from /? to ./login
[2022-04-08T11:03:26.496Z] debug redirecting from /? to ./login

but the corresponding apache debug logs say

[Fri Apr 08 13:03:26.500810 2022] [lua:info] [pid 277115:tid 139633463645952] [client 172.16.128.67:36872] req_accept_charset="" req_is_https="true" res_content_encoding="" req_protocol="HTTP/1.1" req_filename="proxy:http://hpc-cpu-168.cubi.bihealth.org:47283/?" req_is_websocket="false" req_server_name="hpc-portal.cubi.bihealth.org" res_content_location="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" log_time="2022-04-08T11:03:26.500679.0Z" time_proxy="4.034" local_user="holtgrem_c" remote_user="charite\\holtgrem" req_uri="/rnode/hpc-cpu-168.cubi.bihealth.org/47283/" req_origin="" res_content_disp="" log_hook="ood" req_user_agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0" res_content_language="" req_handler="proxy-server" req_port="443" time_user_map="8.11" res_content_length="58" log_id="YlAWfn0UX88KqlhUOfCmFwAAAEk" res_content_type="text/html; charset=utf-8" req_user_ip="172.16.128.67" req_cache_control="" res_location="./rnode/hpc-cpu-168.cubi.bihealth.org/47283/login" req_referer="https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions" req_content_type="" req_accept_language="en-us,en;q=0.5" req_status="302" req_accept_encoding="gzip, deflate, br" req_method="GET" req_hostname="hpc-portal.cubi.bihealth.org", referer: https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions

Which points to that something here is rewritten that should rather not be.

I found the following in /etc/httpd/conf.d/ood-portal.conf:

  # Reverse "relative" proxy traffic to backend webserver through IP sockets:
  #
  #     https://localhost:443/rnode/HOST/PORT/index.html
  #     #=> http://HOST:PORT/index.html
  #
  <LocationMatch "^/rnode/(?<host>hpc-...-[\d]+.cubi.bihealth.org)/(?<port>\d+)(?<uri>/.*|)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^([^/]+//[^/]+)|(?=/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

By the comments I understand that ProxyPassReverse is not used for some reason but rather a custom regex is used.

edit2: I added the following line:

Header edit Location "rnode" "rnodex"

and now I’m getting redirects to

/rnode/hpc-cpu-168.cubi.bihealth.org/47283/rnodex/hpc-cpu-168.cubi.bihealth.org/47283/login

So I assume that we’re adding an unnecessary /rnode prefix

edit2: This looks similar to the following

So, what does the regex “^([^/]+//[^/]+)|(?=/)” mean? My interpretation is that if the path does not start with a slash / but contains a double-slash // and is followed by one or more non-slash characters then then everything up to but excluding the next slash / is matched and replaced with /rnode/.... If this is not the case but the string contains a slash / then we prepend /node....

What happens in the case of redirecting to ./login (If my assumptions above are correct)? Well, we would prepend /rnode/... and we end up with adding too much.

So what about using this here:

Header edit Location "^([^/]+//[^/]+)|(?=(?<!\.)/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

This makes it match an string containing a slash that is not ./… or so. At least the nested redirect does not happen BUT I get to see the login form and cannot get behidn it.

The code server logs now say

[2022-04-08T12:08:48.873Z] debug got cookie doman {"host":"hpc-cpu-168.cubi.bihealth.org"}

edit3: well that was not too helpful at all, what is now happening with the v4.2 version of code-server is it gets redirected to

https://hpc-portal.cubi.bihealth.org/rnode/hpc-cpu-219.cubi.bihealth.org/36929/?folder=/rnode/hpc-cpu-219.cubi.bihealth.org/36929/data/gpfs-1/users/holtgrem_c

phew… I have no clue any more.

1 Like

Same issue with the latest code-server release (4.5.0). Devs over there state that everything should be using the relative path but they have been fixing constant bugs due to things not using relative paths correctly.

Not sure if this is an OOD issue or a code-server issue. Probably needs review from one of the OOD devs to determine.

Sorry for the troubles people have had. I was able to look at code-server 4.5 and address the issue there for the broken login. I’ve got a PR in to fix that specific version, but reading through this thread I would wonder if other versions can be fixed using the same logic.

Essentially we needed to alter the view to use a post with the password but only for certain versions of code-server. This meant we needed the view to know a bit about the context which was done using a conn_params to communicate this information between the pieces of the bc_osc_codeserver app.

The PR can be seen here:

@travert

I have similar setup in our code-server app (NMSU_HPC / ood_bc_codeserver · GitLab) that I have been sharing in a few other posts. It works for my site with RHEL 8, the latest OOD stable, and the official ansible ood install role. However, for others on RHEL 7, or Ubuntu, it has the same redirect issues with rnode is seen previously.

Thanks for the context! I have another issue on my plate currently I need to prioritize, but will circle back around to investigating this with those specific OSs to see why that is happening.

@holtgrewe

It looks like the apache config generated on rhel 8 with Ondemand 2.0.28-1.el8 is a bit different and solved the redirect issue for my site after having migrated from CentOS 7 to RHEL 8.

# Reverse "relative" proxy traffic to backend webserver through IP sockets:
  #
  #     https://ondemand.nmsu.edu:443/rnode/HOST/PORT/index.html
  #     #=> http://HOST:PORT/index.html
  #
  <LocationMatch "^/rnode/(?<host>discovery-(c|g)\\d+-ib0\\.cluster\\.local)/(?<port>\d+)(?<uri>/.*|)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^([^/]+//[^/]+)|(?=/)|^([\./]{1,}(?<!/))" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

If you look closely it looks like the regex in Header edit Location has been adjusted to account for ./ locations. That or the 2 referenced lua scripts are different in RHEL 8 and somehow fixing the issue.

Looking through our git log, the relative path support is an update to 2.0.28 itself and not OS dependent. I know I fixed relative paths for 2.1 and we had to backport it for Ubuntu support (for some reason Debian’s apache2 package needed it).