Code server does not work

mnakao · March 21, 2022, 9:47am

I installed the code server from GitHub/OSC to OOD. But, it does not work.

When launching the code server from OOD, I got a 404 Not Found page. This is because the code server login site redirects a broken URL with duplicated server/port in it.

The broken URL is http://DOMAIN/rnode/HOST/PORT/rnode/HOST/PORT/login

But the right URL must be http://DOMAIN/rnode/HOST/PORT/login

When I go to the right URL, the code server works. This issue may be the same as #882.

The Pull Request made in #882 seems to have been merged into the OOD master branch. But this error continues in my environment.

I use OOD v2.0.23 and tried the code server some versions (v3.3.1, v3.4.1, and v3.9.3, v3.10.2). The reason I’ve tried multiple versions is that this issue may depend on the version of the code server. But all versions got the same error.

Best,

gbyrket · March 21, 2022, 3:06pm

Hi Masahiro.

Thanks for the post. I wanted to acknowledge that we did see your post. We will need to look into this a bit deeper.

Thanks,
-gerald

holtgrewe · April 8, 2022, 8:15am

I’m also facing this issue.

Related information:

I understand that code-server expects to be served under the root directory starting with v3.

holtgrewe · April 8, 2022, 8:43am

… which is what /rnode urls are for…

Looking into the code-server logs

[2022-04-08T08:28:14.555Z] debug redirecting from /? to ./?folder=/data/gpfs-1/users/holtgrem_c

Which is problematic as this apparently causes the bad behaviour. What happens if we directly put ?folder=...?

This!

When looking into the web developer network introspection we see that the server tries to access

wss://HOST/rnode/hpc-cpu-45.cubi.bihealth.org/33778/?type=Management&reconnectionToken=ea1bcd38-cb14-4568-89ae-63e44e9e292d&reconnection=false&skipWebSocketFrames=false

When accessing this with https instead of wss then I see that it gets redirected to the double-nesting URL as previously. So… maybe the stripping of the prefix of rnode is not as complete as one would hope?

holtgrewe · April 8, 2022, 8:50am

Digging a bit deeper, I enabled LogLevel debug in Apache. This is the relevant part of the logs

[Fri Apr 08 10:44:48.576690 2022] [proxy_ajp:debug] [pid 274995:tid 140177701279488] mod_proxy_ajp.c(743): [client 172.16.128.67:34498] AH00894: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576697 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1021): [client 172.16.128.67:34498] AH01076: url: ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false proxyname: (null) proxyport: 0
[Fri Apr 08 10:44:48.576704 2022] [proxy_fcgi:debug] [pid 274995:tid 140177701279488] mod_proxy_fcgi.c(1024): [client 172.16.128.67:34498] AH01077: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576714 2022] [proxy_http:debug] [pid 274995:tid 140177701279488] mod_proxy_http.c(1970): [client 172.16.128.67:34498] AH01113: HTTP: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576722 2022] [proxy_scgi:debug] [pid 274995:tid 140177701279488] mod_proxy_scgi.c(538): [client 172.16.128.67:34498] AH00865: declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576729 2022] [:debug] [pid 274995:tid 140177701279488] mod_proxy_uwsgi.c(466): [client 172.16.128.67:34498] declining URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576738 2022] [proxy_wstunnel:debug] [pid 274995:tid 140177701279488] mod_proxy_wstunnel.c(327): [client 172.16.128.67:34498] AH02451: serving URL ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false
[Fri Apr 08 10:44:48.576747 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2339): AH00942: WS: has acquired connection for (*)
[Fri Apr 08 10:44:48.576757 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2394): [client 172.16.128.67:34498] AH00944: connecting ws://hpc-cpu-45.cubi.bihealth.org:33778/?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778
[Fri Apr 08 10:44:48.576968 2022] [proxy:debug] [pid 274995:tid 140177701279488] proxy_util.c(2620): [client 172.16.128.67:34498] AH00947: connected /?type=Management&reconnectionToken=33bf22f1-99ad-4e89-8f52-bb2d020a7a53&reconnection=false&skipWebSocketFrames=false to hpc-cpu-45.cubi.bihealth.org:33778

So it appears that this again redirects to ./ which then causes problem.

I can confirm that v3.12.0 works with login disabled but has a similar problem when enabling password. Even when going to the /login URL, it tries to redirect to the twice-nested URL.

Looking into this a bit further. The logs of the code-server say

[2022-04-08T11:02:32.341Z] debug redirecting from /? to ./login
[2022-04-08T11:03:18.773Z] debug redirecting from /? to ./login
[2022-04-08T11:03:26.496Z] debug redirecting from /? to ./login

but the corresponding apache debug logs say

[Fri Apr 08 13:03:26.500810 2022] [lua:info] [pid 277115:tid 139633463645952] [client 172.16.128.67:36872] req_accept_charset="" req_is_https="true" res_content_encoding="" req_protocol="HTTP/1.1" req_filename="proxy:http://hpc-cpu-168.cubi.bihealth.org:47283/?" req_is_websocket="false" req_server_name="hpc-portal.cubi.bihealth.org" res_content_location="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" log_time="2022-04-08T11:03:26.500679.0Z" time_proxy="4.034" local_user="holtgrem_c" remote_user="charite\\holtgrem" req_uri="/rnode/hpc-cpu-168.cubi.bihealth.org/47283/" req_origin="" res_content_disp="" log_hook="ood" req_user_agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0" res_content_language="" req_handler="proxy-server" req_port="443" time_user_map="8.11" res_content_length="58" log_id="YlAWfn0UX88KqlhUOfCmFwAAAEk" res_content_type="text/html; charset=utf-8" req_user_ip="172.16.128.67" req_cache_control="" res_location="./rnode/hpc-cpu-168.cubi.bihealth.org/47283/login" req_referer="https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions" req_content_type="" req_accept_language="en-us,en;q=0.5" req_status="302" req_accept_encoding="gzip, deflate, br" req_method="GET" req_hostname="hpc-portal.cubi.bihealth.org", referer: https://hpc-portal.cubi.bihealth.org/pun/sys/dashboard/batch_connect/sessions

Which points to that something here is rewritten that should rather not be.

I found the following in /etc/httpd/conf.d/ood-portal.conf:

  # Reverse "relative" proxy traffic to backend webserver through IP sockets:
  #
  #     https://localhost:443/rnode/HOST/PORT/index.html
  #     #=> http://HOST:PORT/index.html
  #
  <LocationMatch "^/rnode/(?<host>hpc-...-[\d]+.cubi.bihealth.org)/(?<port>\d+)(?<uri>/.*|)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^([^/]+//[^/]+)|(?=/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

By the comments I understand that ProxyPassReverse is not used for some reason but rather a custom regex is used.

edit2: I added the following line:

Header edit Location "rnode" "rnodex"

and now I’m getting redirects to

/rnode/hpc-cpu-168.cubi.bihealth.org/47283/rnodex/hpc-cpu-168.cubi.bihealth.org/47283/login

So I assume that we’re adding an unnecessary /rnode prefix

edit2: This looks similar to the following

github.com/OSC/ondemand

Ubuntu proxy tests are failing

opened 08:58PM - 29 Mar 22 UTC

closed 05:51PM - 31 Mar 22 UTC

johrstrom

Ubuntu proxy tests are failing with the error below for some unknown reason. … Failures: 1) Node and Rnode proxies rnode correctly redirects relative urls Failure/Error: expect(browser.url).to eq("#{ctr_base_url}/rnode/localhost/5000/one/one-level-down") expected: "http://localhost:8080/rnode/localhost/5000/one/one-level-down" got: "http://localhost:8080/rnode/localhost/5000/one/two/rnode/one/one-level-down" (compared using ==) # ./spec/e2e/proxy_spec.rb:53:in `block (2 levels) in <top (required)>' ┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1201735133575781/1202045846683825) by [Unito](https://www.unito.io)

So, what does the regex “^([^/]+//[^/]+)|(?=/)” mean? My interpretation is that if the path does not start with a slash / but contains a double-slash // and is followed by one or more non-slash characters then then everything up to but excluding the next slash / is matched and replaced with /rnode/.... If this is not the case but the string contains a slash / then we prepend /node....

What happens in the case of redirecting to ./login (If my assumptions above are correct)? Well, we would prepend /rnode/... and we end up with adding too much.

So what about using this here:

Header edit Location "^([^/]+//[^/]+)|(?=(?<!\.)/)" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

This makes it match an string containing a slash that is not ./… or so. At least the nested redirect does not happen BUT I get to see the login form and cannot get behidn it.

The code server logs now say

[2022-04-08T12:08:48.873Z] debug got cookie doman {"host":"hpc-cpu-168.cubi.bihealth.org"}

edit3: well that was not too helpful at all, what is now happening with the v4.2 version of code-server is it gets redirected to

https://hpc-portal.cubi.bihealth.org/rnode/hpc-cpu-219.cubi.bihealth.org/36929/?folder=/rnode/hpc-cpu-219.cubi.bihealth.org/36929/data/gpfs-1/users/holtgrem_c

phew… I have no clue any more.

nvonwolf · July 8, 2022, 11:57pm

Same issue with the latest code-server release (4.5.0). Devs over there state that everything should be using the relative path but they have been fixing constant bugs due to things not using relative paths correctly.

Not sure if this is an OOD issue or a code-server issue. Probably needs review from one of the OOD devs to determine.

travert · October 6, 2022, 2:24pm

Sorry for the troubles people have had. I was able to look at code-server 4.5 and address the issue there for the broken login. I’ve got a PR in to fix that specific version, but reading through this thread I would wonder if other versions can be fixed using the same logic.

Essentially we needed to alter the view to use a post with the password but only for certain versions of code-server. This meant we needed the view to know a bit about the context which was done using a conn_params to communicate this information between the pieces of the bc_osc_codeserver app.

The PR can be seen here:

nvonwolf · October 6, 2022, 2:51pm

@travert

I have similar setup in our code-server app (NMSU_HPC / ood_bc_codeserver · GitLab) that I have been sharing in a few other posts. It works for my site with RHEL 8, the latest OOD stable, and the official ansible ood install role. However, for others on RHEL 7, or Ubuntu, it has the same redirect issues with rnode is seen previously.

travert · October 6, 2022, 2:55pm

Thanks for the context! I have another issue on my plate currently I need to prioritize, but will circle back around to investigating this with those specific OSs to see why that is happening.

nvonwolf · October 6, 2022, 3:05pm

@holtgrewe

It looks like the apache config generated on rhel 8 with Ondemand 2.0.28-1.el8 is a bit different and solved the redirect issue for my site after having migrated from CentOS 7 to RHEL 8.

# Reverse "relative" proxy traffic to backend webserver through IP sockets:
  #
  #     https://ondemand.nmsu.edu:443/rnode/HOST/PORT/index.html
  #     #=> http://HOST:PORT/index.html
  #
  <LocationMatch "^/rnode/(?<host>discovery-(c|g)\\d+-ib0\\.cluster\\.local)/(?<port>\d+)(?<uri>/.*|)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^([^/]+//[^/]+)|(?=/)|^([\./]{1,}(?<!/))" "/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit  Set-Cookie "^([^;]+)" "$1; Path=/rnode/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

If you look closely it looks like the regex in Header edit Location has been adjusted to account for ./ locations. That or the 2 referenced lua scripts are different in RHEL 8 and somehow fixing the issue.

jeff.ohrstrom · October 10, 2022, 3:27pm

Looking through our git log, the relative path support is an update to 2.0.28 itself and not OS dependent. I know I fixed relative paths for 2.1 and we had to backport it for Ubuntu support (for some reason Debian’s apache2 package needed it).

system · April 8, 2023, 3:28pm

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Vscode code-server Get Help	14	2605	May 26, 2022
Current Status of Code Server App? General Discussion	4	937	May 26, 2022
VSCode Showcase General Discussion	18	2183	December 14, 2022
Websocket failure 1006 with VS Code Get Help	2	186	September 20, 2023
Rstudio-server Not Found Get Help ondemand2 , question	26	375	March 18, 2023

Code server does not work

Related Topics