RStudio Server 2023.12 Temporary server error issue

Get Help
1

Hi everyone,

I have followed the instructions to install the RStudio Server app in GitHub - uabrc/bc_uab_rstudio_server: RStudio Server app for Open OnDemand and got to the point where I can see the Launch button. The rserver version 2023.12.0 and is run as an installed module in our system. I ssh into the compute node and see that the rserver process is running and listening to an open port.

However, when I click on the Launch button, the Sign In page shows up with Temporary server error. I searched over the OOD Discourse and found that there might be issue with the csrf_token generated and in use for authentication such as this thread: RStudio Server Issues - #9 by brad.traver.

Apparently, I miss something important in my configuration in template/before.script.erb, submit.yaml.erb and view.html.erb as below, but cannot figure it out:

template/before.script.erb:

export ONDEMAND_DIR="/tmp/${USER}/ood-rstudio-${SLURM_JOB_ID}"
mkdir -p "${ONDEMAND_DIR}"

# Create a flag to indicate script is starting
touch "${ONDEMAND_DIR}/starting"

# Set system library path
export LD_LIBRARY_PATH="/lib64:${LD_LIBRARY_PATH}"

# Define a password and export it for RStudio authentication
export password="$(create_passwd 16)"
export PASSWORD="${password}"

# Define CSRF_TOKEN and export it for auth
<%-
  require 'securerandom'
  csrftoken=SecureRandom.uuid
-%>
export csrf_token="<%= csrftoken %>"

submit.yml.erb:

---
cluster: "midway3"
script:
  native:
    <% if !custom_account.blank? %>
    - "-A"
    - "<%= custom_account %>"
    <% end %>
  content: "template/script.sh.erb"
batch_connect:
  template: "basic"
  conn_params:
    - csrf_token
form: "form.yml.erb"
attributes:
  job_environment:
    TERM: "xterm-256color"
    USER: "<%= ENV['USER'] %>"

view.html.erb:

<%-
  base_url  = "/rnode/#{host}/#{port}"
  full_url = "#{base_url}"
%>
<head>
  <meta charset="UTF-8">
</head>
<pre>
 username: "<%= ENV["USER"] %>"
 password: "<%= password %>"
 token: "<%= csrf_token %>"
</pre>
<script>
  document.cookie = "csrf-token=<%= csrf_token %>; path=/rnode/<%= host %>/<%= port %>; secure";
</script>
<form action="<%= full_url %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>">
  <input type="hidden" name="password" value="<%= password %>">
  <input type="hidden" name="staySignedIn" value="1">
  <input type="hidden" name="appUri" value="">
  <input type="hidden" name="csrf-token" value="<%= csrf_token %>" >
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-registered"></i> Connect to RStudio Server
  </button>
</form>

I added the <pre> ... </pre> section to print out the info used for the http request for debugging purposes to make sure they are consistent with that in the output,log file.

Could you please help me resolve the issue with the Temporary server error issue, or point me to where to look at? The rserver.log file doesn’t contain anything useful, other than the exec command that launched the rserver.

Looking at the RStudio Job in the OOD documentation

it seems to fail at steps 15-17 in the diagram.

Thanks in advance
-Trung

2

When I looked at the HTML source of the RStudio Sign In page, I found that there is an input section with rs-csrf-token that shows a different value than csrf_token from before.script.sh and used in view.html.erb (8e768d6c-4923-4e33-9d47-047c958b49df in this session)

   <input
         type="hidden"
         name="rs-csrf-token"
         value="301f51cf-03bd-4434-b2da-c707906b0f6c"
/>

I also modified the view.html.erb script to use rs-csrf-token instead of csrf-token:

<input type="hidden" name="rs-csrf-token" value="<%= csrf_token %>">

but the HTML source still shows another value of rs-csrf-token.

3

hi and welcome!

Can you try this script instead? I’m not 100% here but I think maybe you could use an expiration date and maybe SameSite=strict? My feeling is they can’t hurt anyhow.

<script type="text/javascript">
(function () {
  let date = new Date();
  date.setTime(date.getTime() + (7*24*60*60*1000));
  let expires = "expires=" + date.toUTCString();
  let cookiePath = "path=/rnode/" + "<%= host.to_s %>" + "/" + "<%= port.to_s %>/";
  /**
    rstuido wants a cookie called csrf-token - but that's going to change in 2020!
  */
  let cookie = `csrf-token=<%= csrf_token %>;${expires};${cookiePath};SameSite=strict;secure`;
  document.cookie = cookie;
})();
</script>

One thing you can check (in google chrome) is if these 2 things actually match. In google Chrome you can open your developer tools in a new tab. When you click on ‘connect to Rstudio’ you can verify in the developer tab if the csrf-token in the form is the same as the csrf-token in the cookie. This is what Rstudio is trying to verify, that these 2 things match.

4

Thank you @jeff.ohrstrom, I used the script you shared, and looked into the developer tab as you suggested, but the 2 csrf tokens still do not match. May I ask which version of the RStudio server that is working with this script on your end?

5

The issue appears to be completely on the OOD side, so I’m not sure that Rstudio version matters - especially if you can confirm that they’re different on your side/in your request.

I wonder if you clear you cache or use a private window if there’s any difference? Like you can’t update the cookie that you initially created sometime in the past.

6

I switched to another browser (Safari) with a fresh cache and the issue persists. The reason that made me think of RStudio versions is that there might be issues with recent versions of RStudio that makes the csrf_token in the cookie and the csrf_token used in auth-do-sign-in different ones, something related to Automatic login failed when missing csrf-token item in cookie with www-enable-origin-check=0 · Issue #12405 · rstudio/rstudio · GitHub starting with v2022.

I tried to use --auth-none 1 as suggested in the conversation in that github issue, and see that it passed the sign in stage. I am not sure if skipping authentication to avoid the unmatched csrf token issue can be considered a solution though.

7

No it’s not a solution really - without authentication your users are vulnerable.

OK - I think we’re missing something obvious. The first thing I’d ask is if we can actually confirm there’s a CSRF token issue or if we just jumped to that conclusion.

Can we confirm that by checking the logs?

8

Hi Jeff,

I can confirm that the CSRT token generated by before.script.erb (which is then printed to the connection.yaml) is the same as the one goes into the script view.html.erb. I printed its value by adding

<pre>
csrf_token: <%= csrf_token %>
</pre>

before the script you shared in view.html.erb.

When I click the Launch button, the token is printed out on the card (which is the same as in connection.yaml, as expected). The Sign In page shows up with Temporary server error. The developer tab shows the rs-csrf-token field with a value different from the one shown on the card. Note that it shows rs-csrf-token, instead of csrf-token, as I had showed earlier.

Is there anything important I am missing here, to confirm that it is related to the CSFR token(s)?

I think the issue might be related to how RStudio Server generates a token independently of how we put it in the file document.cookie.

I am wondering how vulnerable the user would be with --auth-none 1 when a single RStudio Server process is launched with --server-user inside the job allocated to the same user. We don’t set --auth-none 1 under /etc/rstudio/rstudio.conf. Could you please elaborate?

Thanks,
-Trung

9

Thanks Jeff for walking through the scripts and for suggesting me remove the seemingly innocent block in view.html.erb

<head>
<meta charset="UTF-8"> 
</head>

which I added in an attempt to fix the rendered font in the sign in page. Removing this block resolved an issue with a misplaced “&” somewhere during the first auth-sign-in page, which then triggered the whole temporary server error. So, --auth-none 1 works now.

Thanks again!