RStudio when launched without Singularity is having strange troubles with authentication

I can’t really reproduce this but I agree it makes more sense to have it in the “connection.yml”.
If you want a real hack you can overwrite the “create_yml” function :smiley:
In the after.sh.erb you can add something like:

<%-
  require 'securerandom'
  csrftoken=SecureRandom.uuid
-%>

# Generate a connection yaml file with given parameters
create_yml () {
  echo "Generating connection YAML file..."
  (
    umask 077
    echo -e "host: $host\nport: $port\npassword: $password\ncsrftoken: <%= csrftoken %>" > "$PWD/connection.yml"
  )
}

Just tested, it seems to work :wink:

In the meantime, just curious, do you or @rena can help me reproducing the error?

If you need to pass something back to the view (like the csrftoken) you can use conn_params for this.

https://osc.github.io/ood-documentation/latest/reference/files/submit-yml/basic-bc-options.html

# submit.yml
conn_params:
  -  csrf_token
# before.sh.erb
# this may actualy be case sensitive, so maybe csrf_token would work
export CSRF_TOKEN="<$= SecureRandom.uuid >"

Then you can use it directly in your view.html.erb like you use host/port and so on.

  document.cookie = "csrf-token=<%= csrf_token %>;

Once you export the variable in the before script, it should get written automatically to connection.yml. It may be problematic to write or rewrite that file, so it’s probably safer to use this mechanism conn_params.

@dugan and @fenz both of your solutions work well. Thank you!

1 Like

Thank you all so much for the support? Am I right in summation of the fixes?

1.3 needs --server-data-dir and --secure-cookie-key-file (both should be in $TMPDIR). And you need to update the comparison in bin/auth. 1.3 compatible is [[ $# -ne 1 ]]

For 1.4 you need to populate a CSRF token. How does RStudio recognize it, through the TOKEN environment variable?

@jeff.ohrstrom csrf_token did end up being case sensitive. With the following in before.sh.erb, your conn_params solution works beautifully. Thank you!

# before.sh.erb
<%-
  require 'securerandom'
  csrftoken=SecureRandom.uuid
-%>
export csrf_token="<%= csrftoken %>"

Sorry that’s not a question, thanks so much for the support!

How does Rstudio recognize it? I mean you’re passing it but how does Rstudio validate it, what does it validate it against?

What is needed for running RStudio using Singularity is mentioned here: Rocker Project by Rocker team (there’s a Slurm section with all the path to mount).
We create those folder in the “before” and remove them in the “clean” but not sure if you want to follow the same approach.
Regarding the 1.4, as we discussed with @dugan , RStudio just checks that the token you use for the POST request is the same you have in the cookie.
In theory the cookie is generated by the server when accessing the “login page” but, as said, that’s not needed at the moment (not sure if RStudio team will change this in the future).
That’s the reason why something like this works at the moment:

 <%-
 csrftoken="MYTOKEN"
-%>
<script defer src="https://use.fontawesome.com/releases/v5.8.2/js/brands.js" integrity="sha384-GtvEzzhN52RvAD7CnSR7TcPw555abR8NK28tAqa/GgIDk59o0TsaK6FHglLstzCf" crossorigin="anonymous"></script>
<script defer src="https://use.fontawesome.com/releases/v5.8.2/js/fontawesome.js" integrity="sha384-Ia7KZbX22R7DDSbxNmxHqPQ15ceNzg2U4h5A8dy3K47G2fV1k658BTxXjp7rdhXa" crossorigin="anonymous"></script>
<script>
  var hostButton=$( "a.btn.btn-primary.btn-sm.fas.fa-terminal" );
  var hostname=hostButton.text();
  hostButton.replaceWith(hostname);
  document.cookie = "csrf-token=<%= csrftoken %>; path=/rnode/<%= host %>/<%= port %>; secure";
</script>
<form action="/rnode/<%= host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>"/>
  <input type="hidden" name="password" value="<%= password %>"/>
  <input type="hidden" name="staySignedIn" value="1"/>
  <input type="hidden" name="appUri" value=""/>
  <input type="hidden" name="csrf-token" value="<%= csrftoken %>"/>
  <button class="btn btn-primary" type="submit">
    <i class="fab fa-r-project"></i> Connect to RStudio Server
  </button>
</form>

A cleaner approach would be to (after RStudio starts):

  1. Send a “GET” request to login page
  2. Parse the response and get the csrf-token generated by RStudio
  3. Store the token in the connection.yml
  4. Use the token in connection.yml in the POST request

But there might be a better approach

I’m finding that after an rstudio session is idle for a while (1 hour I think) I get kicked out and
I get the login screen when I try to reconnect. The server has generated a new csrf-token and
the only way to get back in is to put that token in my connection.yml file. So I think we really
need to get @fenz’s approach implemented if we want to use 1.4 in production.

hi all,
I’m struggling to get this working.
I’ve done everything recommended here:

  1. Added csrf_token to conn_params on rstudio/submit.yml.erb
  2. Added export csrf_token=... to template/before.sh.erb
  3. Added document.cookie ... and <input ... name="csrf-token"... to rstudio/view.html.erb

With all that I can launch the rstudio ood app, it seems to be loading the right credentials and setting the token, however, it throws

Could not connect to the R session on RStudio Server. Error occurred during transmission (6)

Any suggestion what to do to debug that issue?

I’m using
centos7
rstudio-server-1.4.1717-1.x86_64
ood v2.0.10
And the following app repository
https://github.com/guimaluf/bc_example_rstudio/tree/fix_1.4_auth

Chrome will let you open your dev tools when you open a new tab. That’s likely what you’ll have to do here, open your browsers dev tools and see what’s being passed in the POST request across the network.

My guess from that error message - there’s a parsing error. Like you’re missing a ; to delimit the cookie or " around the token. Or maybe you need to URL encode something (depending on what the token is that you’ve generated)?

I’ve copied and configured as you have posted here. I couldn’t find anything strange in the header, cookies and responses. I even tried to remove all the csrf_token shanenigans and inject the one generated by rstudio into the request and send it to get a redirect loop with the signin URL.

I’ll try to run rstudio manually and see what I can spot in its logs.
I had a look on rstudio logs. Seems like it is not related to the csrf_token then…

14 Jul 2021 18:08:59 [rserver] ERROR system error 13 (Permission denied) [request-uri: /rpc/client_init]; OCCURRED AT void rstudio::server::session_proxy::{anonymous}::proxyRequest(int, const rstudio::core::r_util::SessionContext&, rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnection>, const ErrorHandler&, const rstudio::core::http::ConnectionRetryProfile&, const ClientHandler&) src/cpp/server/ServerSessionProxy.cpp:678; CAUSED BY: system error 2 (No such file or directory) [description: User not found., user-value: gb637]; OCCURRED AT rstudio::core::Error rstudio::core::system::User::Impl::populateUser(rstudio::core::system::User::Impl::GetPasswdFunc<T>&, T) [with T = const char*  rstudio::core::system::User::Impl::GetPasswdFunc<T> = std::function<int(const char*, passwd*, char*, long unsigned int, passwd**)>] src/cpp/shared_core/system/User.cpp:68; LOGGED FROM: void rstudio::server::session_proxy::{anonymous}::logIfNotConnectionTerminated(const rstudio::core::Error&, const rstudio::core::http::Request&) src/cpp/server/ServerSessionProxy.cpp:419
14 Jul 2021 18:09:00 [rserver] ERROR system error 13 (Permission denied) [request-uri: /rpc/client_init]; OCCURRED AT void rstudio::server::session_proxy::{anonymous}::proxyRequest(int, const rstudio::core::r_util::SessionContext&, rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnection>, const ErrorHandler&, const rstudio::core::http::ConnectionRetryProfile&, const ClientHandler&) src/cpp/server/ServerSessionProxy.cpp:678; CAUSED BY: system error 2 (No such file or directory) [description: User not found., user-value: gb637]; OCCURRED AT rstudio::core::Error rstudio::core::system::User::Impl::populateUser(rstudio::core::system::User::Impl::GetPasswdFunc<T>&, T) [with T = const char*  rstudio::core::system::User::Impl::GetPasswdFunc<T> = std::function<int(const char*, passwd*, char*, long unsigned int, passwd**)>] src/cpp/shared_core/system/User.cpp:68; LOGGED FROM: void rstudio::server::session_proxy::{anonymous}::logIfNotConnectionTerminated(const rstudio::core::Error&, const rstudio::core::http::Request&) src/cpp/server/ServerSessionProxy.cpp:419

I found this issue on the upstream. We don’t run 1.4 yet (we’re still on 1.2) so we don’t yet use those cookie keys. Are you using a module? Seems like you’re having issues with --secure-cookie-key-file?

I to am struggling with this, trying to run rstudio server without using a container or proot. I have the server starting ok, but can’t get this token thing to work at all. Trying to follwo the timeline above and get everything in the right place I have:

view.html.erb:

<script>
  document.cookie = "csrf-token=<%= csrf_token %>; path=/rnode/<%= host %>/<%= port %>; secure";
</script>
<form action="/rnode/<%= host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>">
  <input type="hidden" name="password" value="<%= password %>">
  <input type="hidden" name="staySignedIn" value="1">
  <input type="hidden" name="appUri" value="">
  <input type="hidden" name="csrf-token" value="<%= csrf_token %>"/>
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-registered"></i> Connect to RStudio Server
  </button>
</form>

submit.yml:

conn_params:
    - csrf_token

and before.sh.erb:

<%-
  require 'securerandom'
  csrftoken = SecureRandom.uuid
-%>
export csrf_token="<%= csrftoken %>"

As noted, the server starts fine but when I view the source of the launch bit in firefox developer tools it shows the hidden csrf-token with value="" and auth to the rstudio-server instance fails. What bits am I missing here?

OK, the above worked. Thanks to @jeff.ohrstrom for pointing me to my YAML syntax problem.

I had a similar problem in OOD v2.0.23-1 with bc_example_rstudio. I also needed to make changes to template/bin/auth. In the first conditional statement, “exit 1” is called.

if [[ $# -ne 1 ]]; then
  echo "Usage: auth USERNAME"
  exit 1
fi

To find out the reason, I wrote the following simple script before the conditional statement.

echo $SHELL
echo $# 
echo $@

The result was as follows.

/bin/bash
3
masahiro.nakao rstudio 1

So the value of “$#” was not 1. When I comment out the conditional statement, RStudio will work. But this would be strange.

Hi @guimaluf, have you managed by any chance to get rid of the occurred during transmission (6) error?

I confirm what @mnakao writes. Only difference is that I eventually get

@ll4strw Also see this topic - specifically where I have rstudio log directory mounted to my $WORKING_DIR (the job’s session directory). I’ve found that having these logs outside of the container are critical to finding any issues.

@ll4strw this may also be helpful. Here are all the updates I made to OSC’s app to support the new Rstudio and R/4.1.0. Some small bits are OSC specific like module directory but most updates are related to upgrading Rstudio & R.

@jeff.ohrstrom Thanks for the messages. I got it to work eventually.

In my case the error below was caused by a faulty singularity environment setup

Here is my working custom Lua module that sets up the singularity environment for OOD

help([[ rstudio - loads rstudio with singularity environment for ondemand apps ]])
whatis("loads rstudio with singularity environment for ondemand")
setenv("RSTUDIO_SERVER_IMAGE","/marisdata/COMMON/rserver-launcher-centos7.simg")
setenv("SINGULARITY_BINDPATH","/mnt,/opt,/srv,/usr,/marisdata,/home")
load("RStudio-Server/1.4.1717-foss-2021a-Java-11-R-4.1.0")
local ldcrap = os.getenv("LD_LIBRARY_PATH")
setenv("SINGULARITYENV_LD_LIBRARY_PATH",ldcrap)

Adding /etc to SINGULARITY_BINDPATH caused the observed connection error.

If it can be of any help for others still struggling to get it to work, I report below some other important app files

# cat template/script.sh.erb
#!/usr/bin/env bash

# Load the required environment
setup_env () {
  
  # as reported above
  module load RStudio-OOD-Container
}
setup_env

#
# Start RStudio Server
#

# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"

# These are needed for the logs in the OOD session dir
export RSTUDIO_LOG_CONF="${PWD}/etc"
mkdir -p "${PWD}/logs"
...

# needed for RStudio run files
mkdir -p "$TMPDIR/var/lib"

set -x
# Launch the RStudio Server
echo "Starting up rserver..."

singularity run -B "$TMPDIR:/tmp,$TMPDIR/var/lib:/var/lib/rstudio-server" \
 -B "$RSTUDIO_LOG_CONF:/etc/rstudio" \
 "$RSTUDIO_SERVER_IMAGE" \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --server-data-dir "$TMPDIR/rstudio-server" \
 --server-pid-file "$TMPDIR/rstudio-server/rstudio-server.pid" \
 --rsession-path "${RSESSION_WRAPPER_FILE}" \
 --server-user=$(whoami)

echo 'Singularity has exited...'

# cat submit.yml.erb
---
batch_connect:
  template: "basic"
  conn_params:
    - csrftoken
script:
...
# cat view.html.erb
<script type="text/javascript">
(function () {
    let d = new Date();
    d.setTime(d.getTime() + (7*24*60*60*1000));
    let expires = "expires="+ d.toUTCString();
    let cookie = `csrf-token=<%= csrftoken %>;${expires};path=/rnode/<%= host %>/<%= port %>/;SameSite=Strict;Secure;`
    document.cookie = cookie;
})();



</script>
<form action="/rnode/<%= host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>">
  <input type="hidden" name="password" value="<%= password %>">
  <input type="hidden" name="staySignedIn" value="1">
  <input type="hidden" name="appUri" value="">
  <input id="csrfToken" type="hidden" name="csrf-token" value="<%= csrftoken %>"/>
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-registered"></i> Connect to RStudio Server
  </button>
</form>

# cat template/before.sh.erb
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module

# Find available port to run server on
port=$(find_port ${host})

# Define a password and export it for RStudio authentication
password="$(create_passwd 16)"
export RSTUDIO_PASSWORD="${password}"

# create CSRF token
<%-
  require 'securerandom'
  csrftoken=SecureRandom.uuid
-%>
export csrftoken="<%= csrftoken %>"

Finally template/bin/auth and template/etc/logging.conf.erb are as you already reported.