RStudio when launched without Singularity is having strange troubles with authentication

It appears from the patch in the upstream ticket all you need to set is the environment variable RS_SESSION_TMP_DIR to overcome the original issue which is a hard-coded temporary file location.

Other than that, I’m not sure what --server-data-dir does without having to research it a bit and it would seem to me that --secure-cookie-key-file is an alternative to PAM, but again without looking into it more I can’t say for sure.

We only used singularity because they hard-coded this temporary file path. I don’t think it has anything to do with auth and it’s as of yet, unclear to me why these other users are using these flags or what they do. so no, I don’t believe the secure cookie is a requirement for Rstudio 1.3 (though we don’t run 1.3 yet, so I could be wrong, but it would seem that they’d keep PAM auth).

That said, I too am looking for online documentation for rstudio-server, but can’t seem to find online docs about the command (I can only find docs on the cfg files which we at OSC don’t use). When I got time I was going to look at --help messages from the command line.

In short, my suggestion is to use PAM and set RS_SESSION_TMP_DIR. And use git to save all your progress as you go through and try this and that.

1 Like

We’ll give it a try.

Okay, I tried adding the RS_SESSION_TMP_DIR variable and keeping the original arguments from the working system in singularity but just removed the singularity bits. While using our source built rstudio 1.3 it doesn’t work still, however the rstudio 1.1 we have rpm installed works just swimmingly… we are going to test our pipeline but it may just be an issue with that version of rstudio. We will post an update if we get it working and what we did but i think the RS_SESSION_TMP_DIR fixed the original issue as you stated. Keep you posted!

Those additional flags ensure that more than one instance of rstudio server can run on a single compute node. This is in the case that you allow shared nodes. The issue with older versions of rstudio server (which Singularity solved) was auto generated files in /tmp. If you had more than one rstudio server job running on single node, they would conflict. The two additional flags ensure that you’re writing these generated files to unique directories on a per job basis. It does not affect authentication. For reference, our site uses the following flags:

rserver \
  --www-port ${port} \
  --auth-none 0 \
  --auth-pam-helper-path "${RSTUDIO_AUTH}" \
  --auth-encrypt-password 0 \
  --rsession-path "${RSESSION_WRAPPER_FILE}" \
  --server-data-dir "${TMPDIR}" \ 
  --secure-cookie-key-file "${TMPDIR}/rstudio-server/secure-cookie-key"

Again, the last two flags aren’t required, especially if you don’t allow jobs to share nodes. However, they are a good precaution.

Also, I’d suggest that you download and dump the RPM contents vs building from source.

2 Likes

We tried updating to 1.3 with the rpm install as suggested however we are now unable to launch the rstudio session and see the following error in the /var/log/messages:

Nov 23 08:47:32 cn002 rserver[25662]: ERROR system error 1 (Operation not permitted) [path: /var/run/rstudio-server/revocation-list, description: Could not set revocation file permissions to 600 for file: /var/run/rstudio-server/revocation-list]; OCCURRED AT rstudio::core::Error rstudio::core::system::changeFileMode(const rstudio::core::FilePath&, mode_t) src/cpp/server/ServerMain.cpp:62; LOGGED FROM: rstudio::core::Error rstudio::server::auth::handler::initialize() src/cpp/server/auth/ServerAuthHandler.cpp:530
Nov 23 08:47:32 cn002 rserver[25662]: ERROR Could not write to revocation list; LOGGED FROM: rstudio::core::Error rstudio::server::auth::handler::initialize() src/cpp/server/auth/ServerAuthHandler.cpp:549
Nov 23 08:47:32 cn002 rserver[25662]: ERROR system error 2 (No such file or directory) [path: /var/run/rstudio-server/revocation-list]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::openForWrite(std::shared_ptr<std::basic_ostream<char> >&, bool) const src/cpp/shared_core/FilePath.cpp:1225; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:674

Based on the rpm install it specifically calls out that we need the flag --server-data-dir however when we add it we are unable to authenticate with the pam auth.
Ref:

  --server-data-dir arg (=/var/run/rstudio-server)
                                        path to data directory where rstudio 
                                        server will write run-time state

Here is what the working rstudio 1.1 HTML looks like:

Which is different from the rstudio 1.3 HTML, could this be a problem with how we are passing the authentication credentials in?

There are a few things going on here. The error in /var/log/messages is caused by an issue with your $TMPDIR, --server-data-dir, and --secure-cookie-key-file setup in script.sh.erb. Looks like rserver isn’t using a unique directory for --server-data-dir, so it tries to write its runtime files to /var/run/rstudio-server. In that case, rserver is running under an unprivileged user, so it can’t write to that root-owned location.

What does your script.sh.erb look like now? Ours is as follows:

#!/usr/bin/env bash
module purge

#
# Start RStudio Server
#

# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"

# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
  #!/usr/bin/env bash
  # Log all output from this script
  export RSESSION_LOG_FILE="${PWD}/rsession.log"
  exec &>>"\${RSESSION_LOG_FILE}"
  # Launch the original command
  echo "Launching rsession..."
  set -x
  exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"

# Set working directory to home directory
cd "${HOME}"

# Create a unique $TMPDIR for runtime files
export TMPDIR="$(mktemp -d)"

# Launch the RStudio Server
echo "Starting up rserver..."
set -x
rserver \
  --www-port ${port} \
  --auth-none 0 \
  --auth-pam-helper-path "${RSTUDIO_AUTH}" \
  --auth-encrypt-password 0 \
  --rsession-path "${RSESSION_WRAPPER_FILE}" \
  --server-data-dir "${TMPDIR}" \
  --secure-cookie-key-file "${TMPDIR}/rstudio-server/secure-cookie-key"

The bottom section with the TMPDIR setting and the rserver command are the probably most important. Perhaps we can cobble together something between your script.sh.erb and ours.

1 Like

We were able to fix our TMPDIR directory issue by specifying the --server-data-dir as we did previously. However, when implementing that we are no longer able to authenticate with the export RSTUDIO_AUTH="${PWD}/bin/auth and it doesn’t give us any output to suggest where the problem with authentication is.

Here is our current script:

# Purge the module environment to avoid conflicts
#module purge

module load shared

# Load the required modules
module load R/3.6.3_GCC-9.3.0
module load python/3.8.0
#module load rstudio/.1.3.1093
module load pbspro/2020.1
# log statements
echo $PATH
module list
which R
which python
python --version
which python3
python3 --version
echo $LD_LIBRARY_PATH

# Load the required environment
setup_env () {
  # Additional environment which could be moved into a module
  # Change these to suit
#  export RSTUDIO_SERVER_IMAGE="/depot/apps/singularity_images/3.6.0/rserver-launcher-centos7.simg"
#  export SINGULARITY_BINDPATH="/gpfs,/cm,/etc,/media,/mnt,/opt,/srv,/usr,/var,/depot,/active,/archive"
  export PATH="$PATH:/usr/lib/rstudio-server/bin"
#  export SINGULARITYENV_PATH="$PATH"
  export MODULEPATH="$MODULEPATH:/depot/Modules/modulefiles"
  # In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH
  # to the singularity process
#  export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
}
setup_env

# Start RStudio Server
#

# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"
export RSTUDIO_HOME="${PWD}"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="${PWD}/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"

  # Launch the original command
  echo "Launching rsession..."
  set -x
  exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"

# Set working directory to home directory
cd "${HOME}"

export TMPDIR="$(mktemp -d)"
export RS_SESSION_TMP_DIR="$TMPDIR"
echo $RS_SESSION_TMP_DIR
mkdir -p "$TMPDIR/rstudio-server"
python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"

set -x
# Launch the RStudio Server
echo "Starting up rserver..."

#singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
rserver \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --rsession-path "${RSESSION_WRAPPER_FILE}" \
 --server-user "$USER" \
 --server-data-dir "${RSTUDIO_HOME}" 
# --server-data-dir "${TMPDIR}" 

#echo 'Singularity has exited...'

I feel like this is a problem with how the RSTUDIO_AUTH parameter is being passed into the URL for ondemand. Unfortunately I just don’t know how to fix that piece as I am not seeing a definitive error to say it is a problem. No logs or errors are being generated with clicking the “connect to Rstudio” but if I try to enter my username and the generated password I see an error in the rstudio 1.3 UI but no where else. If I try to type in an arbitrary (non-existing user) like “user: rstudio” “password: rstudio” it will error out in the logs saying there was a problem with pam auth for user ‘rstudio’

What does your RSTUDIO_AUTH script look like? The original version that worked with 1.1 needs to be modified to work with 1.3. See https://github.com/OSC/bc_osc_rstudio_server/issues/34.

2 Likes
#!/usr/bin/env bash

# Confirm username is supplied
if [[ $# -ne 1 ]]; then
  echo "Usage: auth USERNAME"
  exit 1
fi
USERNAME="${1}"

# Confirm password environment variable exists
if [[ -z ${RSTUDIO_PASSWORD} ]]; then
  echo "The environment variable RSTUDIO_PASSWORD is not set"
  exit 1
fi

# Read in the password from user
read -s -p "Password: " PASSWORD
echo ""

if [[ ${USERNAME} == ${USER} && ${PASSWORD} == ${RSTUDIO_PASSWORD} ]]; then
  echo "Successful authentication"
  exit 0
else
  echo "Invalid authentication"
  exit 1
fi

… Okay so can confirm replacing ‘-ne’ with ‘-lt’ in the template/bin/auth totally fixed our issue. I absolutely love that this worked and thanks for the call out… but dang SO much time for that one issue.

Thanks!

1 Like

Full info + Solution: Thanks to @dugan and @maflister
For those who run into this in the future here is the changes that had to be implemented:
Installed rstudio 1.3.1093 natively with the rpm (will test with source later)
yum install -y rstudio-server-rhel-1.3.1093-x86_64.rpm
In thetemplate/script.sh.erb ensure you use the following :

rserver \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --rsession-path "${RSESSION_WRAPPER_FILE}" \
 --server-data-dir "${TMPDIR}" \
 --secure-cookie-key-file "${TMPDIR}/rstudio-server/secure-cookie-key"

Ref: @maflister specifically https://github.com/mcw-rcc/bc_rcc_rstudio_server

If you haven’t already, you need to update the template/bin/auth and change the ‘-ne’ flag to ‘-lt’

#!/usr/bin/env bash

# Confirm username is supplied

if [[ $# -lt 1 ]]; then

echo "Usage: auth USERNAME"

exit 1

fi

USERNAME="${1}"

# Confirm password environment variable exists

if [[ -z ${RSTUDIO_PASSWORD} ]]; then

echo "The environment variable RSTUDIO_PASSWORD is not set"

exit 1

fi

# Read in the password from user

read -s -p "Password: " PASSWORD

echo ""

if [[ ${USERNAME} == ${USER} && ${PASSWORD} == ${RSTUDIO_PASSWORD} ]]; then

echo "Successful authentication"

exit 0

else

echo "Invalid authentication"

exit 1

fi

Ref: @dugan specifically https://github.com/OSC/bc_osc_rstudio_server/issues/34

What type of logging or error feedback would have helped reduce the amount of time it took to debug this issue? Perhaps there is something we can change in OnDemand to help with debugging problems like this in the future.

Not sure how to accomplish it, but some kind of logging on wether the pam auth found in template/bin/auth was actually being invoked or if it failed out because of syntax would be nice. It was really hard to pinpoint what the actual problem was because as far as we could see there wasn’t anything “erroring”.
If that level of error logging does exist we didn’t have a good place to reference that in the documentation.

I have this same issue with Rstudio server 1.4 container, but not 1.2 or 1.3. Has anyone been able to find the magic to fix it for 1.4 please?
Inspecting the “connect to” part of the web page shows the user/pass being stored correctly, but i have to manually enter them on the next screen in order to enter rstudio

I have issues as well with RStudio server 1.4, it seems authentication request from OOD app is not accepted anymore.
Can anyone help with this?

I think RStudio Server 1.4 still has an issue. Suggest to use RStudio Server 1.3.

I feel something was changed since in the 1.3.x if I enter the “auth-do-sign-in” url I get:

  1. Redirected to “auth-sign-in?error=1”
  2. An error: “Error: Incorrect or invalid username/password”

and that’s expected since I’m just entering it in the browser (GET request without parameters)

But if I do the same in the version 1.4.x I get:

  1. Redirected to “auth-sign-in?appUri=appUri&error=2”
  2. An error: “Error: Temporary server error, please try again”

So either this authentication method got deprecated (maybe it there’s a different url to send the request) or that’s not working “by default” and needs some configuration in “rserver.conf” file.
I’m trying to understand more but if there’s anyone with more expertise which may help it could be useful.
@maflister are you saying this is a known bug which RStudio folks are working on?

@fenz I think might be related to this: