So we recently followed the instructions in this thread to deploy RStudio on OOD without singularity. We did this for a couple reasons. The 1st was that we want to upgrade to newer versions of RStudio and the 2nd was that the singularity container was preventing certain popular tools (future.batchtools) from seeing our clusters scheduler (which was more a limitation caused by the PBS scheduler than the container, but this was one way to solve it).
The only problem is that we turn on authentication with --auth-none, it give us this:
We are VERY mystified about that username. It feels like it should be a major clue, but we donât know where that is coming from.
And we are also confused as to why this does not log us in as it used to before we switched away from the almost empty singularity container. Pasting in the expected username and the generated password while itâs running also does not work. And from other testing, it seems that the --auth-pam-helper-path argument is really not respecting the values for the username and password that are generated and reported by the script.sh.erb
Anyone have an idea about whatâs happening? The script inside of template/bin/auth certainly seems simple enough. But I wish I could get a bit more insight into what it sees when it is running.
Weâre kind of busy with a Slurm migration at OSC this week, but Iâll try to carve out time to replicate this.
Whatâs your view.html.erb look like? Iâm also mystified about endClipDistance. Also what version of rstduio-server are you running? That could help, there seem to be some issues on their github about auth-none.
No stress, we really appreciate you taking a look at it as well. FYI, I donât see the âendClipDistanceâ but not sure where to check for logs regarding the application fails because they donât show up in regular /var/log/messages.
Here is the âview.html.erbâ:
Also, the version of RStudio we are using for this is 1.3.1093. As we understand it, that was required for being able to set up RStudio without a container as it has the arguments to control where the RStudio server writes some of itâs files. And the good news is that this part seems to work and it also solved out problem where RStudio could not see the PBS scheduler.
The only issue we have is that if we turn on the authentication, then it no longer logs in each user to their own session and give us that weird endClipDistance username.
So another part of the mystery is better understood. Why do I see that âendClipDistanceâ user when Bronson does not? I did some more testing and tried alternate browsers. Here is what Edge does:
And here is what the Brave browser reports (which is also what happens for Bronson and that makes sense because he ALSO uses the Brave browser).
The image above with the âendClipDistanceâ username was from the Chrome browser (which I use). So now, that specific detail seems a LOT less interesting in terms of the mystery about why RStudio does not connect!
OK cool. So to be clear, you want to use the secure-cookie auth instead of basic auth? My guess is youâre being prompted because you arenât passing the cookie (or presenting some form of it), but Iâm not entirely sure how this authentication method should work so Iâll have to dig into that a bit.
Does that mean if we want to still use the pam authentication we just need to drop that cookie file? I believe we tried that and it didnât work. The original post was:
So I think itâs more that we wanted the basic auth, but felt like we also were required to use the secure cookie to use RStudio without singularity⌠Is that not the case?
Also, is there a place where these arguments are documented really well (that we perhaps just donât know about)? Because we have looked for help and not found very much to go on for these arguments. Frankly, we could use some help to better understand these.
It appears from the patch in the upstream ticket all you need to set is the environment variable RS_SESSION_TMP_DIR to overcome the original issue which is a hard-coded temporary file location.
Other than that, Iâm not sure what --server-data-dir does without having to research it a bit and it would seem to me that --secure-cookie-key-file is an alternative to PAM, but again without looking into it more I canât say for sure.
We only used singularity because they hard-coded this temporary file path. I donât think it has anything to do with auth and itâs as of yet, unclear to me why these other users are using these flags or what they do. so no, I donât believe the secure cookie is a requirement for Rstudio 1.3 (though we donât run 1.3 yet, so I could be wrong, but it would seem that theyâd keep PAM auth).
That said, I too am looking for online documentation for rstudio-server, but canât seem to find online docs about the command (I can only find docs on the cfg files which we at OSC donât use). When I got time I was going to look at --help messages from the command line.
In short, my suggestion is to use PAM and set RS_SESSION_TMP_DIR. And use git to save all your progress as you go through and try this and that.
Okay, I tried adding the RS_SESSION_TMP_DIR variable and keeping the original arguments from the working system in singularity but just removed the singularity bits. While using our source built rstudio 1.3 it doesnât work still, however the rstudio 1.1 we have rpm installed works just swimmingly⌠we are going to test our pipeline but it may just be an issue with that version of rstudio. We will post an update if we get it working and what we did but i think the RS_SESSION_TMP_DIR fixed the original issue as you stated. Keep you posted!
Those additional flags ensure that more than one instance of rstudio server can run on a single compute node. This is in the case that you allow shared nodes. The issue with older versions of rstudio server (which Singularity solved) was auto generated files in /tmp. If you had more than one rstudio server job running on single node, they would conflict. The two additional flags ensure that youâre writing these generated files to unique directories on a per job basis. It does not affect authentication. For reference, our site uses the following flags:
We tried updating to 1.3 with the rpm install as suggested however we are now unable to launch the rstudio session and see the following error in the /var/log/messages:
Nov 23 08:47:32 cn002 rserver[25662]: ERROR system error 1 (Operation not permitted) [path: /var/run/rstudio-server/revocation-list, description: Could not set revocation file permissions to 600 for file: /var/run/rstudio-server/revocation-list]; OCCURRED AT rstudio::core::Error rstudio::core::system::changeFileMode(const rstudio::core::FilePath&, mode_t) src/cpp/server/ServerMain.cpp:62; LOGGED FROM: rstudio::core::Error rstudio::server::auth::handler::initialize() src/cpp/server/auth/ServerAuthHandler.cpp:530
Nov 23 08:47:32 cn002 rserver[25662]: ERROR Could not write to revocation list; LOGGED FROM: rstudio::core::Error rstudio::server::auth::handler::initialize() src/cpp/server/auth/ServerAuthHandler.cpp:549
Nov 23 08:47:32 cn002 rserver[25662]: ERROR system error 2 (No such file or directory) [path: /var/run/rstudio-server/revocation-list]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::openForWrite(std::shared_ptr<std::basic_ostream<char> >&, bool) const src/cpp/shared_core/FilePath.cpp:1225; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:674
Based on the rpm install it specifically calls out that we need the flag --server-data-dir however when we add it we are unable to authenticate with the pam auth.
Ref:
--server-data-dir arg (=/var/run/rstudio-server)
path to data directory where rstudio
server will write run-time state
There are a few things going on here. The error in /var/log/messages is caused by an issue with your $TMPDIR, --server-data-dir, and --secure-cookie-key-file setup in script.sh.erb. Looks like rserver isnât using a unique directory for --server-data-dir, so it tries to write its runtime files to /var/run/rstudio-server. In that case, rserver is running under an unprivileged user, so it canât write to that root-owned location.
What does your script.sh.erb look like now? Ours is as follows:
#!/usr/bin/env bash
module purge
#
# Start RStudio Server
#
# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
#!/usr/bin/env bash
# Log all output from this script
export RSESSION_LOG_FILE="${PWD}/rsession.log"
exec &>>"\${RSESSION_LOG_FILE}"
# Launch the original command
echo "Launching rsession..."
set -x
exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"
# Set working directory to home directory
cd "${HOME}"
# Create a unique $TMPDIR for runtime files
export TMPDIR="$(mktemp -d)"
# Launch the RStudio Server
echo "Starting up rserver..."
set -x
rserver \
--www-port ${port} \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-data-dir "${TMPDIR}" \
--secure-cookie-key-file "${TMPDIR}/rstudio-server/secure-cookie-key"
The bottom section with the TMPDIR setting and the rserver command are the probably most important. Perhaps we can cobble together something between your script.sh.erb and ours.
We were able to fix our TMPDIR directory issue by specifying the --server-data-dir as we did previously. However, when implementing that we are no longer able to authenticate with the export RSTUDIO_AUTH="${PWD}/bin/auth and it doesnât give us any output to suggest where the problem with authentication is.
Here is our current script:
# Purge the module environment to avoid conflicts
#module purge
module load shared
# Load the required modules
module load R/3.6.3_GCC-9.3.0
module load python/3.8.0
#module load rstudio/.1.3.1093
module load pbspro/2020.1
# log statements
echo $PATH
module list
which R
which python
python --version
which python3
python3 --version
echo $LD_LIBRARY_PATH
# Load the required environment
setup_env () {
# Additional environment which could be moved into a module
# Change these to suit
# export RSTUDIO_SERVER_IMAGE="/depot/apps/singularity_images/3.6.0/rserver-launcher-centos7.simg"
# export SINGULARITY_BINDPATH="/gpfs,/cm,/etc,/media,/mnt,/opt,/srv,/usr,/var,/depot,/active,/archive"
export PATH="$PATH:/usr/lib/rstudio-server/bin"
# export SINGULARITYENV_PATH="$PATH"
export MODULEPATH="$MODULEPATH:/depot/Modules/modulefiles"
# In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH
# to the singularity process
# export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
}
setup_env
# Start RStudio Server
#
# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"
export RSTUDIO_HOME="${PWD}"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
#!/usr/bin/env bash
# Log all output from this script
export RSESSION_LOG_FILE="${PWD}/rsession.log"
exec &>>"\${RSESSION_LOG_FILE}"
# Launch the original command
echo "Launching rsession..."
set -x
exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"
# Set working directory to home directory
cd "${HOME}"
export TMPDIR="$(mktemp -d)"
export RS_SESSION_TMP_DIR="$TMPDIR"
echo $RS_SESSION_TMP_DIR
mkdir -p "$TMPDIR/rstudio-server"
python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"
set -x
# Launch the RStudio Server
echo "Starting up rserver..."
#singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
rserver \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-user "$USER" \
--server-data-dir "${RSTUDIO_HOME}"
# --server-data-dir "${TMPDIR}"
#echo 'Singularity has exited...'
I feel like this is a problem with how the RSTUDIO_AUTH parameter is being passed into the URL for ondemand. Unfortunately I just donât know how to fix that piece as I am not seeing a definitive error to say it is a problem. No logs or errors are being generated with clicking the âconnect to Rstudioâ but if I try to enter my username and the generated password I see an error in the rstudio 1.3 UI but no where else. If I try to type in an arbitrary (non-existing user) like âuser: rstudioâ âpassword: rstudioâ it will error out in the logs saying there was a problem with pam auth for user ârstudioâ
#!/usr/bin/env bash
# Confirm username is supplied
if [[ $# -ne 1 ]]; then
echo "Usage: auth USERNAME"
exit 1
fi
USERNAME="${1}"
# Confirm password environment variable exists
if [[ -z ${RSTUDIO_PASSWORD} ]]; then
echo "The environment variable RSTUDIO_PASSWORD is not set"
exit 1
fi
# Read in the password from user
read -s -p "Password: " PASSWORD
echo ""
if [[ ${USERNAME} == ${USER} && ${PASSWORD} == ${RSTUDIO_PASSWORD} ]]; then
echo "Successful authentication"
exit 0
else
echo "Invalid authentication"
exit 1
fi
