RStudio Server Issues

brad.traver · May 12, 2022, 6:13pm

I’m trying to deploy RStudio server without the use of Singularity. The following is my output.log file:

Script starting...
Waiting for RStudio Server to open port 5615...
/home/brad.traver
No Modulefiles Currently Loaded.
talon04
mkdir: cannot create directory ‘/tmp/run’: File exists
mkdir: cannot create directory ‘/tmp/lib’: File exists
++ date
+ echo 'Starting up rserver... at Thu May 12 13:06:39 CDT 2022'
Starting up rserver... at Thu May 12 13:06:39 CDT 2022
++ whoami
+ rserver --www-port 5615 --auth-none 0 --auth-pam-helper-path /bin/auth --auth-encrypt-password 0 --rsession-path /home/brad.traver/rsession.sh --server-data-dir=/tmp/run --secure-cookie-key-file /tmp/rstudio-server/secure-cookie-key --server-user=brad.traver
Timed out waiting for RStudio Server to open port 5615! on Thu May 12 13:08:43 CDT 2022
Cleaning up...

This is my script file:

#!/usr/bin/env bash

<%-
  session_dir = session.staged_root
%>

#
# Start RStudio Server
#
# PAM auth helper used by RStudio
export RSTUDIO_AUTH="$WORKING_DIR/bin/auth"
export WORKING_DIR=$HOME
echo $WORKING_DIR
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="$WORKING_DIR/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "$WORKING_DIR/rsession.sh" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="$WORKING_DIR/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"
  set -x

  # rsession.sh doesn't share the same env as the outside script, so these
  # need to be set explicitly
  export R_LIBS_SITE="${R_LIBS_SITE}"
  export TZ="US/Eastern"

  env

  # Launch the original command
  echo "Launching rsession..."
  rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "$WORKING_DIR/rsession.sh"

# Output debug info
module list
hostname

# rstudio runtime stuff
mkdir "$TMPDIR/run"
mkdir "$TMPDIR/lib"

# server log directory in this job's working directory
mkdir -p "$WORKING_DIR/logs"

set -x
# Launch the RStudio Server
echo "Starting up rserver... at $(date)"

rserver --www-port "${port}" --auth-none 0 --auth-pam-helper-path "${RSTUDIO_AUTH}" --auth-encrypt-password 0 --rsession-path "${RSESSION_WRAPPER_FILE}" --server-data-dir="/tmp/run" --secure-cookie-key-file "${TMPDIR}/rstudio-server/sec$

I’m not sure what I’m missing here.

travert · May 12, 2022, 7:04pm

Hi and welcome!

One question I’d have is where are you when the rserver command is issued?

Looking at the OSC version here:

github.com

OSC/bc_osc_rstudio_server/blob/master/template/script.sh.erb#L162

      
        
            --auth-none 0 \
            --auth-pam-helper-path "${RSTUDIO_AUTH}" \
            --auth-encrypt-password 0 \
            --rsession-path "${RSESSION_WRAPPER_FILE}" \
            <%- if r_version > '4.1' -%>
            --server-data-dir='/tmp/run' \
            --server-user=$(whoami)
            <%- end -%>
            
            
<%- else -%>
            cd ${SINGULARITY_HOME:=$HOME}
            rserver \
             --www-port 8080 \
             --auth-none 0 \
             --auth-pam-helper-path "${RSTUDIO_AUTH}" \
             --auth-encrypt-password 0 \
             --rsession-path "${RSESSION_WRAPPER_FILE}" \
             <%- if r_version > '4.1' -%>
             --server-data-dir='/tmp/run' \
             --server-user=$(whoami)
             <%- end -%>

It may be as simple as making sure you are in the $HOME before issuing the rserver command.

Also, if you haven’t it may help to take a look at the README.md in that repo which goes over a non-singularity build’s prerequisites:

Because I’m also not seeing any kind of bind mount for singularity in the script posted either.

brad.traver · May 12, 2022, 7:07pm

It’s being issued at the end of the script. All of the prerequisites are installed as listed on the GitHub page.

travert · May 12, 2022, 7:27pm

I’m not sure they are. Your module list is saying there are no modules loaded so the prereqs likely aren’t there from a module purge somewhere maybe? In your before.sh possibly. We can see that here:

At the least we should see R loaded. There should be some module loads in the script for the needed modules such as R.

Also, what version of Rstudio are you running?

I still am not sure where the cd ${HOME} is that you say is ran at the bottom of the script either.

This thread may be of some use to gather what you may need to set to run without singularity:

brad.traver · May 12, 2022, 7:59pm

R is not installed as a module. It’s installed as part of the compute node image. All of the necessary executables should be in $PATH. RStudio and RStudio Server are both version 2022.02.2+485. I’ve added cd $HOME to the top of the script but the same issue persists. I will go through the thread you sent.

brad.traver · May 13, 2022, 6:16pm

I’ve resolved the other issues with my script. It is below:

#!/usr/bin/env bash

<%-
  session_dir = session.staged_root
%>

#
# Start RStudio Server
#
# PAM auth helper used by RStudio
cd $HOME
export RSTUDIO_AUTH="${PWD}/bin/auth"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
echo $RSESSION_WRAPPER_FILE
(
umask 077
sed 's/^ \{2\}//' > "$RSESSION_WRAPPER_FILE" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="${PWD}/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"
  set -x

  # rsession.sh doesn't share the same env as the outside script, so these
  # need to be set explicitly
  export R_LIBS_SITE="${R_LIBS_SITE}"
  export TZ="US/Eastern"

  env

  # Launch the original command
  echo "Launching rsession..."
  rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${PWD}/rsession.sh"

# Output debug info
hostname

# rstudio runtime stuff
export TMPDIR="$(mktemp -d)"

# server log directory in this job's working directory
mkdir -p "${PWD}/logs"

set -x
# Launch the RStudio Server
echo "Starting up rserver... at $(date)"

rserver --www-port "${port}" --auth-none 0 --auth-pam-helper-path "${RSTUDIO_AUTH}" --auth-encrypt-password 0 --rsession-path "${RSESSION_WRAPPER_FILE}" --server-data-dir="${TMPDIR}" --secure-cookie-key-file "${TMPDIR}/rstudio-server/se$

The same problem persists.

Script starting...
Waiting for RStudio Server to open port 47969...
/home/brad.traver/rsession.sh
talon04
++ date
+ echo 'Starting up rserver... at Fri May 13 12:56:41 CDT 2022'
Starting up rserver... at Fri May 13 12:56:41 CDT 2022
++ whoami
+ rserver --www-port 47969 --auth-none 0 --auth-pam-helper-path /home/brad.traver/bin/auth --auth-encrypt-password 0 --rsession-path /home/brad.traver/rsession.sh --server-data-dir=/tmp/tmp.4DIcyRTg9U --secure-cookie-key-file /tmp/tmp.4DIcyRTg9U/rstudio-server/secure-cookie-key --server-user=brad.traver
Timed out waiting for RStudio Server to open port 47969! on Fri May 13 12:58:45 CDT 2022
Cleaning up...

jeff.ohrstrom · May 13, 2022, 7:12pm

A couple things. If you want to speed up debugging cycles, you can follow this pattern:

Launch the job and watch it fail
navigate to it’s session directory (where you’re finding those logs)
At this point you can edit script.sh with what changes you want. (You’ll need to hard somethings like port and so on)
Now you can run script.sh directly and it doesn’t redirect or background itself. You’ll issue rserver in the foreground and it may tell you why it’s not starting.

Back to the RStudio issue - obviously it’s failing to start up, so that pattern above will help you get it running and at least trying to output to stdout what’s wrong. If it’s not saying anything in stdout then you may have to look at system logs that it’s writing (if it’s writing any).

We run RStudio container because (a) I want to override the /etc defaults (shown below so that I can change the log-dir) and (b) a system installed Rstudio really likes to write & read from system directories like /var and /etc. I dont’ know why or how many setting you’d need to change to get it to run, but you’ll probaby have to confer with the default settings and the configurations it may have installed in /etc/rstudio.

github.com

OSC/bc_osc_rstudio_server/blob/1ce2e4fb1ddb25921069a1e31cb3edc02ce90535/template/script.sh.erb#L144

      
        
            # --contain is used to override the bindpaths specified in OSC's
            # singularity.conf which would otherwise conflict in /etc
            # using --contain requires us to bind mount in addition directories including
            # /dev and /proc
            # optional_tutorial_home_bind is used when include_tutorials to change the home
            # directory to get a clean environment without clobbering files
            SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH" \
              singularity run --contain \
             -B $TMPDIR:/tmp \
             -B $WORKING_DIR \
             -B $WORKING_DIR/etc:/etc/rstudio \
             -B /etc/fonts \
             -B /etc/profile \
             -B /etc/profile.d \
             -B /etc/alternatives \
             -B <%= module_dir %> \
             "$RSTUDIO_SERVER_IMAGE" \
             --www-port "${port}" \
             --auth-none 0 \
             --auth-pam-helper-path "${RSTUDIO_AUTH}" \
             --auth-encrypt-password 0 \

maflister · May 17, 2022, 1:25pm

Not sure if this is the exact issue, but RStudio Server 1.4 and above require a --database-config-file option when launching rserver. You’ll find some info here.

brad.traver · May 18, 2022, 9:20pm

After for further tweaking, I managed to get RStudio Server to start. When I click the “Connect to RStudio” button, the RStudio login page comes up with the following error:
Error: Temporary server error. Please try again.

My script is below:

#!/usr/bin/env bash

<%-
  session_dir = session.staged_root
%>

#
# Start RStudio Server
#
# PAM auth helper used by RStudio
cd $HOME
export WORKING_DIR=${HOME}/r_working
export RSTUDIO_AUTH="$/home/brad.traver/ondemand/dev/rstudio/template/bin/auth"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${WORKING_DIR}/rsession.sh"
mkdir r_working
cd $WORKING_DIR
echo $RSESSION_WRAPPER_FILE
(
umask 077
sed 's/^ \{2\}//' > "$RSESSION_WRAPPER_FILE" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="${WORKING_DIR}/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"
  set -x

  # rsession.sh doesn't share the same env as the outside script, so these
  # need to be set explicitly
  export R_LIBS_SITE="${R_LIBS_SITE}"
  export TZ="US/Eastern"

  env

  # Launch the original command
  echo "Launching rsession..."
  rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${WORKING_DIR}/rsession.sh"

# Output debug info
hostname

# rstudio runtime stuff
export TMPDIR="$(mktemp -d)"

# server log directory in this job's working directory
mkdir -p "${WORKING_DIR}/logs"

set -x
# Launch the RStudio Server
echo "Starting up rserver... at $(date)"

rserver --www-port "${port}" --auth-none 0 --auth-pam-helper-path "${RSTUDIO_AUTH}" --auth-encrypt-password 0 --rsession-path "${RSESSION_WRAPPER_FILE}" --server-data-dir="${TMPDIR}" --secure-cookie-key-file "${TMPDIR}/rstudio-server/secure-cookie-key" --server-user=$(whoami) --database-config-file "/home/brad.traver/ondemand/dev/rstudio/template/etc/database.conf"

I think my issues might be caused by lack of configuration in the rsession and database config files. The database config file does have one line of configuration in it.

brad.traver · May 26, 2022, 8:54pm

With help, I managed to figure it out. I seem to remember something about putting apps in /etc/ood/config/apps instead of /var/www/ood/apps/sys even though the docs say to put apps in /var/www/ood/apps/sys. @jeff.ohrstrom was that you? Can you clarify?

jeff.ohrstrom · May 26, 2022, 9:06pm

Sorry, in sum - deploy your apps to /var/www and your app configurations to /etc/ood.

all apps are in /var/www/ood/.... We distribute apps, so they’ll get overwritten - the ones you write, even in that location, we don’t change or modify or anything.
/etc is for configuration of those apps and we look at files there to configure our apps, but never replace them.

brad.traver · May 26, 2022, 9:26pm

I’m very confused now. Where should the scripts and forms be?

jeff.ohrstrom · May 27, 2022, 12:11am

/var/www/ood/.... We provide automation for puppet or ansible too.

system · November 23, 2022, 12:12am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
RStudio without Singularity? Get Help question	15	2982	May 26, 2022
RStudio server on arm64 node Get Help	2	669	November 26, 2023
RStudio App stuck on 'Starting' Get Help ondemand2 , question	3	339	April 9, 2023
Looking for some debugging tips re: rstudio app on el8 Get Help	10	2454	May 26, 2022
Rserver in the Singularity container vs rstudio-server installed from rpm Get Help	9	1621	May 26, 2022

RStudio Server Issues

Related Topics