Hiya, please see copypasta below, but long story short: I’m trying to debug exactly where my app scripting is hitting a wall which leads to logging…
Singularity as exited…
Timed out waiting for RStudio Server to open port <#>
Cleaning up…
…in the output.log file, but nothing else that looks like an error. No other *log file is created (e.g. rsession.log) so I’m hoping for some advice as to what I can add to the script(s) to try and pin down where the issue exists. Please and thanks!
(We’re jumping to this el8 instance from el6 and a much older version of Singularity, which is apparently too many variables for me to work through! Don’t judge the chaotic mess I made in setup_env; I know it’s fugly!)
Here’s a couple files from my last launch attempt…
output.log
Script starting...
Waiting for RStudio Server to open port 48640...
[from script.sh.erb] Module to load: R/4.0.2 ##
[from script.sh.erb] Loaded modules: ##
##
Currently Loaded Modules: ##
1) R/4.0.2 2) anaconda3/2020.07 ##
##
*** which rsession ## Debug reporting
/usr/lib/rstudio-server/bin/rsession ## added by me
*** PATH is ##
/packages/anaconda3/2020.07/condabin ##
/packages/anaconda3/2020.07/bin ##
/packages/singularity/3.6.2/bin ##
/packages/R/4.0.2/bin ##
/common/bin ##
/usr/lib/rstudio-server/bin ##
/usr/local/bin ##
/usr/local/sbin ##
/usr/bin ##
/usr/sbin ##
+ echo 'Starting up rserver...'
Starting up rserver...
+ singularity run -B /tmp/jtb49/35070921/tmp.1c580rbc7Y:/tmp /common/contrib/containers/rserver-launcher-centos6-jb.simg --www-port 48640 --auth-none 0 --auth-pam-helper-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/a148a999-9c63-446d-9345-220412d48a34/bin/auth --auth-encrypt-password 0 --rsession-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/a148a999-9c63-446d-9345-220412d48a34/rsession.sh
+ echo 'Singularity as exited...'
Singularity as exited...
Timed out waiting for RStudio Server to open port 48640!
Cleaning up...
rsession.sh
#!/usr/bin/env bash
# Log all output from this script
export RSESSION_LOG_FILE="/home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/a148a999-9c63-446d-9345-220412d48a34/rsession.log"
exec &>>"${RSESSION_LOG_FILE}"
# Launch the original command
echo "**Which rsession"
which rsession
echo "Launching rsession..."
set -x
exec rsession --r-libs-user "/home/jtb49/R/4.0.2:/home/jtb49/R/%v" "${@}"
script.sh
#!/usr/bin/env bash
# Load the required environment
setup_env () {
# Additional environment which could be moved into a module
# Change these to suit
export RSTUDIO_SERVER_IMAGE="/common/contrib/containers/rserver-launcher-centos6-jb.simg"
export SINGULARITY_BINDPATH="/bin,/usr/bin,/scratch,/o-scratch,/common,/packages,/etc,/opt,/usr,/var,/lib,/lib64"
export PATH="/packages/R/4.0.2/bin:/usr/lib/rstudio-server/bin:/packages/singularity/3.6.2/bin:$PATH"
export SINGULARITYENV_PATH="$PATH"
export R_LIBS_USER="~/R/%v"
export R_LIBS="/packages/R/%v"
# path needs for rgdal, sf
export LD_LIBRARY_PATH="/packages/geos/3.8.1/lib:/packages/PROJ/6.1.0/lib:/packages/gdal/3.0.4/lib:$LD_LIBRARY_PATH"
# In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH
# to the singularity process
export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export SINGULARITYENV_APPEND_PATH="/usr/lib/rstudio-server/bin:/packages/singularity/3.6.2/bin"
echo " [from script.sh.erb] Module to load: R/4.0.2"
module load R/4.0.2
module load anaconda3
echo " [from script.sh.erb] Loaded modules:"
module list
echo "*** which rsession" && which rsession
echo "*** PATH is" && echo "$PATH"
}
setup_env
#
# Start RStudio Server
#
# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"
# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
#!/usr/bin/env bash
# Log all output from this script
export RSESSION_LOG_FILE="${PWD}/rsession.log"
exec &>>"\${RSESSION_LOG_FILE}"
# Launch the original command
echo "**Which rsession"
which rsession
echo "Launching rsession..."
set -x
exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"
# Set working directory to home directory
cd "${HOME}"
export TMPDIR="$(mktemp -d)"
mkdir -p "$TMPDIR/rstudio-server"
python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"
set -x
# Launch the RStudio Server
echo "Starting up rserver..."
singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}"
echo 'Singularity as exited...'
Anyone have additional suggestions, please and thank you?
Here’s a grab bag of places they might have gone as I’ve changed directory targets for this-or-that, and there’s no trace of what’s going on, so far.