Looking for some debugging tips re: rstudio app on el8

Get Help
6

Alright, I’ve done a couple things, and made a little progress.
However, I basically know nothing about R/rserver/rsession, and I think that is massively impacting my ability to help-me-help-myself. …but I’ll get back to that, shortly. But for now I apologize for missing things that are probably very obvious to anyone familiar with that software.

It wasn’t (yet) impacting anything, but I created an el8 container using the suggested definition just the same. (For anyone following behind me, to bootstrap Centos 8, I had to mod MirrorURL to be: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/BaseOS/$basearch/os since the repo is structured differently.)

For anyone’s reference, let’s start with this…

template/script.sh.erb

module load singularity anaconda3 R
export SINGULARITY_BINDPATH="/etc,/media,/mnt,/opt,/srv,/usr,/var,/scratch,/o-scratch,/common,/packages"
export SINGULARITYENV_PATH="$PATH"
export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export RSTUDIO_SERVER_IMAGE="/common/contrib/containers/rserver-launcher-centos8-jb.simg"
## <snip!> 
singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --server-data-dir "$PWD/rstudio-server" \     ## I'LL DISCUSS THESE
 --server-pid-file "$PWD/rstudio-server.pid" \ ## TWO LINES, BELOW!
 --rsession-path "${RSESSION_WRAPPER_FILE}"

…but note that the --server-* lines were added after the first bit of testing. Exactly when, I’ll specifically mention below.

I figured since I didn’t feel confident in messing with the singularity/rserver parameters, the next best thing would be to start with the script hinted above (again, without those server lines)…

  • start an rstudio app session, letting it create the targeted files for me
  • (it’ll time-out and die, and that’s fine)
  • note the business-end of the singularity line from the output.log:

  • singularity run -B /tmp/jtb49/35145515/tmp.QIZSlsyJP7:/tmp /common/contrib/containers/rserver-launcher-centos8-jb.simg –www-port 23631 --auth-none 0 --auth-pam-helper-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/6e17c3c2-ebee-4f74-82b9-832b8adeb40c/bin/auth --auth-encrypt-password 0 --rsession-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/6e17c3c2-ebee-4f74-82b9-832b8adeb40c/rsession.sh

  • ssh into the compute-node
  • load the necessary modules & export variables
  • shell into the container and (mostly-) recycle that singularity line…

$ module load singularity anaconda3 R
$ export SINGULARITY_BINDPATH="/etc,/media,/mnt,/opt,/srv,/usr,/var,/scratch,/o-scratch,/common,/packages"
$ export SINGULARITYENV_PATH="$PATH"
$ export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
$ singularity shell /common/contrib/containers/rserver-launcher-centos8-jb.simg
Singularity> rserver --www-port 23631 --auth-none 0 --auth-pam-helper-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/6e17c3c2-ebee-4f74-82b9-832b8adeb40c/bin/auth --auth-encrypt-password 0 --rsession-path /home/jtb49/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_dev/output/6e17c3c2-ebee-4f74-82b9-832b8adeb40c/rsession.sh

24 Nov 2020 23:37:10 [rserver] ERROR system error 30 (Read-only file system) [path: /var/run/rstudio-server, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:648; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:555

I guess it turns out rserver was angry that it wasn’t allowed to write to /var/run. I don’t recall my exact thought process but since I also am leveraging RStudio 1.3.1093, I ended up trying the “–server-data-dir” parameter mentioned in this post: RStudio when launched without Singularity is having strange troubles with authentication - #14 by charles8ronson …and that allowed the server to start. Yay! I looked around for logs in a number of places and didn’t see them, can someone tell me where they are and if maybe some binding is screwing them up?

Long story short: I added those --server lines to the singularity line in the app’s script.sh.erb. (Only the data-dir made a difference, but both were mentioned in that other thread, so I figured I’d try both too.) The rstudio app session starts and I was greeted with the blue connect button – BUT after clicking that button, I’m sent to a URL that yields a 404 error:

https://CORRECTHOST/rnode/cn104/37628/auth-do-sign-in

Not Found
The requested URL /rnode/cn104/37628/auth-do-sign-in was not found on this server.

So I got some forward progress, and that’s cool. But, I’ve been trying to figure this one out for like [too many] hours now, and am getting nowhere. :confused: Anyone have additional suggestions, please and thank you?

UPDATE :slight_smile:

Apparently the hostname command performs slightly differently between el6/el8, because after I sat down to re-read through my post to make sure I didn’t booboo, I realized the “host-part” of the URL didn’t match that of my other instance, and was missing the host’s fqdn:

https://CORRECTHOST/**rnode**/cn104/37628/auth-do-sign-in

vs.

https://CORRECTHOST/**rnode.local**/cn104/37628/auth-do-sign-in

Boom, baby!

So now I am served a legit page, but with that familiar error from that earlier thread, “Error: Incorrect or invalid username/password”. So at least now I’m in a better position to proceed through that thread, but if anyone can point me to rserver’s logging, I’d appreciate it! Thanks!