Rstudio-server Not Found

I’ve got two installations of rstudio on ood on two different clusters. One works as expected and one just gives me a 404 error on the auth-do-sign-in page. using ood 2.0.27 on the one that’s not working, 1.8.18 on the working one, rstudio-server-1.3.1093-1.x86_64 on both and a centos8 singularity image.

I can confirm via telnet from the ood server that rstudio-server is listening on the node and port that was assigned to it.

I suspected a problem with the reverse proxy but the netcat test works as expected. Any advice on what to try next?

Here’s the relevant parts of my script.sh.erb:

setup_env () {
export RSTUDIO_SERVER_IMAGE=“/bil/packages/RStudio/rstudio-server-c8.sif”
export SINGULARITY_BINDPATH=“/etc,/media,/mnt,/opt,/srv,/usr,/lib64,/lib,/bil,/var”
export SINGULARITYENV_LD_LIBRARY_PATH=“$LD_LIBRARY_PATH”
export PATH=“$PATH:/usr/lib/rstudio-server/bin:/bil/packages/R/R-3.6.3/bin:/bil/packages/anaconda3/4.9.2/envs/rstudio/bin/R”

}
setup_env

export RSTUDIO_AUTH=“${PWD}/bin/auth”

export RSESSION_WRAPPER_FILE=“${PWD}/rsession.sh”
(
umask 077
sed ‘s/^ {2}//’ > “${RSESSION_WRAPPER_FILE}” << EOL

export RSESSION_LOG_FILE=“${PWD}/rsession.log”

exec &>>“${RSESSION_LOG_FILE}”

echo “Launching rsession…”
set -x
exec rsession --r-libs-user “${R_LIBS_USER}” “${@}”
EOL
)
chmod 700 “${RSESSION_WRAPPER_FILE}”

cd “${HOME}”

export TMPDIR=“$(mktemp -d)”
export VARDIR=“$(mktemp -d)”

mkdir -p “$TMPDIR/rstudio-server”
echo “Started the server”
echo $(uuidgen) > “$TMPDIR/rstudio-server/secure-cookie-key”

chmod 0600 “$TMPDIR/rstudio-server/secure-cookie-key”

set -x
echo “Starting up rserver…”

singularity run -B “$TMPDIR:/tmp,$VARDIR:/var” “$RSTUDIO_SERVER_IMAGE”
–www-port “${port}”
–auth-none 0
–auth-pam-helper-path “${RSTUDIO_AUTH}”
–auth-encrypt-password 0
–server-working-dir “/tmp”
–rsession-path “${RSESSION_WRAPPER_FILE}”

This is the version of RStudio in both cases? There was an update to RStudio in terms of how to connect to it upwards of 1.3.

Yes, I deliberately used the version that was working on our other installation. I also tried the most recent version but I get user errors with that one so I was trying to eliminate other issues to make the two setups as similar as possible.

I’m starting to think there’s a problem with the /var setup, I realized it’s slightly different from the other one but it gives me errors (readonly fs) if I try to bind the real /var, while binding the system /var works on the 1.8.18 ood version.

I tried copying the readable contents of the system /var/lib/rstudio-server into $VARDIR and the server again starts up but I still get 404 on the auth-do-sign-in page.

Can you tell where the 404 is coming from? If you open your browsers dev tools and inspect the response you should see a server header that indicates where it’s from.

I mean is it from our apache or is it from RStudio itself? Maybe it’s a simple as you forgot to enable the /rnode Locations on the new instance and that’s where the 404s are coming from.

Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1k

grepping for rnode, I have this line commented out in etc/ood so I guess its the default:

config/ood_portal.yml:#rnode_uri: ‘/node’

I tried changing the value to ‘/rnode’ and restarting the server but got the same result.

and I have this line in the app regarding rnodes:
view.html.erb:

on the working instance it’s set to ‘/rnode’

Sorry it interpreted that as formatting, let me see if I can get the line from view.html.erb:

Preformatted text :<form action="/rnode/<%= host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">

OK - so if this is a new cluster do you need to change/add the host_regex for that cluster?

# ood_portal.yml
host_regex: ...

It’s set to the default right now.

Ok we changed the line in view.html.erb to node instead of rnode and now we get an error from RStudio instead of from Apache:

RStudio Server

The requested page was not found.

Please contact your system administrator for assistance, or click here to go back.

So the reverse proxy is getting the node to reply properly now, but perhaps the url for auth is still not formed right?

OK - keeping it as rnode is likely the correct thing. So this is a 404 not found from Apache and this is different 404 not found than the original issue yes?

I guess, I’m still trying to determine who’s gave you the original 404 - apache or RStudio?

Originally it was apache, just a generic page not found. Then we changed the view.html.erb to point to node instead of rnode and we get an error message from rstudio now.

I believe /rnode is the right one you want. Seems like you may have them backwards in your ood_portal?

You seemed to indicate above that you’ve flipped these? They should be rnode is rnode and node is node.

rnode_uri: '/rnode'
node_uri: '/node'

You have another OOD instance deployed. It does not seem like they both have the same configurations, do they?

I’ve confirmed that the configuration is the same now between our two installations in ood_portal.yml and it’s as you suggested rnode_uri: ‘/rnode’, node_uri: ‘/node’. However, in the working deployment we have the view.html.erb set to /rnode, but when I set the broken deployment that way it generates a 404 error from apache. When I set it to /node I get a page not found from Rstudio. It’s the same version of Rstudio and the same singularity image in both cases, so I’m puzzled as to why we get different behavior in these cases. Am I right in thinking you choose /rnode or /node based on what the application expects?

Also I’ve confirmed that it’s possible to connect with a browser directly to the port that is opened by rstudio so the server can generate pages but I can’t log in even if I save and use the password that’s generated for it (with my username). I can’t get the login screen to appear at all through the reverse proxy.

I’m still concerned a little about configs - especially when the behavior is backwards like that.

When you reconfigured ood_portal.yml did you bounce httpd? Those settings don’t take affect until you bounce httpd, so I want to be sure the configs you have have actually been applied.

Yes we did bounce the server. We’ve made a little progress now, we can get the rstudio loading screen with the loading whee. but then we get rstudio initialization error/unable to connect to service.

I am not sure how I can view any error output from the auth command?

Glad to see you making progress. If you have a helper script like this one - you can redirect output somewhere.

Ok perhaps that’s part of the issue here. We have a line to create that rsession.log but it’s never yet been created so we don’t see any output.