Osc_bc_rstudio_server streamlined submit.yml.erb

Get Help
1

Greetings – I first implemented the RStudio app (osc_bc_rstudio_server) a few years ago. Since then, the code offered in the example features much more ruby, handling details of the OSC resources. Which is of course nice!

For a simple person like me, however, all the optional conditions make it difficult to get to the core of code necessary to launch rstudio server using this server launching container. Does there exist a minimalist version of the code, including the submit script, but also the template/script.sh.erb to help emphasize the basics?

Thanks

2

I don’t know how well this works, but it could be it.

Of course it probably doesn’t have all the updates for newer versions of RStudio. This changelog may help - that’s what I had to do to upgrade. It’s mostly centered around container mount points to configure the rstudio server.

3

Thanks for sharing, Jeff.

My local implementation is currently running into a previously reported problem, which is the:

Timed out waiting for RStudio Server to open port 54321!

Funny that this time it’s reporting port “5” “4” “3” “2” “1”… ( :slight_smile:

in any event, i’m stuck here without being able to resolve this, struggling for an embarrassingly long time period (yes, avoidance has contributed to the length of time) and finding myself reluctant to ask for help. Perhaps next week we can consult?

Cheers – enjoy your weekend

4

The next 2 weeks are busy for us hosting summer institute and vacations.

Are you trying to upgrade/update an existing app that does work? The Rstudio version is important here. 1.3 and beyond for them required changes from us. That’s what the git diff above was all about - us upgrading from 1.2 to 1.3+.

One trick I’ve learned -

  • start a job and it’ll fail you know
  • navigate to it’s output directory
  • edit and run script.sh. You probably need to hard code export port=54321
  • rinse and repeat. editing and running script.sh means you’re not backgrounding the process. You can actually see what it’s standard out is - if it has any and it’ll give you a much quicker feedback cycle of edits and tries.

One really important bit is the log directory and the fact that because we’re mounting the /etc config directory here - we can change the log directory.

This is actually a much smaller commit that I used to upgrade from 1.2 to 1.3+. Though it had bugs in it as well (again, mostly additional mount points).

5

Hi, Jeff – Could you explain a little more about choosing an appropriate $SINGULARITY_HOME value? If one does not have specialized rstudio versions, in particular, but it would be nice to hear a full discussion? A short version here for the “simple” case would be perhaps more appropriate for this thread, of cource.

Also, i’m running into trouble with a conventional error:
error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory

the library is installed on the compute nodes, and the LD_LIBRARY_PATH contains ‘/usr/lib64’, the location under RHEL 8.
I also verify that SINGULARITY_LD_LIBRARY_PATH mirrors system LD_LIBRARY_PATH. I also ensure that ‘/usr/lib64’ appears at the start of the path list in LD_LIBRARY_PATH, since the path is long (we have adopted EasyBuild, if anyone is familiar - lot’s of discrete modules supporting RStudio-Server module).

6

SINGULARITY_HOME is like the containers PWD or CWD. The current directory when all starts up. We use it to shift from ~ to say ~/some_classroom to give a ‘specialized environment’ (i.e., not all the clutter in your HOME).

To be clear it should be: SINGULARITYENV_LD_LIBRARY_PATH (singularityenv vs singularity).

One thing to note is is whether you’re mounting /var/ from the host into the container. I don’t know where this file is in say centos 7, but it may have changed from 7 (the container’s OS?) to 8 the Hosts OS.

7

Thanks, Jeff – and thanks also for catching the typo in this thread. I verify using SINGULARITYENV_LD_LIBRARY_PATH.

We tried installing the i686 version of pam (/usr/lib) to supplement the x86_64 version (/usr/lib64)… No joy in resolving:
error while loading shared libraries: libpam.so.0: cannot open shared object file: No such file or directory

I would think that pam is running at the system level, but is that wrong? Does the path to /usr/lib64/libpam.so.0 need to binding into the container environment?

8

RStudio itself is looking for that so. You could bind it in from the host or install it as a part of the def file.

I just spun a rocky:8 image and needed to install pam for these so files to show up, so my guess is, it’s not there.

[root@3262505b2c94 /]# find /usr/ -name libpam.so*
/usr/lib64/libpam.so.0.84.2
/usr/lib64/libpam.so.0
9

My rserver launches fine as a ‘naked’ application, running on the server without the rserver launcher. So to be clear, the idea is that pam needs installed within the container?

10

Thanks, Jeff – That cleared it up. I switched to a container version having pam installed, and no more problem. Appreciated.

Now, I’m still poised on the edge of success, as my session times out waiting for ‘RStudio server to open the port xxxxx’

11

What generates the port number for the RStudio session? What is the usual range of ports that are used? I’m having trouble finding this information – seems that I once saw it in the docs, so sorry to have to ask for this type of help.

More directly, what ports need opened between compute nodes and the ondemand host?

12

Our latest implementation is on a server running RHEL8 and firewalld. So it looks like we need to better prepare the server to support reverse proxy from the compute nodes.

13

Ports are randomly generated (though I want to say the rand command). You can specify the port range here.

https://osc.github.io/ood-documentation/latest/reference/files/submit-yml/basic-bc-options.html

Which you can set globally for that entire cluster.

14

Thanks, Jeff. I’ve used this to restrict the range to 8787, which I know works if I launch rserver directly, and use ‘http://localhost:8787’ through a local browser instance to connect.

In terms of the reverse proxy from a compute, through the ondemand server, to my remote browser session, still failing on with the “Timed Out waiting for RStudio Server to open port 8787” ( i am still uncertain why this app relies on 'template: “basic” ’ in the submit.yml.erb)

Are there flags to use to force the issue when launching rserver? I’m afraid that I’ve spiraled into that phase space of “Just trying @#$^”, hoping that something works, without really understanding how the parts should work together. It’s not my (nor, I assume, anyone else’s) happy place. sigh.

15

As rgas20 reported last October, the process does not get far enough to generate an rsession.log

Also, I’ve been slow to recognize that the singularity session ends immediately – so nothing really runs on the node. The time out is then a false lead. Time to regroup.

16

This is the pattern of testing I’ve found works well. Instead of going through the whole flow, just focus on getting script.sh to work in some job’s directory. Hack on that to get

This is the droid you’re looking for. Again, the mount points are kinda critical, for functionality sure, but getting this log rserver.log outside of the container is going to help you 1000% percent. It’s likely already inside your container in say /var/rstudio/rserver.log but with that etc mount point + the etc configuration file you can put that log in the working directory of the job.

image
image1438×652 47.1 KB

17

Thanks, Jeff, for sticking with me ( :

I appreciate the suggestion to work with the script directly – however, I think I’m far from having a handle on what it should do. I’m gonna have to start over from the script, and try to tailor to the script to our env.
https://raw.githubusercontent.com/OSC/bc_osc_rstudio_server/master/template/script.sh.erb

Another suggestion that you seem to make is that rsession.log is written internal to the container, and some “reverse binding” is necessary to bring it back to the host file system. Another point for me to study offline.

There is one operational problem I ran into: too many positional options have been specified on the command line
Starting up rserver… at Wed Jun 8 16:24:09 EDT 2022

  • singularity run --contain -B /tmp/job.10922.hpc/tmp.FUVJ23Uk3F:/tmp -B /home/mrd20/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_Pioneer/output/deaac840-8654-4382-9ea5-43768ab9c522 -B /home/mrd20/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_Pioneer/output/deaac840-8654-4382-9ea5-43768ab9c522:/etc/rstudio -B /usr/local/easybuild/software/RStudio-Server /home/mrd20/hpcdemo/rstudio/rstudio-launcher-centos-8.sif rserver --www-port 8080 --auth-none 0 --auth-pam-helper-path /home/mrd20/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_Pioneer/output/deaac840-8654-4382-9ea5-43768ab9c522/bin/auth --auth-encrypt-password 0 --rsession-path /home/mrd20/ondemand/data/sys/dashboard/batch_connect/dev/RStudio_Pioneer/output/deaac840-8654-4382-9ea5-43768ab9c522/rsession.sh
    too many positional options have been specified on the command line

It seems that this is no more than called for – do you recognize the root cause of this error?

Thanks again

18

Not directly, but I’m sure it’s something very simple/silly like an extra space somewhere. I can’t replicate but googling seems to indicate it’s a space issue. That is, there’s an extra space somewhere in that command.

19

Thanks, Jeff –

We have been using EasyBuild for some time here at CWRU. As part of learning to launch rserver from that module, there is guidance to pull from the ‘Sanity Check’ code. That includes:
export TMPDIR="$(mktemp -d)"
echo -e “provider=sqlite\ndirectory=$TMPDIR/db” >> $TMPDIR/db.conf

mkdir -p “$TMPDIR/rstudio-server”
python -c ‘from uuid import uuid4; print(uuid4())’ > “$TMPDIR/rstudio-server/secure-cookie-key”
chmod 0600 “$TMPDIR/rstudio-server/secure-cookie-key”

set -x

Launch the RStudio Server

echo “Starting up rserver…”

rserver --server-daemonize=0 --database-config-file $TMPDIR/db.conf --server-data-dir $TMPDIR/sdd --www-port 8787 --rsession-which-r=$(which R)

When I embed the ‘sanity check’ code into a stripped down script (based on the app implemented here 3 years ago), the rserver will start, and work fine on a local compute node.

I’m continuing to attempt to learn.
Best regards, and thanks for monitoring my situation.

20

Returning to the Singularity-launcher approach, and editing script.sh in the job directory, I encounter the ‘Read-only’ filesystem error that has been reported previously:

09 Jun 2022 15:23:19 [rserver] ERROR system error 30 (Read-only file system) [path: /var/lib/rstudio-server, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:812; LOGGED FROM: int main(int, char const) src/cpp/server/ServerMain.cpp:694**

The code follows. I’ve tried setting the --server-data-dir, which helped me with the EasyBuild sanity check, and was reported by Jason in 2020 as resolving the issue (there was a subsequent auth issue to address in that thread).

I’ve tried variations on the --server-data-dir theme. I appreciate insights/suggestions.

99 singularity run
100 -B “$TMPDIR:/tmp”
101 -B “$WORKING_DIR”
102 -B “$WORKING_DIR/var:/var/lib/rstudio-server”
103 -B /home
104 -B /scratch
105 -B /usr/local
106 “$RSTUDIO_SERVER_IMAGE”
107 --www-port “${port}”
108 --auth-none 0
109 --auth-pam-helper-path “${RSTUDIO_AUTH}”
110 --auth-encrypt-password 0
111 --database-config-file “$TMPDIR/db.conf”
112 --server-data-dir “$TMPDIR/rstudio-server”
113 --server-user “$USER”
114 --rsession-path “${RSESSION_WRAPPER_FILE}”