Singularity fail - TimeZone issue - RStudio issues. Anyone able to help?

Hi Folks, Followed install for Rstudio on the ood site and stuck with this. Any clues? Thx

ie.
FATAL: container creation failed: hook function for tag layer returns error: failed to create file /etc/localtime: open /etc/localtime: permission denied

Full output. Centos 7.7 servers. Thx

[bio@papr-res-compute206 6e889b76-2b03-4f8d-9171-dc455b12d92d]$ ./script.sh

==== AUTHENTICATING FOR org.freedesktop.timedate1.set-timezone ===

Authentication is required to set the system timezone.

Authenticating as: root

Password:

==== AUTHENTICATION COMPLETE ===

  • echo ‘Starting up rserver…’

Starting up rserver…

  • singularity run -B /tmp/tmp.22fFpuE6cu:/tmp /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg --www-port ‘’ --auth-none 0 --auth-pam-helper-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/6e889b76-2b03-4f8d-9171-dc455b12d92d/bin/auth --auth-encrypt-password 0 --rsession-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/6e889b76-2b03-4f8d-9171-dc455b12d92d/rsession.sh

FATAL: container creation failed: hook function for tag layer returns error: failed to create file /etc/localtime: open /etc/localtime: permission denied

  • echo ‘Singularity as exited…’

Singularity as exited…

Hi Folks,
Would really appreciate any insight or assistance with this issue. Has anyone else seen this? Unfortunately this is impacting our release timeline and I am at a loss on how to fix it.

I read somewhere that binding the host servers /etc/localtime on top at container startup my fix it but that did not help either .
Thx

With my flying fingers, I can see /etc/localtime in the image, viz:
[ric@i18n21 singularity_images]$ module load singularity
[ric@i18n21 singularity_images]$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[ric@i18n21 singularity_images]$ singularity --version
singularity version 3.5.3
[ric@i18n21 singularity_images]$ singularity exec rserver-launcher-centos7.simg /bin/bash
Singularity> ls -l /etc/localtime
-rw-r--r-- 2 root root 327 Sep 26 04:34 /etc/localtime
Singularity>
If you do the same thing with your image, is /etc/localtime visible, and world readable?

I’m so sorry for missing this!

Here are some relevant logs

VERBOSE [U=30961,P=19397]  getBindFlags()                Could not mount /etc/localtime as read-write: mounted read-only
DEBUG   [U=30961,P=19397]  mountGeneric()                Mounting /etc/localtime to /var/singularity/mnt/session/final/usr/share/zoneinfo/UTC

And here are my permissions, similar to Rics.

[johrstrom@owens-login02 shelf]$ singularity exec centos.sif ls -l /etc/localtime
lrwxrwxrwx 1 root root 25 Sep 30 21:16 /etc/localtime -> ../usr/share/zoneinfo/UTC
[johrstrom@owens-login02 shelf]$ singularity exec centos.sif ls -l /usr/share/zoneinfo/UTC
-rw-r--r-- 3 root root 3519 Sep 12 07:29 /usr/share/zoneinfo/UTC
[johrstrom@owens-login02 shelf]$ ls -l /etc/localtime 
lrwxrwxrwx 1 root root 38 Jul 20  2016 /etc/localtime -> ../usr/share/zoneinfo/America/New_York
[johrstrom@owens-login02 shelf]$ singularity --version
singularity version 3.5.2-1.el7
johrstrom@owens-login02 shelf]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

Somehow singularity knew to bind it as read only (maybe an update would help? We tend to run a fairly current version of singularity). I used --debug option to see that. I would say use the file command to be sure that you can open/read that file properly.

Though I see this whole entry above ‘Starting up rserver…’ with you authenticating as root and specifically you setting the timezone. What is that about? Or entire script.sh is essentially just module setup then the singularity run command. What are you doing with X11?

Of course you can try to just disable that mount so you can make forward progress, then circle back later.

Hi Azric,
Many thanks for getting back to me. To answer your question, yes, /etc/localtime has those permissions.

#>> singularity exec /config/binaries/singularity/containers_devel/rserver-laun
cher-centos7.simg /bin/bash

Singularity> ls -l /etc/localtime
-rw-r–r-- 2 root root 2197 Sep 26 21:34 /etc/localtime

  • I have now updated to Singularity v3.5.2.
  • I see no symlink to /usr/share/zoneinfo/… when I run the ls -l command. i.e. I see only a file:
    Singularity> ls -l /etc/localtime
    -rw-r–r-- 2 root root 2197 Sep 26 21:34 /etc/localtime

Singularity> cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Singularity> A cat of the /etc/localtime file reveals (last line) :
AEST-10AEDT,M10.1.0,M4.1.0/3

Singularity> stat /etc/localtime
File: ‘/etc/localtime’
Size: 2197 Blocks: 8 IO Block: 4096 regular file
Device: 802h/2050d Inode: 1914324 Links: 2
Access: (0644/-rw-r–r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-03-04 13:54:02.781566605 +1100
Modify: 2019-09-26 21:34:09.000000000 +1000
Change: 2019-11-19 13:53:06.189162281 +1100

Running Rstudio from within OOD logs the following:
cat “output.log”

Script starting…
Waiting for RStudio Server to open port 45478…

  • echo ‘Starting up rserver…’
    Starting up rserver…
  • singularity run -B /tmp/tmp.ujSIvTJhJe:/tmp /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg --www-port 45478 -
    -auth-none 0 --auth-pam-helper-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/e2fc0b08-15f7-418c-b16f-e88815
    048e3a/bin/auth --auth-encrypt-password 0 --rsession-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/e2fc0b08
    -15f7-418c-b16f-e88815048e3a/rsession.sh
    ESC[33mWARNING:ESC[0m Skipping mount /etc/localtime [binds]: /etc/localtime doesn’t exist in container
  • echo ‘Singularity as exited…’
    Singularity as exited…
    Timed out waiting for RStudio Server to open port 45478!
    Cleaning up…

Changes made to script.sh are as follows. nothing else has been changed.

Additional environment which could be moved into a module

Change these to suit

export RSTUDIO_SERVER_IMAGE="/config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg"
export SINGULARITY_BINDPATH="/etc,/media,/mnt,/opt,/srv,/usr,/var"
export PATH="$PATH:/usr/lib/rstudio-server/bin"
export SINGULARITYENV_PATH="$PATH"

In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH

to the singularity process

export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
module load singularity/3.5.2
module load R/3.6.0

Hmmm… I will try the --debug option…

Additionally, Here is the Output.log when running in debug mode. Apologies in advance for the lengthy output.

cat output.log
Script starting…
Waiting for RStudio Server to open port 17482…

  • echo ‘Starting up rserver…’
    Starting up rserver…
  • singularity --debug run -B /tmp/tmp.nRK3Tl33nd:/tmp /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg --www-port
    17482 --auth-none 0 --auth-pam-helper-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/489e7b48-814a-4cf9-83a
    6-526c41bc02b0/bin/auth --auth-encrypt-password 0 --rsession-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/
    489e7b48-814a-4cf9-83a6-526c41bc02b0/rsession.sh
    DEBUG [U=1050,P=2465070] persistentPreRunE() Singularity version: 3.5.2
    DEBUG [U=1050,P=2465070] handleConfDir() /home/bioinf/.singularity already exists. Not creating.
    DEBUG [U=1050,P=2465070] setValue() Updated flag ‘bind’ value to: [/tmp/tmp.nRK3Tl33nd:/tmp,/etc,/media,/mnt,/opt,/srv,
    /usr,/var]
    DEBUG [U=1050,P=2465070] setValue() Updated flag ‘tmpdir’ value to: /config/binaries/singularity/tmp
    DEBUG [U=1050,P=2465070] getCacheBasedir() environment variable SINGULARITY_CACHEDIR not set, using default image cache
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/library
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oci-tmp
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oci
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/net
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/shub
    DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oras
    DEBUG [U=1050,P=2465070] execStarter() Checking for encrypted system partition
    DEBUG [U=1050,P=2465070] Init() Image format detection
    DEBUG [U=1050,P=2465070] Init() Check for sandbox image format
    DEBUG [U=1050,P=2465070] Init() sandbox format initializer returned: not a directory image
    DEBUG [U=1050,P=2465070] Init() Check for sif image format
    DEBUG [U=1050,P=2465070] Init() sif format initializer returned: SIF magic not found
    DEBUG [U=1050,P=2465070] Init() Check for squashfs image format
    DEBUG [U=1050,P=2465070] Init() squashfs image format detected
    VERBOSE [U=1050,P=2465070] SetContainerEnv() Not forwarding SINGULARITY_TMPDIR from user to container environment
    VERBOSE [U=1050,P=2465070] SetContainerEnv() Not forwarding SINGULARITY_BINDPATH from user to container environment
    VERBOSE [U=1050,P=2465070] SetContainerEnv() HOME=/home/bioinf
    DEBUG [U=1050,P=2465070] init() Use starter binary /config/binaries/singularity/3.5.2/libexec/singularity/bin/start
    er-suid
    VERBOSE [U=0,P=2465070] print() Set messagelevel to: 5
    VERBOSE [U=0,P=2465070] init() Starter initialization
    DEBUG [U=1050,P=2465079] startup() singularity runtime engine selected
    DEBUG [U=0,P=2465070] load_overlay_module() Trying to load overlay kernel module
    DEBUG [U=0,P=2465070] load_overlay_module() Overlay seems supported by the kernel
    DEBUG [U=0,P=2465070] get_pipe_exec_fd() PIPE_EXEC_FD value: 7
    VERBOSE [U=0,P=2465070] is_suid() Check if we are running as setuid
    VERBOSE [U=0,P=2465070] priv_drop() Drop root privileges
    DEBUG [U=1050,P=2465070] init() Read engine configuration
    DEBUG [U=1050,P=2465070] init() Wait completion of stage1
    VERBOSE [U=1050,P=2465079] priv_drop() Drop root privileges permanently
    DEBUG [U=1050,P=2465079] set_parent_death_signal() Set parent death signal to 9
    VERBOSE [U=1050,P=2465079] init() Spawn stage 1
    DEBUG [U=1050,P=2465079] startup() singularity runtime engine selected
    VERBOSE [U=1050,P=2465079] startup() Execute stage 1
    DEBUG [U=1050,P=2465079] StageOne() Entering stage 1
    DEBUG [U=1050,P=2465079] prepareAutofs() Found “/proc/sys/fs/binfmt_misc” as autofs mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /tmp/tmp.nRK3Tl33nd: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /etc: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /media: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /mnt: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /opt: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /srv: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /usr: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /var: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for home directory /home/bioinf: no mount point
    DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for current working directory /home/bioinf: no mount point
    DEBUG [U=1050,P=2465079] Init() Image format detection
    DEBUG [U=1050,P=2465079] Init() Check for sandbox image format
    DEBUG [U=1050,P=2465079] Init() sandbox format initializer returned: not a directory image
    DEBUG [U=1050,P=2465079] Init() Check for sif image format
    DEBUG [U=1050,P=2465079] Init() sif format initializer returned: SIF magic not found
    DEBUG [U=1050,P=2465079] Init() Check for squashfs image format
    DEBUG [U=1050,P=2465079] Init() squashfs image format detected
    DEBUG [U=1050,P=2465079] setSessionLayer() Overlay seems supported and allowed by kernel
    DEBUG [U=1050,P=2465079] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
    VERBOSE [U=1050,P=2465070] wait_child() stage 1 exited with status 0
    DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 4
    DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 5
    DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 6
    DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 7
    DEBUG [U=1050,P=2465070] init() Set child signal mask
    DEBUG [U=1050,P=2465070] init() Create socketpair for master communication channel
    DEBUG [U=1050,P=2465070] init() Create RPC socketpair for communication between stage 2 and RPC server
    VERBOSE [U=1050,P=2465070] priv_escalate() Get root privileges
    VERBOSE [U=0,P=2465070] priv_escalate() Change filesystem uid to 1050
    VERBOSE [U=0,P=2465070] init() Spawn master process
    DEBUG [U=0,P=2465104] set_parent_death_signal() Set parent death signal to 9
    VERBOSE [U=0,P=2465104] create_namespace() Create mount namespace
    VERBOSE [U=0,P=2465070] enter_namespace() Entering in mount namespace
    DEBUG [U=0,P=2465070] enter_namespace() Opening namespace file ns/mnt
    DEBUG [U=0,P=2465070] setns() Using syscall() wrapped __NR_setns
    VERBOSE [U=0,P=2465070] priv_drop() Drop root privileges
    VERBOSE [U=0,P=2465104] create_namespace() Create mount namespace
    DEBUG [U=0,P=2465105] set_parent_death_signal() Set parent death signal to 9
    VERBOSE [U=0,P=2465105] init() Spawn RPC server
    DEBUG [U=1050,P=2465070] startup() singularity runtime engine selected
    VERBOSE [U=1050,P=2465070] startup() Execute master process
    DEBUG [U=0,P=2465105] startup() singularity runtime engine selected
    VERBOSE [U=0,P=2465105] startup() Serve RPC requests
    DEBUG [U=1050,P=2465070] setupSessionLayout() Using Layer system: overlay
    DEBUG [U=1050,P=2465070] setupOverlayLayout() Creating overlay SESSIONDIR layout
    DEBUG [U=1050,P=2465070] addRootfsMount() Mount rootfs in read-only mode
    DEBUG [U=1050,P=2465070] addRootfsMount() Image type is 4096
    DEBUG [U=1050,P=2465070] addRootfsMount() Mounting block [squashfs] image: /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg
    DEBUG [U=1050,P=2465070] addKernelMount() Checking configuration file for ‘mount proc’
    DEBUG [U=1050,P=2465070] addKernelMount() Adding proc to mount list
    VERBOSE [U=1050,P=2465070] addKernelMount() Default mount: /proc:/proc
    DEBUG [U=1050,P=2465070] addKernelMount() Checking configuration file for ‘mount sys’
    DEBUG [U=1050,P=2465070] addKernelMount() Adding sysfs to mount list
    VERBOSE [U=1050,P=2465070] addKernelMount() Default mount: /sys:/sys
    DEBUG [U=1050,P=2465070] addDevMount() Checking configuration file for ‘mount dev’
    DEBUG [U=1050,P=2465070] addDevMount() Adding dev to mount list
    VERBOSE [U=1050,P=2465070] addDevMount() Default mount: /dev:/dev
    DEBUG [U=1050,P=2465070] addHostMount() Not mounting host file systems per configuration
    VERBOSE [U=1050,P=2465070] addBindsMount() Found ‘bind path’ = /etc/localtime, /etc/localtime
    VERBOSE [U=1050,P=2465070] addBindsMount() Found ‘bind path’ = /etc/hosts, /etc/hosts
    DEBUG [U=1050,P=2465070] addHomeStagingDir() Staging home directory (/home/bioinf) at /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
    DEBUG [U=1050,P=2465070] addHomeMount() Adding home directory mount [/config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf:/home/bioinf] to list using layer: overlay
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /tmp/tmp.nRK3Tl33nd to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /etc to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /media to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /mnt to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /opt to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /srv to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /usr to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /var to mount list
    DEBUG [U=1050,P=2465070] addUserbindsMount() Checking for ‘user bind control’ in configuration file
    DEBUG [U=1050,P=2465070] addTmpMount() Checking for ‘mount tmp’ in configuration file
    VERBOSE [U=1050,P=2465070] addTmpMount() Default mount: /tmp:/tmp
    VERBOSE [U=1050,P=2465070] addTmpMount() Default mount: /var/tmp:/var/tmp
    DEBUG [U=1050,P=2465070] addScratchMount() Not mounting scratch directory: Not requested
    DEBUG [U=1050,P=2465070] addCwdMount() Using /home/bioinf as current working directory
    VERBOSE [U=1050,P=2465070] addCwdMount() Default mount: /home/bioinf: to the container
    DEBUG [U=1050,P=2465070] addLibsMount() Checking for ‘user bind control’ in configuration file
    DEBUG [U=1050,P=2465070] addFilesMount() Checking for ‘user bind control’ in configuration file
    DEBUG [U=1050,P=2465070] addResolvConfMount() Adding /etc/resolv.conf to mount list
    VERBOSE [U=1050,P=2465070] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
    DEBUG [U=1050,P=2465070] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
    DEBUG [U=1050,P=2465070] create() Mount all
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting tmpfs to /config/binaries/singularity/3.5.2/var/singularity/mnt/session
    DEBUG [U=1050,P=2465070] mountImage() Mounting loop device /dev/loop0 to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs of type squashfs
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting overlay to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
    DEBUG [U=1050,P=2465070] setPropagationMount() Set RPC mount propagation flag to SLAVE
    VERBOSE [U=1050,P=2465070] Passwd() Checking for template passwd file: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
    VERBOSE [U=1050,P=2465070] Passwd() Creating passwd content
    VERBOSE [U=1050,P=2465070] Passwd() Creating template passwd file and appending user data: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
    DEBUG [U=1050,P=2465070] addIdentityMount() Adding /etc/passwd to mount list
    VERBOSE [U=1050,P=2465070] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
    VERBOSE [U=1050,P=2465070] Group() Checking for template group file: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/group
    VERBOSE [U=1050,P=2465070] Group() Creating group content
    DEBUG [U=1050,P=2465070] addIdentityMount() Adding /etc/group to mount list
    VERBOSE [U=1050,P=2465070] addIdentityMount() Default mount: /etc/group:/etc/group
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /dev to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/dev
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc/localtime to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/localtime
    WARNING [U=1050,P=2465070] mountGeneric() Skipping mount /etc/localtime [binds]: /etc/localtime doesn’t exist in container
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc/hosts to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/hosts
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/etc/singularity/actions to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /proc to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/proc
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/proc
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting sysfs to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/sys
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /home/bioinf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/home/bioinf
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /tmp to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /var/tmp to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
    VERBOSE [U=1050,P=2465070] mountGeneric() Skipping mount /home/bioinf, /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/home/bioinf already mounted
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/resolv.conf
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/passwd to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/passwd
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/group to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/group
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /tmp/tmp.nRK3Tl33nd to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /media to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/media
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/media
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /mnt to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/mnt
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/mnt
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /opt to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/opt
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/opt
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /srv to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/srv
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/srv
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /usr to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/usr
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/usr
    DEBUG [U=1050,P=2465070] mountGeneric() Mounting /var to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var
    DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var
    DEBUG [U=1050,P=2465070] create() Chroot into /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
    DEBUG [U=0,P=2465105] Chroot() Hold reference to host / directory
    DEBUG [U=0,P=2465105] Chroot() Called pivot_root on /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
    DEBUG [U=0,P=2465105] Chroot() Change current directory to host / directory
    DEBUG [U=0,P=2465105] Chroot() Apply slave mount propagation for host / directory
    DEBUG [U=0,P=2465105] Chroot() Called unmount(/, syscall.MNT_DETACH)
    DEBUG [U=0,P=2465105] Chroot() Changing directory to / to avoid getpwd issues
    DEBUG [U=1050,P=2465070] create() Chdir into / to avoid errors
    VERBOSE [U=0,P=2465104] wait_child() rpc server exited with status 0
    DEBUG [U=0,P=2465104] apply_container_privileges() Set user ID to 1050
    DEBUG [U=1050,P=2465104] set_parent_death_signal() Set parent death signal to 9
    DEBUG [U=1050,P=2465104] startup() singularity runtime engine selected
    VERBOSE [U=1050,P=2465104] startup() Execute stage 2
    DEBUG [U=1050,P=2465104] StageTwo() Entering stage 2
    DEBUG [U=1050,P=2465070] PostStartProcess() Post start process
    DEBUG [U=1050,P=2465070] Master() Child exited with exit status 1
  • echo ‘Singularity as exited…’
    Singularity as exited…
    Timed out waiting for RStudio Server to open port 17482!
    Cleaning up…

OK at this point, you may have something wrong with RStudio. Looks like singularity is doing it’s job fine after the update. localtime still seems to be wrong somehow, but it doesn’t seem fatal anymore.

There could also be an rsession.log in that directory that may indicate why RStudio is failing.

These options in /etc/singularity/singularity.conf may be relevant to you for properly mounting localtime. Maybe one or more is currently ‘no’?

# ENABLE OVERLAY: [yes/no/try]
# DEFAULT: try
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container.  If 'try' is chosen,
# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = try

# ENABLE UNDERLAY: [yes/no]
# DEFAULT: yes
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container even if overlay is not
# working.  If overlay is available, it will be tried first.
enable underlay = yes

I highly recommend replacing the OSC’s based RStudio Singularity image with that from rocker. We started with the OSC’s RStudio container but soon users started complaining that they can’t install the R libraries that they need.

Rocker has several different Docker Hub containers that all have RStudio Server, have different libraries pre-installed and work out of the box in our OOD instance. All one needs to do is to “singularity build” a Singularity sif container based on the Docker Hub image. Similarly we use a bioconductor Docker Hub container, which is based on rocker, for bio R stuff, as this one has all the system dependencies like libgdal installed in the container so users don’t have to struggle when building R libraries that have system dependencies. Our approach is to provide these base R containers with the system dependencies and have users to install their own libraries to their own disk space.

Our github repo that should cover this is at GitHub - CHPC-UofU/bc_osc_rstudio_server: Batch Connect - OSC RStudio Server. I would recommend to build one of the rocker containers with Singularity, see what you get in the /etc/localtime, I get this inside of the container:
$ singularity shell ood-rstudio-rocker_3.6.2.sif
Singularity> ls -l /etc/localtime
lrwxrwxrwx. 1 root root 27 Dec 23 17:00 /etc/localtime → /usr/share/zoneinfo/Etc/UTC

as opposed to when being on the host:
$ ls -l /etc/localtime
lrwxrwxrwx. 1 root root 36 Oct 24 2017 /etc/localtime → …/usr/share/zoneinfo/America/Denver

Being kind of doubtful on the Etc/UTC zone, whatever it does, time shows up right in the RStudio:

Sys.time()
[1] “2020-03-06 14:51:47 MST”
Sys.timezone()
[1] “Etc/UTC”

HTH,
MC

Hi Folks, Thanks everone for helping out and apologies for not getting back to you. Peter MaCallum Cancer Center as with many other organizations really are flat out because of the Virus issues.

Singularity timezone issue - I have not figured out but will look into the idea mcuma had.
As for Rstudio not working, this was my bad, as I did not bind the modules filesystem to the container, hence Rstudio server could not load the R modules and therefore failed.

I am now tonight looking at why PAM auth is failing (we use SSSD/IPA) for node auth.

Finally Congrats to the OOD team for such a great product! We will likely make this avail to all our staff who are now working from home (no pressure on me right?). I just hve to get it working with Auth now and quickly. :stuck_out_tongue:

1 Like

Got the PAM auth running, PAM was not enabled in IPA at our site. Sorted. Basics work Want to go soft live but Rstudio params I apply to the submit and form files for RStudio cause an error. I think I will open a new thread for this as I also have some questions from our test users that I cannot answer but yuo can.