Hi Folks, Followed install for Rstudio on the ood site and stuck with this. Any clues? Thx
ie.
FATAL: container creation failed: hook function for tag layer returns error: failed to create file /etc/localtime: open /etc/localtime: permission denied
FATAL: container creation failed: hook function for tag layer returns error: failed to create file /etc/localtime: open /etc/localtime: permission denied
Hi Folks,
Would really appreciate any insight or assistance with this issue. Has anyone else seen this? Unfortunately this is impacting our release timeline and I am at a loss on how to fix it.
I read somewhere that binding the host servers /etc/localtime on top at container startup my fix it but that did not help either .
Thx
With my flying fingers, I can see /etc/localtime in the image, viz: [ric@i18n21 singularity_images]$ module load singularity [ric@i18n21 singularity_images]$ cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [ric@i18n21 singularity_images]$ singularity --version singularity version 3.5.3 [ric@i18n21 singularity_images]$ singularity exec rserver-launcher-centos7.simg /bin/bash Singularity> ls -l /etc/localtime -rw-r--r-- 2 root root 327 Sep 26 04:34 /etc/localtime Singularity>
If you do the same thing with your image, is /etc/localtime visible, and world readable?
VERBOSE [U=30961,P=19397] getBindFlags() Could not mount /etc/localtime as read-write: mounted read-only
DEBUG [U=30961,P=19397] mountGeneric() Mounting /etc/localtime to /var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
And here are my permissions, similar to Rics.
[johrstrom@owens-login02 shelf]$ singularity exec centos.sif ls -l /etc/localtime
lrwxrwxrwx 1 root root 25 Sep 30 21:16 /etc/localtime -> ../usr/share/zoneinfo/UTC
[johrstrom@owens-login02 shelf]$ singularity exec centos.sif ls -l /usr/share/zoneinfo/UTC
-rw-r--r-- 3 root root 3519 Sep 12 07:29 /usr/share/zoneinfo/UTC
[johrstrom@owens-login02 shelf]$ ls -l /etc/localtime
lrwxrwxrwx 1 root root 38 Jul 20 2016 /etc/localtime -> ../usr/share/zoneinfo/America/New_York
[johrstrom@owens-login02 shelf]$ singularity --version
singularity version 3.5.2-1.el7
johrstrom@owens-login02 shelf]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Somehow singularity knew to bind it as read only (maybe an update would help? We tend to run a fairly current version of singularity). I used --debug option to see that. I would say use the file command to be sure that you can open/read that file properly.
Though I see this whole entry above ‘Starting up rserver…’ with you authenticating as root and specifically you setting the timezone. What is that about? Or entire script.sh is essentially just module setup then the singularity run command. What are you doing with X11?
Of course you can try to just disable that mount so you can make forward progress, then circle back later.
I see no symlink to /usr/share/zoneinfo/… when I run the ls -l command. i.e. I see only a file:
Singularity> ls -l /etc/localtime
-rw-r–r-- 2 root root 2197 Sep 26 21:34 /etc/localtime
Singularity> cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
Singularity> A cat of the /etc/localtime file reveals (last line) :
AEST-10AEDT,M10.1.0,M4.1.0/3
Additionally, Here is the Output.log when running in debug mode. Apologies in advance for the lengthy output.
cat output.log
Script starting…
Waiting for RStudio Server to open port 17482…
echo ‘Starting up rserver…’
Starting up rserver…
singularity --debug run -B /tmp/tmp.nRK3Tl33nd:/tmp /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg --www-port
17482 --auth-none 0 --auth-pam-helper-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/489e7b48-814a-4cf9-83a
6-526c41bc02b0/bin/auth --auth-encrypt-password 0 --rsession-path /home/bioinf/ondemand/data/sys/dashboard/batch_connect/dev/RStudio/output/
489e7b48-814a-4cf9-83a6-526c41bc02b0/rsession.sh
DEBUG [U=1050,P=2465070] persistentPreRunE() Singularity version: 3.5.2
DEBUG [U=1050,P=2465070] handleConfDir() /home/bioinf/.singularity already exists. Not creating.
DEBUG [U=1050,P=2465070] setValue() Updated flag ‘bind’ value to: [/tmp/tmp.nRK3Tl33nd:/tmp,/etc,/media,/mnt,/opt,/srv,
/usr,/var]
DEBUG [U=1050,P=2465070] setValue() Updated flag ‘tmpdir’ value to: /config/binaries/singularity/tmp
DEBUG [U=1050,P=2465070] getCacheBasedir() environment variable SINGULARITY_CACHEDIR not set, using default image cache
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/library
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oci-tmp
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oci
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/net
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/shub
DEBUG [U=1050,P=2465070] updateCacheSubdir() Caching directory set to /home/bioinf/.singularity/cache/oras
DEBUG [U=1050,P=2465070] execStarter() Checking for encrypted system partition
DEBUG [U=1050,P=2465070] Init() Image format detection
DEBUG [U=1050,P=2465070] Init() Check for sandbox image format
DEBUG [U=1050,P=2465070] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1050,P=2465070] Init() Check for sif image format
DEBUG [U=1050,P=2465070] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1050,P=2465070] Init() Check for squashfs image format
DEBUG [U=1050,P=2465070] Init() squashfs image format detected
VERBOSE [U=1050,P=2465070] SetContainerEnv() Not forwarding SINGULARITY_TMPDIR from user to container environment
VERBOSE [U=1050,P=2465070] SetContainerEnv() Not forwarding SINGULARITY_BINDPATH from user to container environment
VERBOSE [U=1050,P=2465070] SetContainerEnv() HOME=/home/bioinf
DEBUG [U=1050,P=2465070] init() Use starter binary /config/binaries/singularity/3.5.2/libexec/singularity/bin/start
er-suid
VERBOSE [U=0,P=2465070] print() Set messagelevel to: 5
VERBOSE [U=0,P=2465070] init() Starter initialization
DEBUG [U=1050,P=2465079] startup() singularity runtime engine selected
DEBUG [U=0,P=2465070] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=2465070] load_overlay_module() Overlay seems supported by the kernel
DEBUG [U=0,P=2465070] get_pipe_exec_fd() PIPE_EXEC_FD value: 7
VERBOSE [U=0,P=2465070] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=2465070] priv_drop() Drop root privileges
DEBUG [U=1050,P=2465070] init() Read engine configuration
DEBUG [U=1050,P=2465070] init() Wait completion of stage1
VERBOSE [U=1050,P=2465079] priv_drop() Drop root privileges permanently
DEBUG [U=1050,P=2465079] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1050,P=2465079] init() Spawn stage 1
DEBUG [U=1050,P=2465079] startup() singularity runtime engine selected
VERBOSE [U=1050,P=2465079] startup() Execute stage 1
DEBUG [U=1050,P=2465079] StageOne() Entering stage 1
DEBUG [U=1050,P=2465079] prepareAutofs() Found “/proc/sys/fs/binfmt_misc” as autofs mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /tmp/tmp.nRK3Tl33nd: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /etc: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /media: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /mnt: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /opt: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /srv: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /usr: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for user bind path /var: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for home directory /home/bioinf: no mount point
DEBUG [U=1050,P=2465079] prepareAutofs() Could not keep file descriptor for current working directory /home/bioinf: no mount point
DEBUG [U=1050,P=2465079] Init() Image format detection
DEBUG [U=1050,P=2465079] Init() Check for sandbox image format
DEBUG [U=1050,P=2465079] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1050,P=2465079] Init() Check for sif image format
DEBUG [U=1050,P=2465079] Init() sif format initializer returned: SIF magic not found
DEBUG [U=1050,P=2465079] Init() Check for squashfs image format
DEBUG [U=1050,P=2465079] Init() squashfs image format detected
DEBUG [U=1050,P=2465079] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1050,P=2465079] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1050,P=2465070] wait_child() stage 1 exited with status 0
DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 4
DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 5
DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 6
DEBUG [U=1050,P=2465070] cleanup_fd() Close file descriptor 7
DEBUG [U=1050,P=2465070] init() Set child signal mask
DEBUG [U=1050,P=2465070] init() Create socketpair for master communication channel
DEBUG [U=1050,P=2465070] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1050,P=2465070] priv_escalate() Get root privileges
VERBOSE [U=0,P=2465070] priv_escalate() Change filesystem uid to 1050
VERBOSE [U=0,P=2465070] init() Spawn master process
DEBUG [U=0,P=2465104] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=2465104] create_namespace() Create mount namespace
VERBOSE [U=0,P=2465070] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=2465070] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=2465070] setns() Using syscall() wrapped __NR_setns
VERBOSE [U=0,P=2465070] priv_drop() Drop root privileges
VERBOSE [U=0,P=2465104] create_namespace() Create mount namespace
DEBUG [U=0,P=2465105] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=2465105] init() Spawn RPC server
DEBUG [U=1050,P=2465070] startup() singularity runtime engine selected
VERBOSE [U=1050,P=2465070] startup() Execute master process
DEBUG [U=0,P=2465105] startup() singularity runtime engine selected
VERBOSE [U=0,P=2465105] startup() Serve RPC requests
DEBUG [U=1050,P=2465070] setupSessionLayout() Using Layer system: overlay
DEBUG [U=1050,P=2465070] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1050,P=2465070] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1050,P=2465070] addRootfsMount() Image type is 4096
DEBUG [U=1050,P=2465070] addRootfsMount() Mounting block [squashfs] image: /config/binaries/singularity/containers_devel/rserver-launcher-centos7.simg
DEBUG [U=1050,P=2465070] addKernelMount() Checking configuration file for ‘mount proc’
DEBUG [U=1050,P=2465070] addKernelMount() Adding proc to mount list
VERBOSE [U=1050,P=2465070] addKernelMount() Default mount: /proc:/proc
DEBUG [U=1050,P=2465070] addKernelMount() Checking configuration file for ‘mount sys’
DEBUG [U=1050,P=2465070] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1050,P=2465070] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1050,P=2465070] addDevMount() Checking configuration file for ‘mount dev’
DEBUG [U=1050,P=2465070] addDevMount() Adding dev to mount list
VERBOSE [U=1050,P=2465070] addDevMount() Default mount: /dev:/dev
DEBUG [U=1050,P=2465070] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1050,P=2465070] addBindsMount() Found ‘bind path’ = /etc/localtime, /etc/localtime
VERBOSE [U=1050,P=2465070] addBindsMount() Found ‘bind path’ = /etc/hosts, /etc/hosts
DEBUG [U=1050,P=2465070] addHomeStagingDir() Staging home directory (/home/bioinf) at /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
DEBUG [U=1050,P=2465070] addHomeMount() Adding home directory mount [/config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf:/home/bioinf] to list using layer: overlay
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /tmp/tmp.nRK3Tl33nd to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /etc to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /media to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /mnt to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /opt to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /srv to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /usr to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Adding /var to mount list
DEBUG [U=1050,P=2465070] addUserbindsMount() Checking for ‘user bind control’ in configuration file
DEBUG [U=1050,P=2465070] addTmpMount() Checking for ‘mount tmp’ in configuration file
VERBOSE [U=1050,P=2465070] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1050,P=2465070] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1050,P=2465070] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1050,P=2465070] addCwdMount() Using /home/bioinf as current working directory
VERBOSE [U=1050,P=2465070] addCwdMount() Default mount: /home/bioinf: to the container
DEBUG [U=1050,P=2465070] addLibsMount() Checking for ‘user bind control’ in configuration file
DEBUG [U=1050,P=2465070] addFilesMount() Checking for ‘user bind control’ in configuration file
DEBUG [U=1050,P=2465070] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1050,P=2465070] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1050,P=2465070] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1050,P=2465070] create() Mount all
DEBUG [U=1050,P=2465070] mountGeneric() Mounting tmpfs to /config/binaries/singularity/3.5.2/var/singularity/mnt/session
DEBUG [U=1050,P=2465070] mountImage() Mounting loop device /dev/loop0 to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=1050,P=2465070] mountGeneric() Mounting overlay to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
DEBUG [U=1050,P=2465070] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=1050,P=2465070] Passwd() Checking for template passwd file: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1050,P=2465070] Passwd() Creating passwd content
VERBOSE [U=1050,P=2465070] Passwd() Creating template passwd file and appending user data: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=1050,P=2465070] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=1050,P=2465070] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=1050,P=2465070] Group() Checking for template group file: /config/binaries/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1050,P=2465070] Group() Creating group content
DEBUG [U=1050,P=2465070] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=1050,P=2465070] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /dev to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/dev
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc/localtime to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/localtime
WARNING [U=1050,P=2465070] mountGeneric() Skipping mount /etc/localtime [binds]: /etc/localtime doesn’t exist in container
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc/hosts to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/etc/singularity/actions to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /proc to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/proc
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/proc
DEBUG [U=1050,P=2465070] mountGeneric() Mounting sysfs to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/sys
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /home/bioinf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/home/bioinf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/home/bioinf
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /tmp to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /var/tmp to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
VERBOSE [U=1050,P=2465070] mountGeneric() Skipping mount /home/bioinf, /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/home/bioinf already mounted
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/passwd to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/etc/group to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc/group
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /tmp/tmp.nRK3Tl33nd to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/tmp
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /etc to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/etc
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /media to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/media
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/media
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /mnt to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/mnt
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/mnt
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /opt to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/opt
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/opt
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /srv to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/srv
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/srv
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /usr to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/usr
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/usr
DEBUG [U=1050,P=2465070] mountGeneric() Mounting /var to /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var
DEBUG [U=1050,P=2465070] mountGeneric() Remounting /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final/var
DEBUG [U=1050,P=2465070] create() Chroot into /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
DEBUG [U=0,P=2465105] Chroot() Hold reference to host / directory
DEBUG [U=0,P=2465105] Chroot() Called pivot_root on /config/binaries/singularity/3.5.2/var/singularity/mnt/session/final
DEBUG [U=0,P=2465105] Chroot() Change current directory to host / directory
DEBUG [U=0,P=2465105] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=0,P=2465105] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=0,P=2465105] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=1050,P=2465070] create() Chdir into / to avoid errors
VERBOSE [U=0,P=2465104] wait_child() rpc server exited with status 0
DEBUG [U=0,P=2465104] apply_container_privileges() Set user ID to 1050
DEBUG [U=1050,P=2465104] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=1050,P=2465104] startup() singularity runtime engine selected
VERBOSE [U=1050,P=2465104] startup() Execute stage 2
DEBUG [U=1050,P=2465104] StageTwo() Entering stage 2
DEBUG [U=1050,P=2465070] PostStartProcess() Post start process
DEBUG [U=1050,P=2465070] Master() Child exited with exit status 1
echo ‘Singularity as exited…’
Singularity as exited…
Timed out waiting for RStudio Server to open port 17482!
Cleaning up…
OK at this point, you may have something wrong with RStudio. Looks like singularity is doing it’s job fine after the update. localtime still seems to be wrong somehow, but it doesn’t seem fatal anymore.
There could also be an rsession.log in that directory that may indicate why RStudio is failing.
These options in /etc/singularity/singularity.conf may be relevant to you for properly mounting localtime. Maybe one or more is currently ‘no’?
# ENABLE OVERLAY: [yes/no/try]
# DEFAULT: try
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container. If 'try' is chosen,
# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = try
# ENABLE UNDERLAY: [yes/no]
# DEFAULT: yes
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container even if overlay is not
# working. If overlay is available, it will be tried first.
enable underlay = yes
I highly recommend replacing the OSC’s based RStudio Singularity image with that from rocker. We started with the OSC’s RStudio container but soon users started complaining that they can’t install the R libraries that they need.
Rocker has several different Docker Hub containers that all have RStudio Server, have different libraries pre-installed and work out of the box in our OOD instance. All one needs to do is to “singularity build” a Singularity sif container based on the Docker Hub image. Similarly we use a bioconductor Docker Hub container, which is based on rocker, for bio R stuff, as this one has all the system dependencies like libgdal installed in the container so users don’t have to struggle when building R libraries that have system dependencies. Our approach is to provide these base R containers with the system dependencies and have users to install their own libraries to their own disk space.
Our github repo that should cover this is at GitHub - CHPC-UofU/bc_osc_rstudio_server: Batch Connect - OSC RStudio Server. I would recommend to build one of the rocker containers with Singularity, see what you get in the /etc/localtime, I get this inside of the container:
$ singularity shell ood-rstudio-rocker_3.6.2.sif
Singularity> ls -l /etc/localtime
lrwxrwxrwx. 1 root root 27 Dec 23 17:00 /etc/localtime → /usr/share/zoneinfo/Etc/UTC
as opposed to when being on the host:
$ ls -l /etc/localtime
lrwxrwxrwx. 1 root root 36 Oct 24 2017 /etc/localtime → …/usr/share/zoneinfo/America/Denver
Being kind of doubtful on the Etc/UTC zone, whatever it does, time shows up right in the RStudio:
Hi Folks, Thanks everone for helping out and apologies for not getting back to you. Peter MaCallum Cancer Center as with many other organizations really are flat out because of the Virus issues.
Singularity timezone issue - I have not figured out but will look into the idea mcuma had.
As for Rstudio not working, this was my bad, as I did not bind the modules filesystem to the container, hence Rstudio server could not load the R modules and therefore failed.
I am now tonight looking at why PAM auth is failing (we use SSSD/IPA) for node auth.
Finally Congrats to the OOD team for such a great product! We will likely make this avail to all our staff who are now working from home (no pressure on me right?). I just hve to get it working with Auth now and quickly.
Got the PAM auth running, PAM was not enabled in IPA at our site. Sorted. Basics work Want to go soft live but Rstudio params I apply to the submit and form files for RStudio cause an error. I think I will open a new thread for this as I also have some questions from our test users that I cannot answer but yuo can.