RStudio App stuck on 'Starting'

sabdall33 · October 10, 2022, 2:46pm

Hello,

We’re experiencing an issue where RStudio jobs are stuck on Starting

and then time out waiting for it to start…

Here is an output.log:

Script starting...
Waiting for RStudio Server to open port 38439...host  - sfx2
++ date
+ echo 'Starting up rserver...' Mon Oct 10 10:18:54 EDT 2022
Starting up rserver... Mon Oct 10 10:18:54 EDT 2022
++ whoami
+ singularity run -B /tmp/tmp.UG9EmpMA1S:/tmp /wsu/apps/rserver-launcher-centos7.simg --www-port 38439 --auth-none 0 --auth-pam-helper-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/87fcc58f-4cae-43ec-8a2d-ee0b0009dded/bin/auth --auth-encrypt-password 0 --rsession-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/87fcc58f-4cae-43ec-8a2d-ee0b0009dded/rsession.sh --server-user=fh5997
e[33mWARNING:e[0m Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
++ date
+ echo 'Singularity as exited...' Mon Oct 10 10:18:55 EDT 2022
Singularity as exited... Mon Oct 10 10:18:55 EDT 2022
Timed out waiting for RStudio Server to open port 38439!
Cleaning up...

Just to be thorough, I ran singularity --debug to check if maybe that was an issue and here’s a new output.log

Script starting...
Waiting for RStudio Server to open port 19547...host  - mem1
++ date
+ echo 'Starting up rserver...' Mon Oct 10 10:58:07 EDT 2022
Starting up rserver... Mon Oct 10 10:58:07 EDT 2022
++ whoami
+ singularity --debug run -B /tmp/tmp.MkaulLXaH4:/tmp /wsu/apps/rserver-launcher-centos7.simg --www-port 19547 --auth-none 0 --auth-pam-helper-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/69eae5b9-e591-438d-a7b7-4dd13edaf447/bin/auth --auth-encrypt-password 0 --rsession-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/69eae5b9-e591-438d-a7b7-4dd13edaf447/rsession.sh --server-user=fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]persistentPreRunE()           Singularity version: 3.5.2
e[0mDEBUG   e[0m[U=6085997,P=24878]handleConfDir()               /wsu/home/fh/fh59/fh5997/.singularity already exists. Not creating.
e[0mDEBUG   e[0m[U=6085997,P=24878]setValue()                    Updated flag 'bind' value to: [/tmp/tmp.MkaulLXaH4:/tmp,/etc,/media,/mnt,/opt,/srv,/usr,/var,/wsu,/var/run]
e[0mDEBUG   e[0m[U=6085997,P=24878]getCacheBasedir()             environment variable SINGULARITY_CACHEDIR not set, using default image cache
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/library
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oci-tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oci
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/net
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/shub
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oras
e[0mDEBUG   e[0m[U=6085997,P=24878]execStarter()                 Checking for encrypted system partition
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Image format detection
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for sandbox image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        sandbox format initializer returned: not a directory image
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for sif image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        sif format initializer returned: SIF magic not found
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for squashfs image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        squashfs image format detected
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_DIR from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_BINDPATH from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_BIN from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             HOME=/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]init()                        Use starter binary /wsu/el7/gnu7/singularity/3.5.2/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=24878]      print()                       Set messagelevel to: 5
e[0mVERBOSE [U=0,P=24878]      init()                        Starter initialization
e[0mDEBUG   [U=0,P=24878]      load_overlay_module()         Trying to load overlay kernel module
e[0mDEBUG   [U=0,P=24878]      load_overlay_module()         Overlay seems supported by the kernel
e[0mDEBUG   [U=0,P=24878]      get_pipe_exec_fd()            PIPE_EXEC_FD value: 7
e[0mVERBOSE [U=0,P=24878]      is_suid()                     Check if we are running as setuid
e[0mVERBOSE [U=0,P=24878]      priv_drop()                   Drop root privileges
e[0mDEBUG   [U=6085997,P=24878]  init()                        Read engine configuration
e[0mDEBUG   [U=6085997,P=24878]  init()                        Wait completion of stage1
e[0mVERBOSE [U=6085997,P=24889]  priv_drop()                   Drop root privileges permanently
e[0mDEBUG   [U=6085997,P=24889]  set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=6085997,P=24889]  init()                        Spawn stage 1
e[0me[0mDEBUG   e[0m[U=6085997,P=24889]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=6085997,P=24889]startup()                     Execute stage 1
e[0mDEBUG   e[0m[U=6085997,P=24889]StageOne()                    Entering stage 1
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/nfs" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/cvmfs" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /tmp/tmp.MkaulLXaH4: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /etc: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /media: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /mnt: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /opt: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /srv: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /usr: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /var: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /wsu: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /var/run: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for bind path /etc/localtime: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for bind path /etc/hosts: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for home directory /wsu/home/fh/fh59/fh5997: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for current working directory /wsu/home/fh/fh59/fh5997: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Image format detection
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for sandbox image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        sandbox format initializer returned: not a directory image
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for sif image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        sif format initializer returned: SIF magic not found
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for squashfs image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        squashfs image format detected
e[0mDEBUG   e[0m[U=6085997,P=24889]setSessionLayer()             Overlay seems supported and allowed by kernel
e[0mDEBUG   e[0m[U=6085997,P=24889]setSessionLayer()             Attempting to use overlayfs (enable overlay = yes)
VERBOSE [U=6085997,P=24878]  wait_child()                  stage 1 exited with status 0
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 4
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 5
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 6
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 7
e[0mDEBUG   [U=6085997,P=24878]  init()                        Set child signal mask
e[0mDEBUG   [U=6085997,P=24878]  init()                        Create socketpair for master communication channel
e[0mDEBUG   [U=6085997,P=24878]  init()                        Create RPC socketpair for communication between stage 2 and RPC server
e[0mVERBOSE [U=6085997,P=24878]  priv_escalate()               Get root privileges
e[0mVERBOSE [U=0,P=24878]      priv_escalate()               Change filesystem uid to 6085997
e[0mVERBOSE [U=0,P=24878]      init()                        Spawn master process
e[0mDEBUG   [U=0,P=24902]      set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=0,P=24902]      create_namespace()            Create mount namespace
e[0mVERBOSE [U=0,P=24878]      enter_namespace()             Entering in mount namespace
e[0mDEBUG   [U=0,P=24878]      enter_namespace()             Opening namespace file ns/mnt
e[0mDEBUG   [U=0,P=24878]      setns()                       Using syscall() wrapped __NR_setns
e[0mVERBOSE [U=0,P=24902]      create_namespace()            Create mount namespace
e[0mVERBOSE [U=0,P=24878]      priv_drop()                   Drop root privileges
e[0mDEBUG   [U=0,P=24903]      set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=0,P=24903]      init()                        Spawn RPC server
e[0me[0mDEBUG   e[0m[U=0,P=24903]      startup()                     singularity runtime engine selected
e[0mDEBUG   e[0m[U=6085997,P=24878]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=0,P=24903]      startup()                     Serve RPC requests
e[0mVERBOSE e[0m[U=6085997,P=24878]startup()                     Execute master process
e[0mDEBUG   e[0m[U=6085997,P=24878]setupSessionLayout()          Using Layer system: overlay
e[0mDEBUG   e[0m[U=6085997,P=24878]setupOverlayLayout()          Creating overlay SESSIONDIR layout
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Mount rootfs in read-only mode
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Image type is 4096
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Mounting block [squashfs] image: /wsu/apps/rserver-launcher-centos7.simg
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Checking configuration file for 'mount proc'
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Adding proc to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addKernelMount()              Default mount: /proc:/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Checking configuration file for 'mount sys'
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Adding sysfs to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addKernelMount()              Default mount: /sys:/sys
e[0mDEBUG   e[0m[U=6085997,P=24878]addDevMount()                 Checking configuration file for 'mount dev'
e[0mDEBUG   e[0m[U=6085997,P=24878]addDevMount()                 Adding dev to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addDevMount()                 Default mount: /dev:/dev
e[0mDEBUG   e[0m[U=6085997,P=24878]addHostMount()                Not mounting host file systems per configuration
e[0mVERBOSE e[0m[U=6085997,P=24878]addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
e[0mVERBOSE e[0m[U=6085997,P=24878]addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
e[0mDEBUG   e[0m[U=6085997,P=24878]addHomeStagingDir()           Staging home directory (/wsu/home/fh/fh59/fh5997) at /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]addHomeMount()                Adding home directory mount [/wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997:/wsu/home/fh/fh59/fh5997] to list using layer: overlay
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /tmp/tmp.MkaulLXaH4 to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /etc to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /media to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /mnt to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /opt to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /srv to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /usr to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /var to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /wsu to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /var/run to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addTmpMount()                 Checking for 'mount tmp' in configuration file
e[0mVERBOSE e[0m[U=6085997,P=24878]addTmpMount()                 Default mount: /tmp:/tmp
e[0mVERBOSE e[0m[U=6085997,P=24878]addTmpMount()                 Default mount: /var/tmp:/var/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]addScratchMount()             Not mounting scratch directory: Not requested
e[0mDEBUG   e[0m[U=6085997,P=24878]addCwdMount()                 Using /wsu/home/fh/fh59/fh5997 as current working directory
e[0mVERBOSE e[0m[U=6085997,P=24878]addCwdMount()                 Default mount: /wsu/home/fh/fh59/fh5997: to the container
e[0mDEBUG   e[0m[U=6085997,P=24878]addLibsMount()                Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addFilesMount()               Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addResolvConfMount()          Adding /etc/resolv.conf to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
e[0mDEBUG   e[0m[U=6085997,P=24878]addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Mount all
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting tmpfs to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session
e[0mDEBUG   e[0m[U=6085997,P=24878]mountImage()                  Mounting loop device /dev/loop0 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs of type squashfs
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting overlay to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=6085997,P=24878]setPropagationMount()         Set RPC mount propagation flag to SLAVE
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Checking for template passwd file: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Creating passwd content
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Creating template passwd file and appending user data: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
e[0mDEBUG   e[0m[U=6085997,P=24878]addIdentityMount()            Adding /etc/passwd to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addIdentityMount()            Default mount: /etc/passwd:/etc/passwd
e[0mVERBOSE e[0m[U=6085997,P=24878]Group()                       Checking for template group file: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/group
e[0mVERBOSE e[0m[U=6085997,P=24878]Group()                       Creating group content
e[0mDEBUG   e[0m[U=6085997,P=24878]addIdentityMount()            Adding /etc/group to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addIdentityMount()            Default mount: /etc/group:/etc/group
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /dev to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/dev
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc/localtime to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/localtime
e[33mWARNING e[0m[U=6085997,P=24878]mountGeneric()                Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc/hosts to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/hosts
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/etc/singularity/actions to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /proc to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting sysfs to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/sys
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/home/fh/fh59/fh5997 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /tmp to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var/tmp to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
e[0mVERBOSE e[0m[U=6085997,P=24878]mountGeneric()                Skipping mount /wsu/home/fh/fh59/fh5997, /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu/home/fh/fh59/fh5997 already mounted
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/resolv.conf
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/passwd to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/passwd
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/group to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/group
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /tmp/tmp.MkaulLXaH4 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /media to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/media
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/media
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /mnt to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/mnt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/mnt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /opt to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/opt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/opt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /srv to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/srv
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/srv
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /usr to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/usr
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/usr
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var/run to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/run
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/run
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Chroot into /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Hold reference to host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Called pivot_root on /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Change current directory to host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Apply slave mount propagation for host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Called unmount(/, syscall.MNT_DETACH)
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Changing directory to / to avoid getpwd issues
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Chdir into / to avoid errors
VERBOSE [U=0,P=24902]      wait_child()                  rpc server exited with status 0
e[0mDEBUG   [U=0,P=24902]      apply_container_privileges()  Set user ID to 6085997
e[0mDEBUG   [U=6085997,P=24902]  set_parent_death_signal()     Set parent death signal to 9
e[0me[0mDEBUG   e[0m[U=6085997,P=24902]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=6085997,P=24902]startup()                     Execute stage 2
e[0mDEBUG   e[0m[U=6085997,P=24902]StageTwo()                    Entering stage 2
e[0mDEBUG   e[0m[U=6085997,P=24878]PostStartProcess()            Post start process
e[0mDEBUG   e[0m[U=6085997,P=24878]Master()                      Child exited with exit status 1
++ date
+ echo 'Singularity as exited...' Mon Oct 10 10:58:07 EDT 2022
Singularity as exited... Mon Oct 10 10:58:07 EDT 2022

jeff.ohrstrom · October 10, 2022, 5:20pm

If you’re able to boot the process - I doubt it’s an issue with singularity.

One thing that’s really helped us at OSC debug Rstudio’s issues is using a different logfile for it. Right now it’s logging inside the container to a /var/log location. You want those logs to persist on the host so you can read them after the fact.

Here’s the file we template:

github.com

OSC/bc_osc_rstudio_server/blob/5349225dba0f432d3b61ecb199c76c817f85cfdd/template/etc/logging.conf.erb#L4


      
          [*]
          log-level=debug
          logger-type=file
          log-dir=<%= session.staged_root.join('logs') %>

That we then mount to /etc/rstudio to reconfigure it.

github.com

OSC/bc_osc_rstudio_server/blob/5349225dba0f432d3b61ecb199c76c817f85cfdd/template/script.sh.erb#L147


      
          <%- if context.cluster !~ /kubernetes/ -%>
          # --contain is used to override the bindpaths specified in OSC's
          # singularity.conf which would otherwise conflict in /etc
          # using --contain requires us to bind mount in addition directories including
          # /dev and /proc
          # optional_tutorial_home_bind is used when include_tutorials to change the home
          # directory to get a clean environment without clobbering files
          singularity run --contain \
           -B $TMPDIR:/tmp \
           -B $WORKING_DIR \
           -B $WORKING_DIR/etc:/etc/rstudio \
           -B /etc/fonts \
           -B /etc/profile \
           -B /etc/profile.d \
           -B /etc/alternatives \
           -B <%= module_dir %> \
           "$RSTUDIO_SERVER_IMAGE" \
           --www-port "${port}" \
           --auth-none 0 \
           --auth-pam-helper-path "${RSTUDIO_AUTH}" \
           --auth-encrypt-password 0 \

skelting1 · October 11, 2022, 4:27pm

Newer versions of RStudio will place a log by default in ~/.local/share/rstudio/log/rstudio.log. There might be information in there as to why RStudio is not starting correctly.

system · April 9, 2023, 4:27pm

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Timed out waiting for RStudio Server to open port Get Help	3	1378	April 25, 2022
Looking for some debugging tips re: rstudio app on el8 Get Help	10	2650	May 26, 2022
RStudio Server Issues Get Help question	13	1793	November 23, 2022
Issues with RStudio Server Integration in Open OnDemand using Singularity Get Help	8	229	December 9, 2024
Rstudio Server app (non-local) error Get Help	3	443	May 26, 2022

RStudio App stuck on 'Starting'

Related topics