RStudio App stuck on 'Starting'

Hello,

We’re experiencing an issue where RStudio jobs are stuck on Starting


and then time out waiting for it to start…

Here is an output.log:

Script starting...
Waiting for RStudio Server to open port 38439...host  - sfx2
++ date
+ echo 'Starting up rserver...' Mon Oct 10 10:18:54 EDT 2022
Starting up rserver... Mon Oct 10 10:18:54 EDT 2022
++ whoami
+ singularity run -B /tmp/tmp.UG9EmpMA1S:/tmp /wsu/apps/rserver-launcher-centos7.simg --www-port 38439 --auth-none 0 --auth-pam-helper-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/87fcc58f-4cae-43ec-8a2d-ee0b0009dded/bin/auth --auth-encrypt-password 0 --rsession-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/87fcc58f-4cae-43ec-8a2d-ee0b0009dded/rsession.sh --server-user=fh5997
e[33mWARNING:e[0m Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
++ date
+ echo 'Singularity as exited...' Mon Oct 10 10:18:55 EDT 2022
Singularity as exited... Mon Oct 10 10:18:55 EDT 2022
Timed out waiting for RStudio Server to open port 38439!
Cleaning up...

Just to be thorough, I ran singularity --debug to check if maybe that was an issue and here’s a new output.log

Script starting...
Waiting for RStudio Server to open port 19547...host  - mem1
++ date
+ echo 'Starting up rserver...' Mon Oct 10 10:58:07 EDT 2022
Starting up rserver... Mon Oct 10 10:58:07 EDT 2022
++ whoami
+ singularity --debug run -B /tmp/tmp.MkaulLXaH4:/tmp /wsu/apps/rserver-launcher-centos7.simg --www-port 19547 --auth-none 0 --auth-pam-helper-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/69eae5b9-e591-438d-a7b7-4dd13edaf447/bin/auth --auth-encrypt-password 0 --rsession-path /wsu/home/fh/fh59/fh5997/ondemand/data/sys/dashboard/batch_connect/sys/RStudio/output/69eae5b9-e591-438d-a7b7-4dd13edaf447/rsession.sh --server-user=fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]persistentPreRunE()           Singularity version: 3.5.2
e[0mDEBUG   e[0m[U=6085997,P=24878]handleConfDir()               /wsu/home/fh/fh59/fh5997/.singularity already exists. Not creating.
e[0mDEBUG   e[0m[U=6085997,P=24878]setValue()                    Updated flag 'bind' value to: [/tmp/tmp.MkaulLXaH4:/tmp,/etc,/media,/mnt,/opt,/srv,/usr,/var,/wsu,/var/run]
e[0mDEBUG   e[0m[U=6085997,P=24878]getCacheBasedir()             environment variable SINGULARITY_CACHEDIR not set, using default image cache
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/library
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oci-tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oci
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/net
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/shub
e[0mDEBUG   e[0m[U=6085997,P=24878]updateCacheSubdir()           Caching directory set to /wsu/home/fh/fh59/fh5997/.singularity/cache/oras
e[0mDEBUG   e[0m[U=6085997,P=24878]execStarter()                 Checking for encrypted system partition
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Image format detection
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for sandbox image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        sandbox format initializer returned: not a directory image
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for sif image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        sif format initializer returned: SIF magic not found
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        Check for squashfs image format
e[0mDEBUG   e[0m[U=6085997,P=24878]Init()                        squashfs image format detected
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_DIR from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_BINDPATH from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             Not forwarding SINGULARITY_BIN from user to container environment
e[0mVERBOSE e[0m[U=6085997,P=24878]SetContainerEnv()             HOME=/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]init()                        Use starter binary /wsu/el7/gnu7/singularity/3.5.2/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=24878]      print()                       Set messagelevel to: 5
e[0mVERBOSE [U=0,P=24878]      init()                        Starter initialization
e[0mDEBUG   [U=0,P=24878]      load_overlay_module()         Trying to load overlay kernel module
e[0mDEBUG   [U=0,P=24878]      load_overlay_module()         Overlay seems supported by the kernel
e[0mDEBUG   [U=0,P=24878]      get_pipe_exec_fd()            PIPE_EXEC_FD value: 7
e[0mVERBOSE [U=0,P=24878]      is_suid()                     Check if we are running as setuid
e[0mVERBOSE [U=0,P=24878]      priv_drop()                   Drop root privileges
e[0mDEBUG   [U=6085997,P=24878]  init()                        Read engine configuration
e[0mDEBUG   [U=6085997,P=24878]  init()                        Wait completion of stage1
e[0mVERBOSE [U=6085997,P=24889]  priv_drop()                   Drop root privileges permanently
e[0mDEBUG   [U=6085997,P=24889]  set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=6085997,P=24889]  init()                        Spawn stage 1
e[0me[0mDEBUG   e[0m[U=6085997,P=24889]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=6085997,P=24889]startup()                     Execute stage 1
e[0mDEBUG   e[0m[U=6085997,P=24889]StageOne()                    Entering stage 1
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/nfs" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Found "/cvmfs" as autofs mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /tmp/tmp.MkaulLXaH4: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /etc: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /media: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /mnt: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /opt: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /srv: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /usr: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /var: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /wsu: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for user bind path /var/run: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for bind path /etc/localtime: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for bind path /etc/hosts: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for home directory /wsu/home/fh/fh59/fh5997: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]prepareAutofs()               Could not keep file descriptor for current working directory /wsu/home/fh/fh59/fh5997: no mount point
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Image format detection
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for sandbox image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        sandbox format initializer returned: not a directory image
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for sif image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        sif format initializer returned: SIF magic not found
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        Check for squashfs image format
e[0mDEBUG   e[0m[U=6085997,P=24889]Init()                        squashfs image format detected
e[0mDEBUG   e[0m[U=6085997,P=24889]setSessionLayer()             Overlay seems supported and allowed by kernel
e[0mDEBUG   e[0m[U=6085997,P=24889]setSessionLayer()             Attempting to use overlayfs (enable overlay = yes)
VERBOSE [U=6085997,P=24878]  wait_child()                  stage 1 exited with status 0
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 4
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 5
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 6
e[0mDEBUG   [U=6085997,P=24878]  cleanup_fd()                  Close file descriptor 7
e[0mDEBUG   [U=6085997,P=24878]  init()                        Set child signal mask
e[0mDEBUG   [U=6085997,P=24878]  init()                        Create socketpair for master communication channel
e[0mDEBUG   [U=6085997,P=24878]  init()                        Create RPC socketpair for communication between stage 2 and RPC server
e[0mVERBOSE [U=6085997,P=24878]  priv_escalate()               Get root privileges
e[0mVERBOSE [U=0,P=24878]      priv_escalate()               Change filesystem uid to 6085997
e[0mVERBOSE [U=0,P=24878]      init()                        Spawn master process
e[0mDEBUG   [U=0,P=24902]      set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=0,P=24902]      create_namespace()            Create mount namespace
e[0mVERBOSE [U=0,P=24878]      enter_namespace()             Entering in mount namespace
e[0mDEBUG   [U=0,P=24878]      enter_namespace()             Opening namespace file ns/mnt
e[0mDEBUG   [U=0,P=24878]      setns()                       Using syscall() wrapped __NR_setns
e[0mVERBOSE [U=0,P=24902]      create_namespace()            Create mount namespace
e[0mVERBOSE [U=0,P=24878]      priv_drop()                   Drop root privileges
e[0mDEBUG   [U=0,P=24903]      set_parent_death_signal()     Set parent death signal to 9
e[0mVERBOSE [U=0,P=24903]      init()                        Spawn RPC server
e[0me[0mDEBUG   e[0m[U=0,P=24903]      startup()                     singularity runtime engine selected
e[0mDEBUG   e[0m[U=6085997,P=24878]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=0,P=24903]      startup()                     Serve RPC requests
e[0mVERBOSE e[0m[U=6085997,P=24878]startup()                     Execute master process
e[0mDEBUG   e[0m[U=6085997,P=24878]setupSessionLayout()          Using Layer system: overlay
e[0mDEBUG   e[0m[U=6085997,P=24878]setupOverlayLayout()          Creating overlay SESSIONDIR layout
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Mount rootfs in read-only mode
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Image type is 4096
e[0mDEBUG   e[0m[U=6085997,P=24878]addRootfsMount()              Mounting block [squashfs] image: /wsu/apps/rserver-launcher-centos7.simg
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Checking configuration file for 'mount proc'
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Adding proc to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addKernelMount()              Default mount: /proc:/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Checking configuration file for 'mount sys'
e[0mDEBUG   e[0m[U=6085997,P=24878]addKernelMount()              Adding sysfs to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addKernelMount()              Default mount: /sys:/sys
e[0mDEBUG   e[0m[U=6085997,P=24878]addDevMount()                 Checking configuration file for 'mount dev'
e[0mDEBUG   e[0m[U=6085997,P=24878]addDevMount()                 Adding dev to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addDevMount()                 Default mount: /dev:/dev
e[0mDEBUG   e[0m[U=6085997,P=24878]addHostMount()                Not mounting host file systems per configuration
e[0mVERBOSE e[0m[U=6085997,P=24878]addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
e[0mVERBOSE e[0m[U=6085997,P=24878]addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
e[0mDEBUG   e[0m[U=6085997,P=24878]addHomeStagingDir()           Staging home directory (/wsu/home/fh/fh59/fh5997) at /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]addHomeMount()                Adding home directory mount [/wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997:/wsu/home/fh/fh59/fh5997] to list using layer: overlay
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /tmp/tmp.MkaulLXaH4 to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /etc to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /media to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /mnt to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /opt to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /srv to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /usr to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /var to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /wsu to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Adding /var/run to mount list
e[0mDEBUG   e[0m[U=6085997,P=24878]addUserbindsMount()           Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addTmpMount()                 Checking for 'mount tmp' in configuration file
e[0mVERBOSE e[0m[U=6085997,P=24878]addTmpMount()                 Default mount: /tmp:/tmp
e[0mVERBOSE e[0m[U=6085997,P=24878]addTmpMount()                 Default mount: /var/tmp:/var/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]addScratchMount()             Not mounting scratch directory: Not requested
e[0mDEBUG   e[0m[U=6085997,P=24878]addCwdMount()                 Using /wsu/home/fh/fh59/fh5997 as current working directory
e[0mVERBOSE e[0m[U=6085997,P=24878]addCwdMount()                 Default mount: /wsu/home/fh/fh59/fh5997: to the container
e[0mDEBUG   e[0m[U=6085997,P=24878]addLibsMount()                Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addFilesMount()               Checking for 'user bind control' in configuration file
e[0mDEBUG   e[0m[U=6085997,P=24878]addResolvConfMount()          Adding /etc/resolv.conf to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
e[0mDEBUG   e[0m[U=6085997,P=24878]addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Mount all
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting tmpfs to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session
e[0mDEBUG   e[0m[U=6085997,P=24878]mountImage()                  Mounting loop device /dev/loop0 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs of type squashfs
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting overlay to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=6085997,P=24878]setPropagationMount()         Set RPC mount propagation flag to SLAVE
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Checking for template passwd file: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Creating passwd content
e[0mVERBOSE e[0m[U=6085997,P=24878]Passwd()                      Creating template passwd file and appending user data: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/passwd
e[0mDEBUG   e[0m[U=6085997,P=24878]addIdentityMount()            Adding /etc/passwd to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addIdentityMount()            Default mount: /etc/passwd:/etc/passwd
e[0mVERBOSE e[0m[U=6085997,P=24878]Group()                       Checking for template group file: /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/rootfs/etc/group
e[0mVERBOSE e[0m[U=6085997,P=24878]Group()                       Creating group content
e[0mDEBUG   e[0m[U=6085997,P=24878]addIdentityMount()            Adding /etc/group to mount list
e[0mVERBOSE e[0m[U=6085997,P=24878]addIdentityMount()            Default mount: /etc/group:/etc/group
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /dev to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/dev
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc/localtime to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/localtime
e[33mWARNING e[0m[U=6085997,P=24878]mountGeneric()                Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc/hosts to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/hosts
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/etc/singularity/actions to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/.singularity.d/actions
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /proc to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/proc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting sysfs to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/sys
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/home/fh/fh59/fh5997 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/wsu/home/fh/fh59/fh5997 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu/home/fh/fh59/fh5997
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /tmp to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var/tmp to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/tmp
e[0mVERBOSE e[0m[U=6085997,P=24878]mountGeneric()                Skipping mount /wsu/home/fh/fh59/fh5997, /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu/home/fh/fh59/fh5997 already mounted
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/resolv.conf to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/resolv.conf
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/passwd to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/passwd
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/etc/group to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc/group
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /tmp/tmp.MkaulLXaH4 to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/tmp
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /etc to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/etc
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /media to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/media
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/media
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /mnt to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/mnt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/mnt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /opt to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/opt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/opt
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /srv to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/srv
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/srv
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /usr to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/usr
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/usr
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /wsu to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/wsu
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Mounting /var/run to /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/run
e[0mDEBUG   e[0m[U=6085997,P=24878]mountGeneric()                Remounting /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final/var/run
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Chroot into /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Hold reference to host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Called pivot_root on /wsu/el7/gnu7/singularity/3.5.2/var/singularity/mnt/session/final
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Change current directory to host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Apply slave mount propagation for host / directory
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Called unmount(/, syscall.MNT_DETACH)
e[0mDEBUG   e[0m[U=0,P=24903]      Chroot()                      Changing directory to / to avoid getpwd issues
e[0mDEBUG   e[0m[U=6085997,P=24878]create()                      Chdir into / to avoid errors
VERBOSE [U=0,P=24902]      wait_child()                  rpc server exited with status 0
e[0mDEBUG   [U=0,P=24902]      apply_container_privileges()  Set user ID to 6085997
e[0mDEBUG   [U=6085997,P=24902]  set_parent_death_signal()     Set parent death signal to 9
e[0me[0mDEBUG   e[0m[U=6085997,P=24902]startup()                     singularity runtime engine selected
e[0mVERBOSE e[0m[U=6085997,P=24902]startup()                     Execute stage 2
e[0mDEBUG   e[0m[U=6085997,P=24902]StageTwo()                    Entering stage 2
e[0mDEBUG   e[0m[U=6085997,P=24878]PostStartProcess()            Post start process
e[0mDEBUG   e[0m[U=6085997,P=24878]Master()                      Child exited with exit status 1
++ date
+ echo 'Singularity as exited...' Mon Oct 10 10:58:07 EDT 2022
Singularity as exited... Mon Oct 10 10:58:07 EDT 2022

If you’re able to boot the process - I doubt it’s an issue with singularity.

One thing that’s really helped us at OSC debug Rstudio’s issues is using a different logfile for it. Right now it’s logging inside the container to a /var/log location. You want those logs to persist on the host so you can read them after the fact.

Here’s the file we template:

That we then mount to /etc/rstudio to reconfigure it.

Newer versions of RStudio will place a log by default in ~/.local/share/rstudio/log/rstudio.log. There might be information in there as to why RStudio is not starting correctly.