# Interactive sessions stuck at starting stage

**URL:** <https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919>\
**Category:** Get Help\
**Created:** [May 18, 2020, 5:12pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919 "2020-05-18T17:12:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![romxero](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/romxero/32/441_2.png) [@romxero](https://discourse.openondemand.org/u/romxero)\
**Post date:** [May 18, 2020, 5:12pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/1 "2020-05-18T17:12:33Z")

</div>

Hi there,

It appears our on demand interactive apps are stuck at the starting phase.

The jobs run, I can verify that the apps actually work outside of the dashboard. But it never gives a link to the app. It just stalls at this point.

Was wondering if I can find any pointers on how to fix this issue?

 ![Screenshot_2020-05-18 My Interactive Sessions](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/fd4c91158e02e631ab3fb125f401d18c87952a51.png)

Thank you for such a cool tool!

RC

---

<div class="post-metadata">

**Author:** ![mario](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mario/32/662_2.png) [@mario](https://discourse.openondemand.org/u/mario)\
**Post date:** [May 18, 2020, 5:31pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/2 "2020-05-18T17:31:39Z")

</div>

Can you share the `output.log` generated from the session? You can get to `output.log` by clicking on the Session ID in your Interactive Sessions list.

output.log will be in a directory like like this:

 ![Screen Shot 2020-05-18 at 1.30.57 PM](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/f2303716d6c971bda3b030695334ed0553122c79.png)

---

<div class="post-metadata">

**Author:** ![romxero](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/romxero/32/441_2.png) [@romxero](https://discourse.openondemand.org/u/romxero)\
**Post date:** [May 18, 2020, 8:03pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/3 "2020-05-18T20:03:57Z")

</div>

Hey Mario,

I don’t have the output.log in the file explorer when I click the session id.

To clarify a bit more, our login node can read/write to the $HOME directories. The compute nodes can only read from the $HOME directories, but their writes happen through an overlayfs mount.

I had to login to the compute node to get the output.log:

```
cat output.log 
/var/spool/slurm/d/job1572404/slurm_script: line 3: module: command not found
Script starting...
Waiting for Jupyter server to open port 7449...
TIMING - Starting main script at: Mon May 18 09:48:34 PDT 2020
TIMING - Starting jupyter at: Mon May 18 09:48:34 PDT 2020
+ jupyter-lab --config=/home/rcwhite/ondemand/data/sys/dashboard/batch_connect/dev/jupyter_test/output/694741ad-a664-4eee-924a-c0db3dd9961b/config.py
[W 09:48:35.689 LabApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[W 09:48:35.700 LabApp] JupyterLab server extension not enabled, manually loading...
[I 09:48:35.707 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.6/site-packages/jupyterlab
[I 09:48:35.707 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 09:48:35.711 LabApp] Serving notebooks from local directory: /home/rcwhite
[I 09:48:35.711 LabApp] The Jupyter Notebook is running at:
[I 09:48:35.711 LabApp] http://(maz044 or 127.0.0.1):7449/node/maz044/7449/
[I 09:48:35.711 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Discovered Jupyter server listening on port 7449!
Generating connection YAML file...

```

Would the type of configuration we have cause the apps to stall? Is the home directory used as an IPC of sort?

We have beegfs shares that is writable by all system, just our home directory is the one that has this caveat.

---

<div class="post-metadata">

**Author:** ![mario](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mario/32/662_2.png) [@mario](https://discourse.openondemand.org/u/mario)\
**Post date:** [May 18, 2020, 8:50pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/4 "2020-05-18T20:50:22Z")

</div>

Make sure that compute nodes are syncing their writes properly. `connection.yml` is generated on the compute node directly. If writes from compute nodes aren’t being synced quickly enough to your file system, then that would explain why the login node is stalled because its waiting for `connection.yml` to exist.

The home directory is similar to IPC, for example the login node is looking for `/jupyter_test/output/694741ad-a664-4eee-924a-c0db3dd9961b/connection.yml`

---

<div class="post-metadata">

**Author:** ![romxero](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/romxero/32/441_2.png) [@romxero](https://discourse.openondemand.org/u/romxero)\
**Post date:** [May 18, 2020, 10:23pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/5 "2020-05-18T22:23:36Z")

</div>

Hi Mario,

Thank you for the clarification. I had figured it was something along that lines. The group I work with will meet next Friday and discuss setting up read/write on home directory in the cluster as a whole.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [September 23, 2020, 7:25pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/6 "2020-09-23T19:25:56Z")

</div>

Hey @romxero, hope you resolved your overlay issue.

There’s a way to set this directory to be something else in case you’re interested and/or need to. You set this environment variable in the `/etc/ood/config/apps/dashboard/env` file.

```auto
# /etc/ood/config/apps/dashboard/env
OOD_DATAROOT: "/some/writable/mount/location"

```

---

<div class="post-metadata">

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\
**Post date:** [May 26, 2022, 3:43pm UTC](https://discourse.openondemand.org/t/interactive-sessions-stuck-at-starting-stage/919/7 "2022-05-26T15:43:06Z")

</div>