# Problems starting VNC on OOD with LSF manager

**URL:** <https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544>\
**Category:** Get Help\
**Created:** [October 10, 2019, 11:34pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544 "2019-10-10T23:34:43Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rotaugenlaubfrosch](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rotaugenlaubfrosch/32/314_2.png) [@rotaugenlaubfrosch](https://discourse.openondemand.org/u/rotaugenlaubfrosch)\
**Post date:** [October 10, 2019, 11:34pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/1 "2019-10-10T23:34:43Z")

</div>

Hello All,  
First of all, the OOD platform is really impressive - nice work OSC!  
I’m sure it’s something simple, but I get problems when starting a VNC session on LSF manager.  
When starting it from the OOD interface, all the processes related to VNC get started.  
However, in the frontend it keeps saying “Your session is currently starting… Please be patient as this process can take a few minutes.”, and the “Launch Desktop” button will never appear.  
I tried it with SLURM and it worked, while on LSF there is this issue.  
What am I missing?  
Thanks!  
Rotaugenlaubfrosch

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [October 11, 2019, 2:54pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/2 "2019-10-11T14:54:53Z")

</div>

Hi @rotaugenlaubfrosch, what does your `$HOME/ondemand/data/sys/dashboard/batch_connect/sys/$VNC_APP_SUBDIR/output/$UUID/output.log` say? Or the `vnc.log`?

---

<div class="post-metadata">

**Author:** ![cjh](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/cjh/32/198_2.png) [@cjh](https://discourse.openondemand.org/u/cjh)\
**Post date:** [October 11, 2019, 7:46pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/3 "2019-10-11T19:46:51Z")

</div>

Rotaugenlaubfrosch,

This sounds similar to an issue I had when setting up OOD with LSF, in my case it was a race condition, if LSF didn’t show a submitted job via ‘bjobs’ immediately after submission, the startup procedure would never continue.

My solution was to make a wrapper script for the bjobs command to have it sleep for a few seconds before running the actual ‘bjobs’ command, then pointed to it in _cluster_.yml:

```
job:
  bin_overrides:
    bjobs: "/path/to/bjobs/wrapper"

```

The delay was enough so that the launched job was found, and the startup worked afterwards:

> [@Jupyter interactive app question - launching, but not providing link to](https://discourse.openondemand.org/t/jupyter-interactive-app-question-launching-but-not-providing-link-to/435/3):
>
> Thank you for the response. We’re running LSF 9.1.3 currently (and working on migrating to 10.x). It looks like the problem is a delay between when bsub returns the jobid for a new job, and when the bjobs command starts returning that jobid’s status. I’d been aware of this gap, though it’s usually less than a second and can run longer, obviously it’s too long, and OnDemand considers the job to have failed since it doesn’t return for that id immediately. I made a wrapper script for the bjobs …

---

<div class="post-metadata">

**Author:** ![rotaugenlaubfrosch](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rotaugenlaubfrosch/32/314_2.png) [@rotaugenlaubfrosch](https://discourse.openondemand.org/u/rotaugenlaubfrosch)\
**Post date:** [October 17, 2019, 5:34pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/4 "2019-10-17T17:34:46Z")

</div>

Hi cjh,  
Thank you very much for your answer - this looks promising.  
Do you have an example of how the wrapper script looks like?  
Rotaugenlaubfrosch

---

<div class="post-metadata">

**Author:** ![cjh](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/cjh/32/198_2.png) [@cjh](https://discourse.openondemand.org/u/cjh)\
**Post date:** [October 17, 2019, 6:17pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/5 "2019-10-17T18:17:47Z")

</div>

The wrapper I used is pretty straightforward:

```
#!/bin/bash
# Wrapper to sleep bjobs before running

SLEEP=5

OPERATION=/lsf/9.1/linux2.6-glibc2.3-x86_64/bin/bjobs

# Run
sleep $SLEEP
exec $OPERATION "$@"
```

---

<div class="post-metadata">

**Author:** ![rotaugenlaubfrosch](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rotaugenlaubfrosch/32/314_2.png) [@rotaugenlaubfrosch](https://discourse.openondemand.org/u/rotaugenlaubfrosch)\
**Post date:** [October 18, 2019, 3:39pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/6 "2019-10-18T15:39:35Z")

</div>

Thanks cjh,  
I created a wrapper script as you posted it but it doesn’t seem to help.  
In the wrapper script, bjob gets executed correctly and the running job is visible.  
However, OOD says that the job is starting, although it is already running on the node.

 ![screenshot](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/59f8864b7f2734a2bb93685ce91bd05556c48d7b.png)  
I noticed that there is an ajax request every ~10 seconds, probably to check the status of the submitted job. The wrapper script also gets executed every 10 seconds.  
Do you know how OOD checks if the job turned from pending to running?  
Thanks

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 18, 2019, 4:10pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/7 "2019-10-18T16:10:59Z")

</div>

Please look in your log directory for an indication, and post/share any relevant info from that log.

It would be somewhere like this.  
`$HOME/ondemand/data/sys/dashboard/batch_connect/sys/$VNC_APP_SUBDIR/output/$UUID/output.log`

Where `$VNC_APP_SUBDIR` is maybe lsf\_poc\_desktop and `$UUID` is the `ce25e64b...`that you’ve just shared.

Also just to clarify the situation, when you start this job, you’re saying that it sits in the `starting` state forever? Or does it sit in that state then eventually delete itself?

Also during this state, what does LSF itself say about the job? I mean from LSF’s perspective, what state is the job in, like `running` or `queued`, etc?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 30, 2019, 5:24pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/8 "2019-10-30T17:24:45Z")

</div>

@rotaugenlaubfrosch were you able to make any progress? Again, we’d want to see things from either `output.log` or `vnc.log` or both.

> [@rotaugenlaubfrosch](#):
>
> Do you know how OOD checks if the job turned from pending to running?

The job goes from `starting` to `running` state when a `connection.yml` is created in the job directory (the Session ID links to this directory in that screenshot you have). What happens is this: We boot everything up, then read the log file (vnc.log) for a particular line that indicates that everything went well. We then write this file out, at which point everything is good to go.

```bash
echo "Scanning VNC log file for user authentications..."
while read -r line; do
  if [[${line} =~ "Full-control authentication enabled for"]]; then
    change_passwd
    create_yml
  fi
done < <(tail -f --pid=${SCRIPT_PID} "vnc.log") &

```

```bash
Scanning VNC log file for user authentications...
# this is the logline right here 
Generating connection YAML file...
Launching desktop 'xfce'...

```

Hope that’s helpful.

---

<div class="post-metadata">

**Author:** ![rotaugenlaubfrosch](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rotaugenlaubfrosch/32/314_2.png) [@rotaugenlaubfrosch](https://discourse.openondemand.org/u/rotaugenlaubfrosch)\
**Post date:** [October 31, 2019, 6:54pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/9 "2019-10-31T18:54:26Z")

</div>

Hi All,  
By default, LSF runs batch jobs using `/bin/sh`.  
However, in order to interpret all commands correctly the `/bin/bash` shell must be used.  
When adding the `#!/bin/bash` line to the wrapper script, an empty line gets inserted to the top of `job_script_content.sh` which results in LSF still using `/bin/sh`.

In a nutshell, the solution was to create a `#!/bin/bash` header as shown here:

```
---
batch_connect:
  template: vnc
  header: |
    #!/bin/bash
  script_wrapper: |
    export PATH="/system/apps/TurboVNC/bin:$PATH"
    export WEBSOCKIFY_CMD="/system/apps/websockify/run"
    %s
  ...

```

I couldn’t find anything regarding that in the OOD documentation - so adding this to the docs may be helpful for others too.  
Thanks

---

<div class="post-metadata">

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\
**Post date:** [May 26, 2022, 3:25pm UTC](https://discourse.openondemand.org/t/problems-starting-vnc-on-ood-with-lsf-manager/544/10 "2022-05-26T15:25:10Z")

</div>