# Simple Question: Execute python code on a LSF submit host

**URL:** <https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560>\
**Category:** Get Help\
**Tags:** question\
**Created:** [June 12, 2024, 2:57pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560 "2024-06-12T14:57:39Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 2:57pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/1 "2024-06-12T14:57:39Z")

</div>

Hi  
Everyone. I run LSF v10 and ondemand-3.1.4-1.el8.x86\_64. I can’t load a module (for this example we will use python) and execute it on the submit host. I can just run a basic batch code from the gui and execute it on the submit host. (example: echo “hello world!”). I have 2 problems. The environment won’t copy over, but I can bypass this by “copy environment” and it works. But more importantly it won’t execute the python script. It says it can’t find it. And the reason it can’t find it, is that it’s looking in my home directory to execute it but in reality it’s in that really long “script location”. Does anybody have any suggestions? Someone suggested to use submit.yml.erb. That’s fine but I just want to this simple thing to work universally.  
Thanks,  
–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 3:17pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/2 "2024-06-12T15:17:13Z")

</div>

It seems like you’re using the job composer so the note about `submit.yml.erb` only applies to batch connect applications.

When I’m in the job composer and click on `Job Options` I can see there’s a dropdown menu for which script to run -

 ![image](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/8/8f4a9d0c423014b3fcd02041a1c5c8db97c38817.png)

This is the script I’m trying to submit to the scheduler. Are you saying the python script is here in this dropdown and LSF can’t find it?

Or rather that you’re submitting a shell script that is then invoking your python script and _it_ (the shell script) can’t find the python script?

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 4:46pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/3 "2024-06-12T16:46:53Z")

</div>

Hi,  
So let’s just assume submit.yml.erb doesn’t exist. Because really I made it and it doesn’t really work. So yes to your second question:  
Or rather that you’re submitting a shell script that is then invoking your python script and _it_ (the shell script) can’t find the python script?  
Here is the example:  
#!/bin/bash  
#BSUB -q normal  
#BSUB -J pythonjob #LSF Job Name  
#BSUB -o pythonjob.%J.out #Name of the job output file  
#BSUB -e pythonjob.%J.out #Name of the job error file

### – send notification at start –

#BSUB -B

### – send notification at completion –

#BSUB -N  
module load python/3.10

python --version  
python hello.py

Error is, if I say “copy environment” in the gui:  
Python 3.10.5  
python: can’t open file ‘/home/david/hello.py’: [Errno 2] No such file or directory

As you can see it did load the module but tried to execute the python script in /home/dweise  
but the script location per the gui was:  
/home/david/ondemand/data/sys/myjobs/projects/default/70

–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 5:01pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/4 "2024-06-12T17:01:03Z")

</div>

OK thanks for that explaination. I can’t replicate on Slurm. We seem to be submitting the job from the directory of that contains the scripts and input.

So if I add things like this in my shell script (`sequential_job.sh`)

```bash
echo "CWD is: $CWD"
echo "submit dir is $SLURM_SUBMIT_DIR"
echo "pwd is $(pwd)"

```

I get output like this:

```auto
CWD is: 
submit dir is /users/PZS0714/johrstrom/ondemand/data/sys/myjobs/projects/default/34
pwd is /users/PZS0714/johrstrom/ondemand/data/sys/myjobs/projects/default/34

```

So the script should start with a `pwd` of that script/job.

Looking at the code for copying environments - I see we’re passing the `-env` flag will all the keys.

If I try to replicate on Slurm, and I dump my `env` from withing the shell script - I see that my `PWD` is always the same. Could it be that LSF is preserving this environment variable `PWD` and Slurm is resetting it? (If LSF is preserving it, it’s likely `/var/www/ood/apps/sys/dashboard` and since that doesn’t exist during the jobs execution it falls back to `HOME`?)

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 5:06pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/5 "2024-06-12T17:06:05Z")

</div>

Oddly enough - the documentation says that `PWD` is not propagated to the job.

> **[\-env](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=o-env)**
>
> Controls the propagation of the specified job submission environment variables to the execution hosts.

And just to be clear - when I tested with Slurm I tested both with and without the copy environment flag and got the same `PWD`.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 5:56pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/6 "2024-06-12T17:56:53Z")

</div>

Hi,  
Great Idea to include env. Summed up here are some of the env variables:  
LS\_EXECCWD=/home/david/  
PWD=/home/dweise  
Accoriding to IBM

- LS\_EXECCWD: Sets the current working directory for job execution.  
So, technically it’s working correctly. hello.py isn’t there. But how to change it that the script location is the same sas LS\_EXECCWD.  
Or in slurm case slurm\_submit\_dir = pwd  
Where would you set that?  
–David

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 5:58pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/7 "2024-06-12T17:58:13Z")

</div>

Hi,  
And I want to also point out that env had no mention what so ever of the real slurm\_submit\_dir, aka in the above message: /users/PZS0714/johrstrom/ondemand/data/sys/myjobs/projects/default/34  
–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 6:14pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/8 "2024-06-12T18:14:11Z")

</div>

Yea I think `SLURM_SUBMIT_DIR` in Slurm is the same as `LS_EXECCWD` in LSF.

Can we remove OOD from this for a second and run a test from the command line?

What I’d like to test is providing the full path of the script and the relative path. I’m not 100% how the environment fits into this -but I’m guessing it doesn’t at all. Rather that it’s a separate problem.

`bsub /full/path/to/the_file.sh` and `bsub the_file.sh` — I’d like to see how LSF behaves when you supply the full path (in any directory AND in the correct directory) vs when you are in the directory and submit the job with the relative path (note there’s no `./` there).

I checked our code base and we do change directory into the correct directory when submitting the job. We are however supplying the full path of the file - so I’m thinking that’s what’s throwing LSF off. If it were the relative path maybe LSF would behave differently.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 6:22pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/9 "2024-06-12T18:22:54Z")

</div>

Sorry I’m getting ahead of myself. Rereading the code (or scrutinizing it) we’re actually passing the script through `stdin`. So to actually replicate the behavior here - you’d need to `cat the_file.sh | bsub`. We do definitely change directories into the correct directory before submitting - we just read the file and pass that to bsub through `stdin`.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 6:56pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/10 "2024-06-12T18:56:31Z")

</div>

Hi,  
So the ondemand server isn’t a submit host. ondemand submits to the login host. Let’s call it vulcan. vulcan then does the submitting. So when I tell ondemand to open the directory it does so and i can do a bsub \< batchsubscript.sh. And that will work. And it will dump the output into the “script location” directory.  
–David

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 6:58pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/11 "2024-06-12T18:58:22Z")

</div>

Hi,  
Ok, but since the directory is dynamic, is there a variable for “script location”, so I could put in the code:  
cd $SCRIPT\_LOCATION  
python ./hello.py

–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 7:05pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/12 "2024-06-12T19:05:34Z")

</div>

> [@dweise1776](#):
>
> vulcan then does the submitting

Now it’s all starting to make sense. OK - well if you’re sshing into another machine, then you are in-fact submitting the job from your `$HOME` and the fact that we `cd` into the correct directory before submitting (or sshing to submit) does not do anything for you.

> [@dweise1776](#):
>
> Ok, but since the directory is dynamic, is there a variable for “script location”, so I could put in the code:

I don’t think so, because even if we set the environment on the OOD machine - we ssh somewhere else and thus get an entirely new environment.

I’ll need a minute to contemplate what solution could work for you. I’m not entirely sure right off the top how to deal with this. Obviously you can setup the OOD machine to be a submit host and I think stuff would start working for you there.

But if you must ssh to a submit\_host then I’ll need to think of a workaround for you.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 7:17pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/13 "2024-06-12T19:17:34Z")

</div>

Hi,  
Well it used to be a submit host but when you login it loads .bashrc and most people in this environment have a preloaded modules etc etc. So the modules provided by ondemand rpm conflicted with our nfs mounted modules command. I work around would be good.  
–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 7:35pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/14 "2024-06-12T19:35:56Z")

</div>

> [@dweise1776](#):
>
> So the modules provided by ondemand rpm conflicted with our nfs mounted modules command

I see - I’m finding now that `scl-utils` sets up a module system which is unfortunate.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 8:37pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/15 "2024-06-12T20:37:20Z")

</div>

Hi,  
So I made the ondemand server a submit server. I finally got it to submit a job from the gui. That being said it still have same problems. Actually we have a new one:  
/home/david/.lsbatch/1718224327.29911075.shell: line 10: module: command not found  
Python 3.10.5  
python: can’t open file ‘/home/david/hello.py’: [Errno 2] No such file or directory

BUT!!! now that it’s a submit server.  
If I just use the shell and do:  
bsub \< submitscript.sh

Works fine.  
–David

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 8:51pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/16 "2024-06-12T20:51:11Z")

</div>

Hi,  
Do you know anybody that has this running on a LSF 10.1 system?  
–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 12, 2024, 8:56pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/17 "2024-06-12T20:56:11Z")

</div>

Only through inference answering questions here. I’m looking at the LSF documentation - and can’t find how they build environments and so on. To get modules working, yes you could source /etc files, but you’d have to do it in every script all the time which is not a very good UX. So I’m looking here on this site to see if anyone’s seen the same issue with batch jobs and the job composer.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 12, 2024, 9:58pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/18 "2024-06-12T21:58:56Z")

</div>

Hi,  
Yes, that is why we have modules on an NFS mount and it’s in our bashrc.  
What about OOD\_DATAROOT? and is there a variable for “script location?”  
–David

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 13, 2024, 2:07pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/19 "2024-06-13T14:07:38Z")

</div>

There is no environment variable for script location. I’m not sure how the `OOD_DATAROOT` can help you - if you can’t find modules in your $HOME, you won’t be able to find them in some other directory either.

---

<div class="post-metadata">

**Author:** ![dweise1776](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/dweise1776/32/1295_2.png) [@dweise1776](https://discourse.openondemand.org/u/dweise1776)\
**Post date:** [June 13, 2024, 3:00pm UTC](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560/20 "2024-06-13T15:00:44Z")

</div>

Hi,  
So let us take a step back. I figured out how to get modules to work. I just click “copy environment” The real problem is that when you execute a batch script is says it can’t find the script. example:  
python hello.py - can’t find hello.py.  
Why?  
Because hello.py is in the script location and not at home or $H0ME where LSF is looking. So we just need to tell LSF that hello.py is in “script location”. Can we change the “script location” to $HOME. Some posts suggest if I change OOD\_DATAROOT that might help the situation but isn’t a long term solution.  
–David

[Next page](https://discourse.openondemand.org/t/simple-question-execute-python-code-on-a-lsf-submit-host/3560.md?page=2)