# Jupyter + Spark "port\_used\_\*: command not found"

**URL:** <https://discourse.openondemand.org/t/jupyter-spark-port-used-command-not-found/1346>\
**Category:** Get Help\
**Created:** [March 2, 2021, 7:35pm UTC](https://discourse.openondemand.org/t/jupyter-spark-port-used-command-not-found/1346 "2021-03-02T19:35:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://discourse.openondemand.org/u/ndusek)\
**Post date:** [March 2, 2021, 7:35pm UTC](https://discourse.openondemand.org/t/jupyter-spark-port-used-command-not-found/1346/1 "2021-03-02T19:35:10Z")

</div>

Hello,

I’m trying to setup Spark to work with Jupyter notebooks at our site. I have several issues I’m working through, but one is that I keep seeing these error messages in the output.log file (full job path replaced with “…”):

```nohighlight
.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found
.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found

```

I see that these are all defined in `job_script_content.sh`. Is there any reason these functions shouldn’t be visible to my `before.sh` and `script.sh`?

Thanks,  
Nick

---

<div class="post-metadata">

**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://discourse.openondemand.org/u/ndusek)\
**Post date:** [March 2, 2021, 8:27pm UTC](https://discourse.openondemand.org/t/jupyter-spark-port-used-command-not-found/1346/2 "2021-03-02T20:27:01Z")

</div>

Just a quick update: Manually running `source_helpers` at the beginning of each script and inside the Spark worker script seems to have solved this problem.

Although, I see that this is already being run in `job_script_content.sh`. On further inspection, it looks like some functions within `source_helpers` are exported in their own right, while the `port_used_*` commands are not. For example, `port_used` is exported:

```bash
  port_used () {
    local port="${1#*:}"
    local host=$((expr "${1}" : '\(.*\):' || echo "localhost") | awk 'END{print $NF}')
    local port_strategies=(port_used_nc port_used_lsof port_used_bash port_used_python port_used_python3)

    for strategy in ${port_strategies[@]};
    do
      $strategy $host $port
      status=$?
      if [["$status" == "0"]] || [["$status" == "1"]]; then
        return $status
      fi
    done

    return 127
  }
  export -f port_used

```

but `port_used_nc` is not:

```bash
  port_used_nc(){
    nc -w 2 "$1" "$2" < /dev/null > /dev/null 2>&1
  }

```

Is there a reason why `source_helpers` would export `port_used` but not the specific `port_used_*` subroutines that it depends on?

I’m sure there’s something I’m missing, but I’m going to keep the thread going in hopes that it’s helpful to someone else!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [May 19, 2022, 1:36pm UTC](https://discourse.openondemand.org/t/jupyter-spark-port-used-command-not-found/1346/3 "2022-05-19T13:36:38Z")

</div>

Closed on May 19, 2022, 1:36pm UTC

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.