Jupyter + Spark "port_used_*: command not found"

Hello,

I’m trying to setup Spark to work with Jupyter notebooks at our site. I have several issues I’m working through, but one is that I keep seeing these error messages in the output.log file (full job path replaced with “…”):

.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found
.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found

I see that these are all defined in job_script_content.sh. Is there any reason these functions shouldn’t be visible to my before.sh and script.sh?

Thanks,
Nick

Just a quick update: Manually running source_helpers at the beginning of each script and inside the Spark worker script seems to have solved this problem.

Although, I see that this is already being run in job_script_content.sh. On further inspection, it looks like some functions within source_helpers are exported in their own right, while the port_used_* commands are not. For example, port_used is exported:

  port_used () {
    local port="${1#*:}"
    local host=$((expr "${1}" : '\(.*\):' || echo "localhost") | awk 'END{print $NF}')
    local port_strategies=(port_used_nc port_used_lsof port_used_bash port_used_python port_used_python3)

    for strategy in ${port_strategies[@]};
    do
      $strategy $host $port
      status=$?
      if [[ "$status" == "0" ]] || [[ "$status" == "1" ]]; then
        return $status
      fi
    done

    return 127
  }
  export -f port_used

but port_used_nc is not:

  port_used_nc(){
    nc -w 2 "$1" "$2" < /dev/null > /dev/null 2>&1
  }

Is there a reason why source_helpers would export port_used but not the specific port_used_* subroutines that it depends on?

I’m sure there’s something I’m missing, but I’m going to keep the thread going in hopes that it’s helpful to someone else!

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.