Jupyter + Spark "port_used_*: command not found"

[ndusek (Nick Dusek)]

Hello,

I’m trying to setup Spark to work with Jupyter notebooks at our site. I have several issues I’m working through, but one is that I keep seeing these error messages in the output.log file (full job path replaced with “…”):

.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found
.../script.sh: line 5: port_used_nc: command not found
.../script.sh: line 5: port_used_lsof: command not found
.../script.sh: line 5: port_used_bash: command not found
.../script.sh: line 5: port_used_python: command not found
.../script.sh: line 5: port_used_python3: command not found

I see that these are all defined in job_script_content.sh. Is there any reason these functions shouldn’t be visible to my before.sh and script.sh?

Thanks,
Nick

[ndusek (Nick Dusek)]

Just a quick update: Manually running source_helpers at the beginning of each script and inside the Spark worker script seems to have solved this problem.

Although, I see that this is already being run in job_script_content.sh. On further inspection, it looks like some functions within source_helpers are exported in their own right, while the port_used_* commands are not. For example, port_used is exported:

  port_used () {
    local port="${1#*:}"
    local host=$((expr "${1}" : '\(.*\):' || echo "localhost") | awk 'END{print $NF}')
    local port_strategies=(port_used_nc port_used_lsof port_used_bash port_used_python port_used_python3)

    for strategy in ${port_strategies[@]};
    do
      $strategy $host $port
      status=$?
      if [[ "$status" == "0" ]] || [[ "$status" == "1" ]]; then
        return $status
      fi
    done

    return 127
  }
  export -f port_used

but port_used_nc is not:

  port_used_nc(){
    nc -w 2 "$1" "$2" < /dev/null > /dev/null 2>&1
  }

Is there a reason why source_helpers would export port_used but not the specific port_used_* subroutines that it depends on?

I’m sure there’s something I’m missing, but I’m going to keep the thread going in hopes that it’s helpful to someone else!