Ood v4 bug: Job Composer completely ignores "#SBATCH --chdir" directive in scripts

Get Help
1

Hi, I’m really hoping some OOD devs can help me out quickly on a workaround here, because I didn’t discover this change until the morning after we updated our production server. :sob: :grimacing:

This seems to be a new behavior since OOD v3.1.15

All of our documentation, and all the job-templates available for our OOD instance include a chdir directive at the top of the script. For example:

#!/bin/bash
#SBATCH --job-name=chdir_test_dev
#SBATCH --output=/home/jtb49/out-%j.txt
#SBATCH --chdir=/scratch/jtb49

echo -n "pwd --> "
pwd
echo -n "PWD --> "
echo "$PWD"

echo -n "mkdir -p ... --> "
mkdir -v -p xyz/aaa/bbb

When the above script is submitted via sbatch on the command line (jobid 26339926), all printed paths are relative to my scratch, as expected, and as the v3 behavior did it. However when I submit the script via Job Composer (jobid 26339922), all printed paths are relative to the “Script location” associated with the JC job (/home/jtb49/ondemand/data/sys/myjobs/projects/default/10)…

## contents of both CLI and JC submissions' output files:
$ grep ^ ~/out-2633992*
/home/jtb49/out-26339922.txt:pwd --> /home/jtb49/ondemand/data/sys/myjobs/projects/default/10
/home/jtb49/out-26339922.txt:PWD --> /home/jtb49/ondemand/data/sys/myjobs/projects/default/10
/home/jtb49/out-26339922.txt:mkdir -p ... --> mkdir: created directory 'xyz'
/home/jtb49/out-26339922.txt:mkdir: created directory 'xyz/aaa'
/home/jtb49/out-26339922.txt:mkdir: created directory 'xyz/aaa/bbb'
/home/jtb49/out-26339926.txt:pwd --> /scratch/jtb49
/home/jtb49/out-26339926.txt:PWD --> /scratch/jtb49
/home/jtb49/out-26339926.txt:mkdir -p ... --> mkdir: created directory 'xyz'
/home/jtb49/out-26339926.txt:mkdir: created directory 'xyz/aaa'
/home/jtb49/out-26339926.txt:mkdir: created directory 'xyz/aaa/bbb'

## the mkdir command builds directories in different places
$ find /home/jtb49/ondemand/data/sys/myjobs/projects/default/10/xyz /scratch/jtb49/xyz
/home/jtb49/ondemand/data/sys/myjobs/projects/default/10/xyz
/home/jtb49/ondemand/data/sys/myjobs/projects/default/10/xyz/aaa
/home/jtb49/ondemand/data/sys/myjobs/projects/default/10/xyz/aaa/bbb
/scratch/jtb49/xyz
/scratch/jtb49/xyz/aaa
/scratch/jtb49/xyz/aaa/bbb

Sooo… please help! I’m hoping I can do something simple like add an initializer to /etc/ood/config/apps/myjobs/initializers/ that can patch this? At least, that initializer approach is how we’ve hotwired JC in the past.

Thank you for any help here!!!

-jason buechler, NAU Monsoon

2

You can supply an initializer for this method here that specifies the workdir.

Looking through the git blame we used that flag for centers that use a different submit host. I.e., they ssh somewhere so when they submitted jobs the CWD was always their HOME.

1 Like
3

Thanks Jeff, can you confirm if this is going to be the Job Composer’s behavior from now on?

If I correctly read this line of the slurm adapter’s source, a nil workdir value won’t concat any -D option into the arguments, right? I have zero confidence in my Ruby literacy/ability but can you verify the below (myjobs-) initializer is the right idea?

# /etc/ood/config/apps/myjobs/initializers/force_nil_workdir.rb
require "ood_core"

module OodCore
  module Job
    class Script
      # Redefine the reader to always return nil,
      # effectively ignoring the value set during initialization.
      def workdir
        nil
      end
    end
  end
end

For anyone else who might be more comfortable with bash than ruby or if my stab at the above initializer is faulty… the following 3-part approach is what we’ve been using since the day we found the Job Composer was adding its own -D/--chdir argument to its sbatch invocations:

  1. add NAUHPC_OOD_MYJOBS_ENV=1 to /etc/ood/config/apps/myjobs/env
    Our wrapper-script will only strip the -D when it sees this variable.

  2. append to v2.job inside /etc/ood/config/clusters.d/YOURCLUSTER.yml…

# force OOD to use our sbatch wrapper script
    bin_overrides:
      sbatch: "/usr/local/bin/sbatch"
  1. Place (executable) wrapper script on OOD host at override-target; e.g.:
#!/bin/bash
# strip the -D option from OODs sbatch invocation
if [[ "$NAUHPC_OOD_MYJOBS_ENV" != 1 ]]; then
    exec sbatch "$@"
else
    declare -a new_args

    while [[ "$#" -gt 0 ]]; do
        case "$1" in
            -D)
                shift
                if ! [[ "$1" =~ ^-.*$ ]]; then
                    shift
                fi
                ;;
            *)
                :
                ;;
        esac
        new_args+=( "$1" )
        shift
    done

    exec sbatch "${new_args[@]}"
fi
  1. Remember you’ll need to restart your PUN from the Develop menu:
    </> > Restart Web Server