Job Composer Additional Fields

Feature Requests and Roadmap Discussion
1

We have a punchlist of changes we want to implement for the job composer, to begin with adding a few fields to the Job Options.

Right now, I’m trying add: data.cores (which will get passed as -c to sbatch or srun). At some point, I’d like to be able to at least hard-code (a hack, I know) partition names - I realize this isn’t in keeping with some of the design intent - a dynamic query of the particular batch system to enumerate partitions is probably preferable - but for now, a quick and dirty hack is what I’m trying to implement.

I’d love a pointer to where in the code this is buried…I’ve been spending a bit of time trying to read the code itself - but all I find is @request_job_data inside of workflow.js.coffee, and a few others in and I’m sure that the data fields have to be enumerated in the sqlite db as well. A hint about where to start pulling on this thread would be deeply appreciated!

2

Check this out for an example of how to modify the current Job Composer app: https://github.com/OSC/ood-myjobs/pull/265/files

The changes in that PR are as follows:

  1. Pass in workflow instance as an argument to the ResourceMgrAdapter so you can use attributes on the workflow instance

Then you follow these steps to add a new custom field to the Job Composer:

  1. Add new form fields to app/views/workflows/_edit_form_job_attrs.html.erb
  2. Add the corresponding attributes to the array of “accessor” names for the JSON store on the app/models/workflow.rb
  3. Update the ResourceMgrAdapter to use the values for these in OodCore::Job::Script submit the job (see app/models/resource_mgr_adapter.rb)

You can see the options for different job submission parameters here: https://github.com/OSC/ood_core/blob/a87f59ea9d24cdcbe505c9ab859bcb120bae9951/lib/ood_core/job/script.rb#L103-L125 and the specific implementation here (converting the Submit script to command line args) https://github.com/OSC/ood_core/blob/a87f59ea9d24cdcbe505c9ab859bcb120bae9951/lib/ood_core/job/adapters/slurm.rb#L252-L312

We haven’t added a sensible abstraction yet for requesting nodes and cores so you need to use the native option, which lets you just specify an array of command line arguments to sbatch. So in this case you might do something like this in the ResourceMgrAdapter if your custom field name is “data_cores”:

script = OodCore::Job::Script.new(
  content: script_path.read,
  accounting_id: account_string,
  native: ['-c',  workflow.data_cores]
)

If you drop this file in config/initializers/ (or /etc/ood/apps/myjobs/initializers/) you will see the arguments passed to execve when sbatch is called by the Job Composer. This will appear in the log file at /var/log/nginx/USERNAME/error.log. You can see the resulting command line that is excecuted when submitting a job.

About the JSON store: since these attributes will be stored in a single column in the database that already exists, you can add more attributes without requiring users to migrate their database. We put it there as an insurance policy to change requests like these. Of course the drawback is you can’t do optimized queries on these fields.

1 Like
3

I opened related https://github.com/OSC/ood-myjobs/issues/273

4

Scott,

Have you had any success? It just occurred to me that the example code modifications that I provided might have not covered everything. You probably would like the job options currently selected to appear in the details pane to the right when a user selects a job in the table. If thats the case I can update my example with updating this view as well.

Also, as soon as I get time I want to add a feature to properly support this type of customization, and your advice would be helpful. For sure whatever feature we implement should support your use case.

Thanks,
Eric