Open Composer: Web application to generate and submit batch job scripts

Allow me to introduce Open Composer, a web application we are developing. Open Composer operates on Open OnDemand as a Passenger application.

Open Composer is designed to simplify batch job submissions, a core task for HPC cluster users. It offers job management capabilities, including status monitoring and job deletion, promotes parameter reuse, and enables seamless launching of related Open OnDemand applications with a single click.

For more information, please visit:

This application was inspired by discussions on the Open OnDemand discourse. It dynamically generates batch job scripts, similar to BYU Job Script Generator, and streamlines web form creation for various applications, similar to Drona Composer.

We would greatly appreciate your feedback and comments.

Best regards,

6 Likes

I pulled this down and booted it up. It ‘just worked’ at OSC without configuration, so that’s nice.

It looks good!

1 Like

Install is super easy for sure, followed the documentation and edited the conf.yml.erb. I need to configure the partition definitions and test job submission. I’m really looking forward to seeing and using the History feature. Thanks for the post!

Kenny

1 Like

Hi Nakao-san,

On behalf of the OnDemand team, thank you so much for building this application and sharing it here on Discourse. We are very impressed with the application and the detailed documentation you provided. We greatly appreciate seeing the OnDemand user community address common challenges and share their solutions.

We installed it today on our Ohio Supercomputer systems and have made it available to other internal teams to try out. Our OnDemand team is currently in the process of building our Project Manager, which will serve as a replacement and enhancement to the Job Composer. What’s fantastic about Open Composer is that it appears to streamline proper job submission, offer better monitoring, and provide more control in innovative ways within OnDemand! While the goals of the Project Manager are slightly different, they are highly complementary to Open Composer’s capabilities.

I personally appreciate how accessible everything is (job history, batch script editing, email notifications), as well as the readability and thoroughness of your GitHub documentation and poster. Plus, the purple color theme is a great touch :smile:

Thank you again! We will let you know if we receive any feedback from our internal teams after they’ve had a chance to test it.

Emily

2 Likes

It looks like so good. :+1:

1 Like

Does Open Composer require specific versions of OOD for an install ? We are running 3.0.3 - is this sufficient ?

1 Like

Open Composer should not depend on the OOD version, but if it is too old, it is unknown. I have confirmed that it works with OOD version 3.1.10 and 4.0.0.

Thanks,

Hi Everyone

@mnakao First a big Thank for this amazing tool.

It’s not working properly for me, certainly due to the format of my forms.

Here an example:

---
cluster: "Onyxia"

title: "MATLAB on Baobab"
attributes:
  num_cores:
    widget: "number_field"
    label: "Number of cores"
    value: 1
    help: "Maximum of 6 cpu on public-interactive-cpu partition"
    min: 1
    max: 6
    step: 1
  bc_num_hours:
    value: 1
    min: 1
    max: 8
    step: 1
    help: "Maximum public-interactive-cpu walltime: 8 hours"

[...]

form:
  - auto_modules_MATLAB
  - num_cores
  - bc_num_hours
  - memtask
  - auto_accounts
  - auto_queues
  - bc_vnc_resolution
  - bc_email_on_started

I got the error message:

App 2502393 output: 37.170.42.108 - - [23/Jan/2025:12:15:34 +0100] "GET /pun/dev/OpenComposer/MatLab HTTP/1.1" 500 30 0.0186
App 2502393 output: 2025-01-23 12:18:19 - NoMethodError - undefined method `each_key' for ["modules", "modulesextra", "extra_jupyter_args", "auto_accounts", "auto_queues", "bc_num_hours", "cpu", "memory", "gpu", "bc_email_on_started"]:Array
App 2502393 output: 
App 2502393 output:         invalid_keys = @body["form"].each_key.reject { |key| key.match?(/^[a-zA-Z][a-zA-Z0-9_]*$/) }

Because form is a list and not a dictionary.

like in the example: sample_apps/Slurm/form.yml

---
form:
  partition:
    widget: select
    label: Partition
    options:
      - [ gpu1, gpu1, set-label-cores: Number of cores (1 - 72),  set-max-cores: 72,  set-label-memory: Memory (up to 186GB),  set-max-memory: 186,  set-label-time_1: Maximum run time hours (0 - 3), set-max-time_1: 3 ]
      - [ gpu2, gpu2, set-label-cores: Number of cores (1 - 36),  set-max-cores: 36,  set-label-memory: Memory (up to 93GB),   set-max-memory: 93   ]
      - [ mem1, mem1, set-label-cores: Number of cores (1 - 224), set-max-cores: 224, set-label-memory: Memory (up to 5020GB), set-max-memory: 5020, set-label-time_1: Maximum run time hours (0 - 3), set-max-time_1: 3 ]
      - [ mem2, mem2, set-label-cores: Number of cores (1 - 56),  set-max-cores: 56,  set-label-memory: Memory (up to 1500GB), set-max-memory: 1500 ]
      - [ ondemand-reserved, ondemand-reserved, set-cores-label: Number of cores (1 - 8), set-max-cores: 8, set-label-memory: Memory, set-label-time_1: Maximum run time hours (0 - 720), set-max-time_1: 720 ]
  
  cores:
    widget: number
    value:  1
    
  time:
    widget:   number
    label:    [ Maximum run time hours (0 - 24), Maximum run minutes (0 - 59) ]
    size:     2
    value:    [  1,  0 ]
    min:      [  0,  0 ]
    max:      [ 24, 59 ]
    step:     [  1,  1 ]

I use the attributes to fix all my variabless and the form to list what should be display in interactive app selection.

But while I am writting this post, The great Light is lighting me up… There is no sense to request an interactive app with Job Composer which use sbatch. Am I right ?

So my question, should we create distinct apps and interactives app directories and then create a form compatible with JobComposer ?

Thank your for your Lights :pray:

1 Like

@garadar
Thank you for trying it so quickly.

Sorry for the confusion, but the form.yml syntax for Open OnDemand is different from the one for Open Composer. See HERE for Open Composer syntax.

There is no sense to request an interactive app with Job Composer which use sbatch. Am I right ?

(When you say Job Composer here, you mean Open Composer, right?)
Yes, that’s correct.
Open Composer has been developed to submit only batch jobs. If you run interactive apps from Open OnDemand, it will set up reverse proxies and other settings, but Open Composer does not have such functionality.

So my question, should we create distinct apps and interactives app directories and then create a form compatible with Job Composer ?

We assume that the application directory for Open OnDemand is different from the application directory for Open Composer. Please note that Open OnDemand form.yml will not work as Open Composer form.yml - they are not compatible.

Best,

1 Like

Many thank for clarity.

Yes I confused OpenComposer with Jobcomposer

1 Like

Thanks for this. I tried it it looks good and is easy to use. Only, I would prefer the path selector widget to work differently. It now creates a modal popup and if the folder contains a lot of files the modal overflows and you always have to scroll to the bottom to click on the select button. I would prefer a modal with fixed height and a scroll bar in the modal body to scroll through the file list. Other than this, I think it is a great app.

1 Like

Thank you for your comments and pull requests.
It has been merged.

Thank you for your cooperation.

@mnakao Open Composer looks very nice and has some great features Thanks for acknowledging Drona Composer ( Drona Wiki, YouTube video, that is us :slight_smile: ). I think you are correct; there are some similarities between OpenComposer and Drona Composer with respect to environments but I do think they have a bit different goals. Open Composer provides a powerful graphical interface to specify job and software parameters, where the overarching goal of Drona is to automate the process of running workflows on compute clusters as much as possible.

FYI, I did see some inaccuracies in your description of Drona in your paper. I can explain more in dm if you are interested.

1 Like

@pennings
Thank you very much for taking the time to review my paper written in Japanese. Drona Composer is a great application and was a great reference for developing Open Composer.

We sincerely apologize for any mistakes that may have been present. As we are planning to translate this paper into English when the opportunity arises, we would greatly appreciate it if you could kindly let us know about any errors you noticed via DM.

1 Like