Interactive "module avail" button in form.yml and Job Viewer

Hi folks,

In our institution, most applications are installed as modules.

When users launch an app in OOD, say Jupyter, it is difficult for them to load additional modules as they have to spawn a shell, and get the correct name of the module from the ssh terminal, and then put it in their application form in OOD.

My suggestion: In form.yml, give users a text field and a button. Once the button is clicked, the input of the text field would be passed to module avail <input_here> and the output of this command would be displayed to the user.

Additionally, I was thinking something similar would be appropriate for the Job Viewer (now Project Viewer?) as users have to hop to a shell to get the module names for their batch submission scripts as well.

Thanks for considering this! :smile:

Edit: We are using Environment Modules instead of lmod. Potentially, this feature would be cross-compatible.

In one of our apps, we do have a custom text field that’s intended for module loads, environment activations, etc.

In form.yml.erb, we have:

    label: Custom Pre-load Commands
    cacheable: false
    widget: text_area

In we have:

<%- if not context.load_commands.blank? -%>
load_commands="<%= context.load_commands.encode(context.load_commands.encoding, universal_newline: true) %>"
load_commands="$(grep -Ev '^$|^#' <<< "$load_commands")"
export load_commands
<%- end -%>

And finally, in we have:

while IFS= read -r cmd; do 
  eval "$cmd"
done <<< "$load_commands"

I stole the implementation from another site, and I wish I remembered whom. If you’re from that site, please feel free to claim credit!

1 Like

Sorry, re-read your post and this isn’t the problem you wanted solved. :slight_smile:


Thank you for this! This looks great, I’m already doing something similar, but this looks like a better approach.

Does this give the user the ability to query available modules? Or does it only load modules if users know the module name beforehand?

Cheers! :grin:

Unfortunately, it won’t report available modules, like you requested. The users still have to know the module names. If they don’t know, they would have to get them from module avail in the terminal.

1 Like

We have installed jupyter-lmod into our JupyterLab app’s default Python virtual environment which displays which modules are loaded and allows users to search and load available Lmod software modules. Although you do have to make sure you select compatible Core modules from the displayed list which shows ‘module avail’ based on the modules loaded in the default environment.

1 Like

It’s not exactly what you’re looking for, but we have a piece of code that we use for our RStudio Server app that shows the users which modules they have loaded in the info card after the job runs. This could definitely be adapted to show users which modules are available.

In template/ (you could swap the command with module avail instead of module list):

export module_file="<%= session.staged_root %>/modules.txt"
module -t list 2> $module_file

In info.yml.erb (setup as an accordian so users can expand and minimize as desired):

<div id="faq-accordian-<%= id %>">
if (File.exist?("#{staged_root}/modules.txt"))
  <div class="card">
    <div class="card-header" id="#loaded-modules-header-<%= id %>">
      <h5 class="mb-0">
        <button class="btn btn-link" data-toggle="collapse" data-target="#loaded-modules-<%= id %>" aria-expanded="false" aria-controls="loaded-modules-<%= id %>">
          Currently loaded modules
    <div class="collapse" id="loaded-modules-<%= id %>" aria-labelledby="#loaded-modules-header<%= id %>" data-parent="#faq-accordian-<%= id %>">
      <div class="card-body">
    "#{staged_root}/modules.txt", "r") do |f|
                f.each_line do |line|
                  if (!line.strip.empty?)
            <li><%= line %></li>

The only issue is that this information is available after the job has started, not as you’ve suggested, before. This is potentially more useful for R because it can interact directly with the module system, so it helps to have that real-time information. But with @RonRahaman’s implementation above (we also do something similar), a user could see what modules were available to them at the end of a job, then start a new one using something from that list.

1 Like

You could use an initializer to enumerate and parse the output of module avail into a list (the ruby equivalent). The OOD folks will correct me but I think that runs when the dashboard is loaded. You can then reference this “list” to populate an option dropdown widget.

You can do the same to initialize accessible partitions or even reservations on a per user basis. Then in your forms, you can then dynamically populate the dropdown widget.

I have had this as a project on my back burner for a while but have not really gotten into it since we also have to take into account different OS’s and such but its definitely possible. I might be able to dig up some code examples.

1 Like

Yep, that’s what we do for populating dropdowns of Slurm accounts, reservations, etc. Reference: # Suggestions on generating dynamic form data - #6 by Micket

Example from /etc/ood/config/apps/dashboard/initializers/ood.rb :

# Cache the data when possible, rather than running the same commands every
# time. Should be able to have the user restart their PUN if there's an
# immediate update.
class SlurmData
  def self.accounts
    username = Etc.getlogin
    accounts = []
    IO.popen 'sacctmgr --noheader --parsable show assoc format=account,user' do |io|
      io.each do |line|
        acc = line.split('|')
        if acc[1] == username
    return accounts


and the corresponding field in a form.yml:

    widget: "select"
    label: "Job account"
    options: [<%= SlurmData.accounts.sort.join(', ') %>]
    help: |
      Select the job account assigned to your class or research project. 

@renfro Heads up on the SlurmData example: that implementation will make each single page load very significantly slower. When slurm account information was added into OOD itself, it was done so behind a caching mechanism so it doesn’t probe sacctmgr every time. If you haven’t switched to the built-in slurm data i highly recommend it, it’s sooo much nicer.

Not sure if the caching mechanism is available in the initializer itself, but it would absolutely be worth doing that.

On the topic i found that pretty much every single user i had need some kind of customization; their own containers, their own venv on top of modules, combining a bunch of different modules etc.
I thought it was hopeless to even attempt defining any menu system that was flexible enough to support all that freedom so I simple opted to allowing users to bash scripts that i source in my jupyter app.
In my form i i let them pick their own environment by a simple glob:

    widget: "select"
    label: "Runtime"
      System provided:
<% glob_prettify_lookup("/apps/portal/jupyter/*.sh").each do |pretty_path, path| %>
        - [ "<%= pretty_path %>", "<%= path %>" ]
<% end %>
      User provided:
        - [ "~/portal/jupyter/*.sh",  "x", disabled: true ]
<% glob_prettify_lookup("#{Dir.home}/portal/jupyter/*.sh").each do |pretty_path, path| %>
        - [ "<%= pretty_path %>", "<%= path %>" ]
<% end %>

where my system wide /apps/portal/jupyter/*.sh files just work as inspiring examples to users of what they can do.
An advantage is that users can save multiple customized environments (which some actually do).
Downside of this total freedom is course that they get total freedom to shoot themselves in the foot, which also many do.

1 Like

Found the link to the builtin smart-attribute (which replaces the need to do SlurmData in the initializer). It was straight up the Rails.cache that was used:

so I would assume that works just as easily to implement in the initializers.

While searching for this piece of code, I also found this

which sounds to me like exactly what you are asking for @walidabualafia . If so, good news it’s been supported for almost 2 years :slight_smile:

1 Like

we have a slightly different solution, where we generate a json from the lmod cache, have some python code to convert the modules in a specific JSON output (we have cluster/os+architecture dependencies via MOUDLEPATH and only want to show modules for the selected clusters), and then use memcache caching to show the avail modules in the app dropdown. there is also some javascript sauce to change modules based on selected cluster.

anwyay, without memcache caching shared for all users, it is simply too slow. disadvantage is that if you always show all modules, so can’t hide any modules that are only shown to certain users.
(we also use this caching for anything that interacts with the system, like showing reservations. that suffers more from the sensitive data part, but only expose data that can be retrieved by all users. so we eg show a list of all possible reservations, in the hope this narrows it down for the user looking for his/her reservation. that list can be retrieved by anyone, so no leaking of info there)

having a way to encrypt/decrypt the memcache paths and values, we could go further and show real per-user data from the system.

if only i could spend some time on it…

1 Like

@stdweird exactly what I was looking for! This sounds awesome. I want the users to be able to find the module names without the extra hop to the terminal. Seems rather inconvenient having to open a terminal (at all) for a user who is opting to use OnDemand as their main HPC platform. Your solution will go a long way!

Do you have the application publicly accessible? Would I be able to look at a repo to get some ideas? I’d be happy to contribute to it as well. :grin:

Thank you all for the great responses! I did not expect this thread to gain this much attention.

@walidabualafia this is part of 3 separate private repos. i need to think how to make this available,and explain how it all works together. you can probably figure it out, but it’s way faster if i show it i guess.

if there is wider interest, i can setup some online meeting, and find a way to hand the code over. if someone from the ondemand project itself could also join, they can say if they are interested in any of the code (or not :wink:

1 Like

Hi Stijn,
this may be a good topic for the monthly Tips and Tricks call. Any interest in doing that?

hi martin,

sue, that might also work. i see it’s “first thursday of the month”, so next week thursday at 19h00 CET ;). i’ll try to make it work. i can probably demo our site changes, and then go over the code.
do you expect formal slide deck or soemthing?


Hi Stijn, thanks, yes, it’s next Thursday. We are just figuring out what to do in this call so let us know if you can make it. Some slides are usually helpful as it can help people to navigate what is presented, but, it does not have to be exhaustive, just a handful of slides to list the main points is fine. Please, let me know if you can make it ASAP, we are going to make a plan for the call today or on Monday at the latest.

hi martin,

yes next thursday is ok for me.


Great, thanks Stijn. Would you please write a title for the talk and a few sentences abstract which I can send in the announcement on Monday or Tuesday. Looking forward to this, I think we may have some good discussion on the different approaches sites take on serving the modules.

@mcuma @stdweird

Thank you for facilitating and presenting. This topic should be very interesting indeed. Very excited! :smile: