Autogenerating module list in OnDemand interactive apps

In an effort to be more flexible with supporting available versions of programs, I wrote a small piece of Ruby code into each interactive app to list all the available versions of the program the interactive app runs, as (in form.yml.erb):
<%
app_name = “Abaqus”
mod_name = “abaqus”
template_root = “/var/www/ood/apps/templates/”
modulecmd = template_root + ‘getmodules.sh’ + " " + mod_name
modules = %x[ #{modulecmd}]
%>
attributes:
version:
widget: select
label: <%= app_name %> version
help: “This defines the version of <%= app_name %> you want to load.”
options:
<% modules.lines.reverse.each do |line| %>
<% unless line.to_s.strip == “” %>
- [ “<%= line.split(’/’)[1] %>”, “<%= line %>” ]
<% end %>
<% end %>

where the “getmodules.sh” script needs to source our Lmod (because Lmod does not get set up automatically in non-interactive shell) and then run the module spider command:
#!/bin/bash
source /uufs/chpc.utah.edu/sys/etc/profile.d/module.sh
module -t spider $1 2>&1

This generates the list of modules, e.g. like:
- [ “2016”, “abaqus/2016” ]
- [ “2019”, “abaqus/2019” ]
- [ “2021”, “abaqus/2021” ]

This works fairly well except that I have noticed a huge slowdown of the OnDemand server responsiveness when I have more than a handful of interactive apps like this. From looking at the server load, it appears to me that the module command is being called a lot every time a webpage is being refreshed, even when that page does not do anything at all with the interactive apps. This makes things so slow that the OOD server is very hard to use.

Does Ondemand regenerate the form.yml files for all the apps whenever any OOD webpage refreshes? If it does, any way to reduce this so e.g. only whenever the given interactive app page needs to be shown the form.yml.erb is processed?

Or any other thoughts in this regard?

Thanks,
Martin

Hi Martin.

Glad to see some work on something we are trying to also provide support around. The issue I think you are having is that the ERB is constantly being rendered and updated. So, each time that happens those module queries are running again and sending the info back.

To avoid this, try to cache those module result locally and use an initializer to set that location. There was a previous discourse that discussed something similar with slurm queues here Suggestions on generating dynamic form data - #3 by blixuga

But the main strategy here is to use the initializer to build that array of modules then just use that array for responses that’s cached, and not have the entire module query re-run each time the page updates.

Hopefully that helps, but let us know if you have any other questions or need some clarification.

If people are willing to consider using scheduled updates to their module list, at least for Lmod, the JSON files created by Lmod’s spider could be queried as files if an appropriate parser is used. That eliminates OnDemand’s reliance on network and responsivity of any real-time queries (except for file system). We don’t do this for producing a module list for OnDemand, but we update a software catalog hourly this way, selecting and reformatting only the fields of interest.

Just a thought.

Thanks Travis and Bennet,

I ended up setting up a cron job to generate the list of module versions into a text file and ingesting it into the form.yml.erb. Not as elegant but simple. Diving into the dynamic forms is on my todo list.

The simple script for generating the modules list is at OOD-apps-v3/genmodulefiles.sh at master · CHPC-UofU/OOD-apps-v3 · GitHub and read in the form.yml.erb is at many apps from GitHub - CHPC-UofU/OOD-apps-v3: Version 3 of OnDemand apps.

BTW, these apps also contain the templated input for most of the form parameters that I asked about at the last office hours call.

Thanks,
Martin