Suggestions on generating dynamic form data

Get Help
1

An example of this would be to generate a list of partitions(queues) that the user accessing the form has access to, and populating the drop-down options list. I have all the commands/code to generate the data, but this is causing major issues due to https://github.com/OSC/ondemand/issues/210

So this post is looking for suggestions on how to generate data for interactive applications form.yml without causing extra load on the system.

Reference topic:
https://discourse.openondemand.org/t/ood-parses-and-executes-all-form-yml-erb-for-all-interactive-applications-no-matter-what-action/1668/6

Hopeful for help,

Bruce

2

Maybe you can try an initializer like how we build short cuts to the files menu for project & storage space. This way the queues are build once during the app startup. Of course, you still need a form.yml.erb to access them, but they’ll already be computed.

https://osc.github.io/ood-documentation/latest/customization.html?highlight=initializer#add-shortcuts-to-files-menu

class CustomQueues
  def self.queues
    @queues ||= begin
        # here's the logic to return an array of strings
    end
  end
end

# call it once during the initiazlier so that it'll be cached for later.
CustomQueues.queues
3

Jeff,

Thanks for this suggestion, yes this will cut the computation down tremendously!

Bruce

4

Follow-up for anyone interested, as I was able to eliminate virtually all the processing delays I had due to the individual apps issue with this work-around:

In our ood dashboard initializer /etc/ood/config/apps/dashboard/initializers/ood.rb I have added code for some static classes as suggested by Jeff. In case anyone is curious on our implementation; here is the first one gets all the users visible Slurm partitions:

require 'open3' # Required for capture3 command line call

class CustomQueues ### GET PARTITIONS FOR THIS USER ###
  def self.queues
    @queues ||= begin
        # here's the logic to return an array of strings
        sinfo_cmd = "/opt/apps/slurm/prod/bin/sinfo"
        args = ["--noheader","-o=\"%R\""]
        @partitions_available = []
        o, e, s = Open3.capture3(sinfo_cmd , *args) 
        o.each_line do |v|
          #v.gsub(/=\"(\S+)\"|=\"(\S+_p)\"/, '\1')
          @partitions_available.append(v.gsub(/=\"(\S+)\"|=\"(\S+_p)\"/, '\1'))
        end
      @partitions_available
    end
  end
end

# call these once during the initiazlier so that it'll be cached for later.
CustomQueues.queues

This data is then consumed in our apps form.yml.erb as like so:

queue:
    label: "Partition"
    widget: select
    options:
	  <%- CustomQueues.queues.each do |g| %>
      - "<%= g %>"
    <%- end %>
    cacheable: false

I’ve done another to populate application versions from LMOD Spider json data. If anyone is interested in it I can post it as well.

Thanks again to @jeff.ohrstrom for the help!

5

This is absolute gold, thank you very much.

Just a note, you may need to strip white space from g. I had to:

 <%- CustomQueues.queues.each do |g| %>
      - "<%= g.strip %>"
    <%- end %>

Otherwise my (very old) slurm complained. My guess is that this is because in ooooolllldddeeee slurms, it did not strip white space.

1 Like
6

Really helpful thread! I managed to do the same for the slurm accounts;

class SlurmData
  def self.accounts
    username = Etc.getlogin
    accounts = []
    IO.popen 'sacctmgr -np show accounts withassoc format=account,user' do |io|
      io.each do |line|
        acc = line.split('|')
        if acc[1] == username
          accounts.append(acc[0])
        end
      end
    end
    return accounts
  end
end

combined with

  bc_account:
    widget: "select"
    label: "Project"
    options: [<%= SlurmData.accounts.join(', ') %>]

(Note: I had to modify bc_account to allow it to specify a custom widget type, as it was hardcoded to text_field)

Edit: The use of the bracket list yml syntax here was to avoid errors if for some reason the list happened to be empty. An empty

   options:

causes errors.

2 Likes
7

For Robert Settlage from OOD Open Office Hours:

require 'json'
require 'open3' # Required for capture3 command line call

class CustomQueues ### GET PARTITIONS FOR THIS USER ###
  def self.queues
    @queues ||= begin
        #puts("Grabbing partitions for user.")
        sinfo_cmd = "/opt/apps/slurm/prod/bin/sinfo"
        args = ["--noheader","-o=\"%R\""]
        @partitions_available = []
        o, e, s = Open3.capture3(sinfo_cmd , *args) 
        o.each_line do |v|
          #v.gsub(/=\"(\S+)\"|=\"(\S+_p)\"/, '\1')
          @partitions_available.append(v.gsub(/=\"(\S+)\"|=\"(\S+_p)\"/, '\1'))
        end
      @partitions_available
    end
  end
end


class LmodSpiderData ### GET LMOD Spider info for interactive apps ###
    def self.spider_data
      puts("Grabbing Spider Data.")
        @spider_data ||= begin
            spider_data_file = File.open "/opt/ood_data/daily-lmod.json"
            @sd = JSON.load spider_data_file
            spider_data_file.close
            @sd
        end
    end
    def self.returnAppVersions(app_name)
      puts("Grabbed Versions for: " + app_name)
      begin
        _app_hash = spider_data[app_name]
        _app_hash.map {|k,v|  v["fullName"]}
      end
    end
end

# call these once during the initiazlier so that it'll be cached for later.
CustomQueues.queues
LmodSpiderData.spider_data

And here is the script that I run as a cronjob to generate the json data:

#!/bin/bash

# Runs LMOD Spider and generates the json file OOD needs to display modules in interactive apps.
/apps/lmod/lmod/libexec/spider -o spider-json /apps/mf/gb:/apps/mf/eb/all:/apps/easybuild/modules/all | python -mjson.tool > /opt/ood_data/daily-lmod.json
8

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.