Frequency of OnDemand polling Slurm

milberg · August 11, 2021, 8:57pm

Is there a way to determine the frequency at which OnDemand polls Slurm nodes (login or ssh hosts) for job state? And is there a way to control this frequency?

travert · August 12, 2021, 3:57pm

The frequency should be set in the slurm.conf file itself using the JobAcctGatherFrequency. That setting has more documentation here:
https://slurm.schedmd.com/slurm.conf.html

Looking at the entry for JobAcctGatherFrequency it looks like for tasks the interval is 30 seconds and for any other datatype the default is 0 seconds, meaning it will wait until termination of the job.

OnDemand will just use the settings in that file to configure itself for polling which is why the path for the slurm.conf in the /etc/ood/config/clusters.d/my_cluster.yml is included.

milberg · August 12, 2021, 4:20pm

That’s exactly what I was looking for, thank you!

travert · August 12, 2021, 5:06pm

Hey @milberg I was incorrect in what I posted earlier. You can actually set this polling for OnDemand.

There is some javascript that will actually poll which is setup here:

github.com

OSC/ondemand/blob/master/apps/dashboard/app/assets/javascripts/batch_connect/sessions.coffee

# Place all the behaviors and hooks related to the matching controller here.
# All this logic will automatically be available in application.js.
# You can use CoffeeScript in this file: http://coffeescript.org/

#
# Timer object
#

class Timer
  constructor: (@callback, @delay) ->
    @remaining = @delay
    @active = true
    @resume()
  resume: () ->
    return unless @active
    @start = new Date()
    clearTimeout(@timerId)
    @timerId = setTimeout(@callback, @remaining)
  restart: () ->
    return unless @active

This file has been truncated. show original

And then the controller will generate new html for the cards under the ‘interactive sessions’ and send it back to the javascript:

github.com

OSC/ondemand/blob/df8e3ce019202e7a21408cda2da963f339bda216/apps/dashboard/app/views/batch_connect/sessions/index.html.erb#L34

    
      
                  partial: "batch_connect/shared/app_menu",
                  locals: {
                    sys_app_groups: @sys_app_groups,
                    usr_app_groups: @usr_app_groups,
                    dev_app_groups: @dev_app_groups
                  }
                )
              %>
            </div>
            <div class="col-md-9">
              <div class="batch-connect sessions" data-toggle="poll" data-url="<%= batch_connect_sessions_path(format: :js) %>" data-delay="<%= ENV["POLL_DELAY"] || 10000 %>">
                <% if @sessions.empty? %>
                  <div class="ood-appkit markdown">
                    <p><%= t('dashboard.batch_connect_no_sessions') %></p>
                  </div>
                <% else %>
                  <%= render partial: "panel", collection: @sessions, as: :session %>
                <% end %>
              </div>
            </div>
          <%- else -%>

The default on the last piece shows we have a default of 10 seconds actually.

You can set this in your env file like so:

#/etc/ood/config/apps/dashboard/env
POLL_DELAY=10000

And this should change those polling intervals for the javascript to update the card in the ‘my interactive session’ view. Sorry about the confusion.

milberg · August 12, 2021, 5:26pm

That’s very helpful, thanks!

Topic		Replies	Views
Slurm Job Status not showing QOSMaxWallDurationPerJobLimit General Discussion question	0	309	May 23, 2023
SLURM Queue for OnDemand Get Help	2	988	May 26, 2022
Interactive jobs "disappearing" Get Help	7	559	May 19, 2022
Off topic: python threading in Slurm (and number of CPUs per job) General Discussion	2	402	August 5, 2022
Submit to scheduler on seperate node from OndDemand node Get Help	5	631	May 26, 2022

Frequency of OnDemand polling Slurm

Related Topics