Form auto_queues missing a partition

hansen-m · January 13, 2025, 4:47pm

When using the auto_queues field on our cluster one of our partitions is missing from the dropdown list and we aren’t exactly sure why since all the others appear correctly? We suspect it might have something to do with the ReqResv or AllowAccounts. Is there a list of Slurm commands we could step through to replicate how that list generated?

OnDemand version: 3.1.10

PartitionName=sla-prio Nodes=basic_nodes,standard_nodes,himem_nodes TRESBillingWeights=CPU=0,Mem=0,GRES/gpu:a100=0,GRES/gpu:p100=0,GRES/gpu:v100=0,GRES/gpu:a100_3g=0,GRES/gpu:a100_1g=0,GRES/gpu=0 CpuBind=core DefaultTime=24:00:00 PriorityTier=100 ReqResv=YES AllowAccounts=reserved_allocation

hansen-m · January 14, 2025, 3:36pm

We were able to track this back a little further.

Seems to be related to AllowAccounts settings. Partitions without AllowAccounts set or “AllowAccounts=open” the partition appears correctly in the auto_queues drop-down menu but “AllowAccounts=reserved_allocation” it does not appear, and that is regardless of if I’m included in “reserved_allocation” or not.

jeff.ohrstrom · January 14, 2025, 3:51pm

One thing you can do is inspect the HTML and see what options are hidden. Here I’m looking at auto_accounts, but the strategy is similar. My guess is is that sla-prio is there, just hidden and is only available for the reserved_allocation auto_account value.

hansen-m · January 14, 2025, 4:23pm

@jeff.ohrstrom Does it support an account hierarchy? I’m in an account that has reserved_allocation as a parent. If we flatten that and use the account directly then it appears in the list.

hansen-m · January 23, 2025, 2:19pm

@jeff.ohrstrom Any confirmation on if it supports nested accounts? If not, how it determines the list for a user and where we would look in the code to update that? It seems that the new Project Manager app has similar auto fields and has the same issue.

jeff.ohrstrom · January 23, 2025, 2:33pm

I don’t know what a nested account is, so I doubt it.

We pull accounts through this sacctmgr command

github.com/OSC/ood_core

lib/ood_core/job/adapters/slurm.rb

ea172947f


      
              end
              jobs
            end
          rescue SlurmTimeoutError
            # TODO: could use a log entry here
            return [{ id: id, state: 'undetermined' }]
          end
          
          def accounts
            user = Etc.getlogin
            args = ['-nP', 'show', 'users', 'withassoc', 'format=account,cluster,partition,qos', 'where', "user=#{user}"]
          
            [].tap do |accts|
              call('sacctmgr', *args).each_line do |line|
                acct, cluster, queue, qos = line.split('|')
                next if acct.nil? || acct.chomp.empty?
          
                args = {
                  name: acct,
                  qos: qos.to_s.chomp.split(','),
                  cluster: cluster,

We pull partitions through this scontrol command.

github.com/OSC/ood_core

lib/ood_core/job/adapters/slurm.rb

ea172947f


      
              core_specialization: "%X",
              nice: "%y",
              scheduled_nodes: "%Y",
              sockets_cores_threads: "%z",
              work_dir: "%Z",
              gres: "%b",  # must come at the end to fix a bug with Slurm 18
            }
          end
          
          def queues
            info_raw = call('scontrol', 'show', 'part', '-o')
          
            [].tap do |ret_arr|
              info_raw.each_line do |line|
                ret_arr << str_to_queue_info(line)
              end
            end
          end
          
          def all_sinfo_node_fields
            {

jeff.ohrstrom · January 24, 2025, 2:30pm

I am learning about nested accounts. Apparently we use them at OSC. I’ve created this ticket upstream, but I’m not sure when it’ll come and indeed would add an extra CLI call that could be expensive so I’m not sure when it’ll get fixed.

github.com/OSC/ood_core

Slurm - account for nested accounts

opened 02:29PM - 24 Jan 25 UTC

johrstrom

The `accounts` API in slurm should account for nested accounts as they relate to… partitions and their accessibly. From discourse https://discourse.openondemand.org/t/form-auto-queues-missing-a-partition/3958/6 Here's a partition that has a `AllowAccounts` value that's actually a parent account. Any AccountInfo that is a child of this account should be allowed in this queue. ``` PartitionName=sla-prio Nodes=basic_nodes,standard_nodes,himem_nodes TRESBillingWeights=CPU=0,Mem=0,GRES/gpu:a100=0,GRES/gpu:p100=0,GRES/gpu:v100=0,GRES/gpu:a100_3g=0,GRES/gpu:a100_1g=0,GRES/gpu=0 CpuBind=core DefaultTime=24:00:00 PriorityTier=100 ReqResv=YES AllowAccounts=reserved_allocation ``` I'm fairly sure we have to make another CLI call to accomplish this.

Topic		Replies	Views
Auto_accounts/qos/queues not working after upgrade to 3.0.2 Get Help	5	268	April 24, 2024
Cache of auto_queues? Get Help	3	126	October 23, 2024
Automatic attributes and dynamic form widgets Get Help question	4	407	February 12, 2024
Access to "auto_accounts" data within form Get Help question	2	45	November 12, 2024
Evaluating auto_accounts to populate the field text Get Help question	3	131	July 14, 2024

Form auto_queues missing a partition

Related topics