Cache of auto_queues?

Trigil · April 25, 2024, 1:26pm

Hello!

I am doing some testing of the auto_queues feature and there is some behavior I don’t fully understand.

We have users that can have access to cpu (and cpu partitions), gpu (gpu partitions) or both, based on slurm.

Let’s say I am a user with only cpu access. If I use sinfo, I see only cpu partitions. And on the ondemand remote desktop, I also see only cpu partitions, as intended. Now, if I change my access to only gpu, with sinfo I see only gpu partitions, but if I log back on ondemand remote desktop I still see cpu partitions. Then if I reboot the ondemand node, I can now see only gpu partitions as intended.

So, I am wondering how exactly the partitions are cached between each connections, and how to force it to update? I first though of the file in ~/ondemand/data/sys/dashboard/batch_connect/cache/sys_bc_desktop.json but since I never fully reserved a node with that user, this file doesn’t exist. Plus, when it exists, it only contain one partition in the field auto_queues, not the full list of it.

Thanks for your help and have a nice day!

Tristan

travert · April 25, 2024, 3:29pm

Hi and thanks for the question! The sinfo command that gathers this up can be seen here in ood_core:

github.com

OSC/ood_core/blob/4b21ae7bfb9335e248b547296cbac21137fc818f/lib/ood_core/job/adapters/slurm.rb#L112


      
            @conf                 = conf    && Pathname.new(conf.to_s)
            @bin                  = Pathname.new(bin.to_s)
            @bin_overrides        = bin_overrides
            @submit_host          = submit_host.to_s
            @strict_host_checking = strict_host_checking
          end
          
          # Get a ClusterInfo object containing information about the given cluster
          # @return [ClusterInfo] object containing cluster details
          def get_cluster_info
            node_cpu_info = call("sinfo", "-aho %A/%D/%C").strip.split('/')
            gres_length = call("sinfo", "-o %G").lines.map(&:strip).map(&:length).max + 2
            gres_lines = call("sinfo", "-ahNO ,nodehost,gres:#{gres_length},gresused:#{gres_length}")
                         .lines.uniq.map(&:split)
            ClusterInfo.new(active_nodes: node_cpu_info[0].to_i,
                            total_nodes: node_cpu_info[2].to_i,
                            active_processors: node_cpu_info[3].to_i,
                            total_processors: node_cpu_info[6].to_i,
                            active_gpus: gres_lines.sum { |line| Slurm.gpus_from_gres(line[2]) },
                            total_gpus: gres_lines.sum { |line| Slurm.gpus_from_gres(line[1]) }
            )

So, there is a call to sinfo -aho %A/%D/%C which is then processed and stored in an array for later access for CPU info. Then there is another call with information about the gpu stored in gres_lines. So, this is the initial place that information is gathered.

On that remote desktop, this command has not rerun and so you just have that original information, whereas when you restarted, it was updated as the PUN initialized. There’s not any kind of cache server or repeated calls to update. It looks to work more by calling on the initial launch and only then, which is then called by OOD dashboard and we use a Rails cache to store this:

github.com

OSC/ondemand/blob/1a35d7eb54b975d7577c6d566ecfca678103bcbe/apps/dashboard/app/lib/smart_attributes/attributes/auto_queues.rb#L13


      
          module SmartAttributes
            class AttributeFactory
              extend AccountCache
          
              # Build this attribute object. No 'options' are used as this Attribute
              # is meant to be dynamically generated from the users' available accounts
              # from a given scheduler.
              #
              # @param opts [Hash] attribute's options
              # @return [Attributes::AutoQueues] the attribute object
              def self.build_auto_queues(opts = {})
                static_opts = {
                  options: queues
                }.merge(opts.without(:options).to_h)
          
                Attributes::AutoQueues.new('auto_queues', static_opts)
              end
            end
          
            module Attributes
              class AutoQueues < Attribute

And the Rails cache is then used to store this:

github.com

OSC/ondemand/blob/1a35d7eb54b975d7577c6d566ecfca678103bcbe/apps/dashboard/app/lib/account_cache.rb#L60


      
          
                [account_name, account_name, data]
              end
            end
          end
          
          # To be used with dynamic forms. This method stithes together data
          # about the queue's availablity WRT clusters.
          #
          # @return [Array] - the dynamic form options
          def queues
            Rails.cache.fetch('queues', expires_in: 4.hours) do
          
              queues_per_cluster.map do |cluster, cluster_queues|
                other_clusters = queues_per_cluster.reject do |c, _queues|
                  c == cluster
                end.map do |c, _queues|
                  c.to_s
                end
          
                cluster_data = other_clusters.map do |other_cluster|

And some work with the Configuration as well later in that file:

github.com

OSC/ondemand/blob/1a35d7eb54b975d7577c6d566ecfca678103bcbe/apps/dashboard/app/lib/account_cache.rb#L142


      
          def blocked_queue?(queue)
            allow_accounts = queue.allow_accounts
          
            if allow_accounts.nil?
              false
            else
              allow_accounts.intersection(account_names).empty?
            end
          end
          
          def queues_per_cluster
            Rails.cache.fetch('queues_per_cluster', expires_in: 24.hours) do
              {}.tap do |hash|
                Configuration.job_clusters.each do |cluster|
                  hash[cluster.id] = cluster.job_adapter.queues
                end
              end
            end
          end
          
          def queue_account_data(queue)

The short of it is that is all updated on initialization of your PUN.

Trigil · April 26, 2024, 6:52am

Hello Travis,

Whoa, thank you for your very detailed reply and the links to code! So it works as I expected it to work, and the weird part during my tests was more likely due to the propagation time of the change in user’s access. I must have reconnected to ondemand the first time to soon after the command to change the access. I tried again and it takes sinfo a couple of minutes to show the correct information.

Thanks again, have a nice day!

Tristan

Topic		Replies	Views
Auto_accounts/qos/queues not working after upgrade to 3.0.2 Get Help	5	185	April 24, 2024
SLURM Queue for OnDemand Get Help	2	1001	May 26, 2022
Off topic: python threading in Slurm (and number of CPUs per job) General Discussion	2	421	August 5, 2022
Install on single server Get Help	5	1354	May 26, 2022
Multiple instances issue with interactive apps Get Help	6	155	September 26, 2023

Cache of auto_queues?

Related Topics