I have a form.yml.erb that reads files from file system and lists them under a drop down menu:
version: widget: "select" label: "Runtime" options: System provided: <% glob_prettify_lookup(["/apps/portal/jupyter/*.sh"]).each do |pretty_path, path| %> - [ "<%= pretty_path %>", "<%= path %>" ] <% end %>
but this won’t update when only refreshing the page. I’m forced to (?)->“Restart Web Server” each time to pick up new or removed entries. This glob is very cheap, I’d like to run it each time i simply refresh the page. Is this possible?
It won’t be possible with ERB. When you load the form, the dashboard builds a BatchConnect::App object, and that object reads and renders form.yml.erb exactly once - the result (including your glob) gets memoized in an instance variable (@form_config).
That App object is then cached and reused across requests, so a plain page refresh keeps hitting the same cached object with the already-rendered form. The ERB never re-runs, which is why your new/removed files don’t show up until you restart the web server and force a fresh App object.
There’s even a comment in the code that spells this out - the form config is deliberately read at object construction “so it’s there when this object is cached in upper layers.”
So a page refresh fundamentally can’t re-run the glob with the current design. If you need the runtime list to update live, the way to do it is to populate that dropdown client-side instead of in ERB - have form.js hit a small endpoint that returns the current .sh files and fills the options on page load would be an idea. That moves the lookup out of the cached server-side render and into something that runs on every refresh. There’s work-arounds, but it will basically involve JS.
Hm, when auto_queues were introduced, i know that one relies on AccountCache, which I see has an expiration time of 4 hours. AccountCache was added 3 years ago, but I see that this new render-once basically turns thus into a forever cache, after this new change was introduced late last year:
So, AccountCache timeout is now useless? Might as well put it infinite.
That’s a good catch, and you’re right that there’s a real interaction here. The render-once change (#4377) caches the App object - and its rendered form - for the life of the PUN, so any expiry-based cache that feeds into form rendering won’t actually surface its refreshed values on a plain page reload the way it would have before. For AccountCache’s 4-hour expiration, that does seem to mean the timeout no longer has a visible effect through the form, since the form itself isn’t re-rendered when the cache refreshes.
I don’t want to speak for the design intent here since the render-once change and AccountCache came from different places. @johrstrom - since you authored #4377, can you weigh in on whether the AccountCache expiration is still meant to be meaningful after the render-once change, or whether these two need to be reconciled?
AccountCache or rather auto_accounts is used by the project manager forms which aren’t cached. In any case, AccountCache could still be useful beyond these, though we don’t have any use for them yet there could be other widgets, panels or pages that can utilize it.
We started to cache batch connect applications in #4377 because while your ERB may be inexpensive many aren’t. And even worse - they were being processed to generate the navigation bar, not just when you land on the form itself.
Well, ok there might be some secondary uses like this, but the bread-and-butter if of course the interactive apps, which is what the account cache was originally made for.
because while your ERB may be inexpensive many aren’t.
Anytime sacctmgr (user memberships, accounts, partitions etc.) is touched, you’d now have to also force restart any openondemand sessions (certainly, not a single one of my users would realize they have to visit the (?) button and force a web server restart).
Even if you knew your slurm account name, the form is now a drop down list with no option of selecting it, it would be showing outdated and missing account information for a very long time.
The fact that the account cache was limited to a few hours by default makes me think there is some agreement that having it forever locked isn’t a good, but that’s now how essentially all the interactive apps behave now.
Reading through this has been interesting, and digging into the code I think Micket has a sound point. From the user’s side there’s just “the batch connect form,” and having its account list behave differently depending on internal caching layers they can’t see is confusing.
Tracing it: the account dropdown gets populated when the form ERB renders, and that render gets triggered as part of listing the app, not just opening it. SysRouter.apps constructs a BatchConnect::App for each batch connect app, and that constructor eagerly calls form_config (which renders the form ERB and memoizes it). On top of that, should_appear_in_nav? → invalid_batch_connect_app? walks sub_app_list, and build_sub_app_list constructs a fresh BatchConnect::App per sub-app — each of which renders its own form the same way. So any dynamic work in that ERB (Slurm account lookups via AccountCache, globs, etc.) runs at app-listing time, before the user ever opens the form. Then #4377 memoizes that render for the life of the PUN, which is what freezes the account list.
What stands out to me is that we now have three different cache lifetimes all touching this one behavior:AccountCache at 4 hours, the sys_apps list cache at 6 hours (Rails.cache.fetch('sys_apps', expires_in: 6.hours)), and the render-once app cache which is effectively infinite until a PUN restart. Those were added at different times for different reasons and I don’t think they were ever reconciled against each other. The 4-hour default on AccountCache reads like a deliberate statement that account data shouldn’t be considered fresh forever, and the render-once change quietly overrides that intent for the interactive apps, which were AccountCache’s original use case per #1970.
I don’t think there’s one obvious fix, and I think that’s exactly why it’s worth a design discussion rather than a quick patch. The options I can see:
The sub-app validity check shouldn’t require rendering the full form, including the dynamic parts that hit schedulers, just to decide whether the app is valid enough to list.
The render-once cache should respect a TTL so expiry-bearing data (like AccountCache) isn’t frozen indefinitely.
Sub-app enumeration should be separated from form evaluation, so listing an app doesn’t force a full render.
Each of those is a different tradeoff and probably deserves a dedicated design discussion rather than being settled here.
