I have a form.yml.erb that reads files from file system and lists them under a drop down menu:
version: widget: "select" label: "Runtime" options: System provided: <% glob_prettify_lookup(["/apps/portal/jupyter/*.sh"]).each do |pretty_path, path| %> - [ "<%= pretty_path %>", "<%= path %>" ] <% end %>
but this won’t update when only refreshing the page. I’m forced to (?)->“Restart Web Server” each time to pick up new or removed entries. This glob is very cheap, I’d like to run it each time i simply refresh the page. Is this possible?
It won’t be possible with ERB. When you load the form, the dashboard builds a BatchConnect::App object, and that object reads and renders form.yml.erb exactly once - the result (including your glob) gets memoized in an instance variable (@form_config).
That App object is then cached and reused across requests, so a plain page refresh keeps hitting the same cached object with the already-rendered form. The ERB never re-runs, which is why your new/removed files don’t show up until you restart the web server and force a fresh App object.
There’s even a comment in the code that spells this out - the form config is deliberately read at object construction “so it’s there when this object is cached in upper layers.”
So a page refresh fundamentally can’t re-run the glob with the current design. If you need the runtime list to update live, the way to do it is to populate that dropdown client-side instead of in ERB - have form.js hit a small endpoint that returns the current .sh files and fills the options on page load would be an idea. That moves the lookup out of the cached server-side render and into something that runs on every refresh. There’s work-arounds, but it will basically involve JS.
Hm, when auto_queues were introduced, i know that one relies on AccountCache, which I see has an expiration time of 4 hours. AccountCache was added 3 years ago, but I see that this new render-once basically turns thus into a forever cache, after this new change was introduced late last year:
So, AccountCache timeout is now useless? Might as well put it infinite.
That’s a good catch, and you’re right that there’s a real interaction here. The render-once change (#4377) caches the App object - and its rendered form - for the life of the PUN, so any expiry-based cache that feeds into form rendering won’t actually surface its refreshed values on a plain page reload the way it would have before. For AccountCache’s 4-hour expiration, that does seem to mean the timeout no longer has a visible effect through the form, since the form itself isn’t re-rendered when the cache refreshes.
I don’t want to speak for the design intent here since the render-once change and AccountCache came from different places. @johrstrom - since you authored #4377, can you weigh in on whether the AccountCache expiration is still meant to be meaningful after the render-once change, or whether these two need to be reconciled?
AccountCache or rather auto_accounts is used by the project manager forms which aren’t cached. In any case, AccountCache could still be useful beyond these, though we don’t have any use for them yet there could be other widgets, panels or pages that can utilize it.
We started to cache batch connect applications in #4377 because while your ERB may be inexpensive many aren’t. And even worse - they were being processed to generate the navigation bar, not just when you land on the form itself.
Well, ok there might be some secondary uses like this, but the bread-and-butter if of course the interactive apps, which is what the account cache was originally made for.
because while your ERB may be inexpensive many aren’t.
Anytime sacctmgr (user memberships, accounts, partitions etc.) is touched, you’d now have to also force restart any openondemand sessions (certainly, not a single one of my users would realize they have to visit the (?) button and force a web server restart).
Even if you knew your slurm account name, the form is now a drop down list with no option of selecting it, it would be showing outdated and missing account information for a very long time.
The fact that the account cache was limited to a few hours by default makes me think there is some agreement that having it forever locked isn’t a good, but that’s now how essentially all the interactive apps behave now.
Reading through this has been interesting, and digging into the code I think Micket has a sound point. From the user’s side there’s just “the batch connect form,” and having its account list behave differently depending on internal caching layers they can’t see is confusing.
Tracing it: the account dropdown gets populated when the form ERB renders, and that render gets triggered as part of listing the app, not just opening it. SysRouter.apps constructs a BatchConnect::App for each batch connect app, and that constructor eagerly calls form_config (which renders the form ERB and memoizes it). On top of that, should_appear_in_nav? → invalid_batch_connect_app? walks sub_app_list, and build_sub_app_list constructs a fresh BatchConnect::App per sub-app — each of which renders its own form the same way. So any dynamic work in that ERB (Slurm account lookups via AccountCache, globs, etc.) runs at app-listing time, before the user ever opens the form. Then #4377 memoizes that render for the life of the PUN, which is what freezes the account list.
What stands out to me is that we now have three different cache lifetimes all touching this one behavior:AccountCache at 4 hours, the sys_apps list cache at 6 hours (Rails.cache.fetch('sys_apps', expires_in: 6.hours)), and the render-once app cache which is effectively infinite until a PUN restart. Those were added at different times for different reasons and I don’t think they were ever reconciled against each other. The 4-hour default on AccountCache reads like a deliberate statement that account data shouldn’t be considered fresh forever, and the render-once change quietly overrides that intent for the interactive apps, which were AccountCache’s original use case per #1970.
I don’t think there’s one obvious fix, and I think that’s exactly why it’s worth a design discussion rather than a quick patch. The options I can see:
The sub-app validity check shouldn’t require rendering the full form, including the dynamic parts that hit schedulers, just to decide whether the app is valid enough to list.
The render-once cache should respect a TTL so expiry-bearing data (like AccountCache) isn’t frozen indefinitely.
Sub-app enumeration should be separated from form evaluation, so listing an app doesn’t force a full render.
Each of those is a different tradeoff and probably deserves a dedicated design discussion rather than being settled here.
I admit I’m not sure what others with those really expensive form.yml.erb evaluations are doing (they are still somehow happy with for-ever caching). Testing the code a bit (undoing PR#4377), I see that even without any sub-apps the same form_config gets re-evaluated 6 times on every page load. 7 if you actually visit the app and view the form for real. With 20 apps, that’s 121 evaluations, instead of what should have been “only” 21, when accounting for the navigation bar (once) + the speific app view itself.
Maybe there are some ways of treating clusters and empty form_config separately but currently the validity checks verifies the data type of form (Array) and attributes (Hash). You just have to parse the yml to check that. Unless one is willing to forgo some checks here (after all, no checks are done for submit, view, form.js erb files, and there is plenty of other errors you can introduce into the yml file that are not caught). One of the validity checks has actually bothered me before; the assumption that the form can’t be empty (I had a perfectly valid app that did not need any inputs, just the submit button, it was for launching an VNC on a login node).
A few thoughts
Whatever triggers those extra 5 form_config evaluations should be investigated.
def form_config internal memoization could easily be replaced with yet another Rails.cache with no downside… except complexity and yet one more cache layer to keep track of logically. I prefer option 3:
For the actual form view page, one could afford to invalidate the form_config cache before rendering. Very small cost, always get up-to-date information. Just great. E.g. have form_view just call form_view_render , which always updated @form_view. This conveniently allows for manually triggering form_view_render before actually displaying it. If your form.yml.erb file is to expensive to even do just that I think the onus should be you to move that stuff into an initializer instead (like with Slurm account data). See below
On that note, perhaps batch connect apps should have a handy little per-app “initializer.rb” support. Instead of jamming a bunch of complex code into the top of the form.yml.erb. It would be a little bit closer to how dashboard apps work, and gives an obvious place to put extra route endpoints, pre-compute stuff. I realized with some testing you can literally modify any global state from within any .erb file, but that seems like coding sin. I know i could put that stuff into initializers.d, but having BC apps self-contained is convenient.
Perhaps some caching should rather be happening inside the navigation bar.
Just any of these many of these would probably restore both the slurm data, as well making it possible for dynamic form contents via erb files, at minimal costs. Suggestion 3 I imagine would just be:
def form_config(binding: nil)
return @form_config if @form_config
return render_form_config(binding)
end
def render_form_config(binding: nil)
# All the stuff form_config used to do compute hsh.
@form_config = hsh
end
In my opinions, some of these caches are very long so it would be nice to have some convenient way of customizing them via config files. sacctmgr likely takes < 1 second for most clusters, so 4 hours of caching really just pushes well beyond what’s necessary for a smooth experience.
