# Help troubleshooting poor dashboard load performance

**URL:** <https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908>\
**Category:** Get Help\
**Created:** [August 17, 2023, 3:06pm UTC](https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908 "2023-08-17T15:06:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://discourse.openondemand.org/u/ndusek)\
**Post date:** [August 17, 2023, 3:06pm UTC](https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908/1 "2023-08-17T15:06:11Z")

</div>

Hello,

We are seeing some performance issues with loading the dashboard. I know this issue has been discussed before, so I’ll do my best to cover all the bases.

**Environment:**  
OOD 3.0.1  
RHEL 8.5  
CAS for auth with AD backend via SSSD

**Issue:**  
Loading the dashboard with a PUN already running takes \<1s (~300-800ms) whereas if a PUN is not already running, or when restarting the web server, the dashboard takes 4-16s to load – about an order of magnitude slower. I don’t have a large sample size of test cases, but anecdotally, accounts with sandbox development enabled seem to be on the longer end of both ranges, i.e. closer to 1s with a PUN running and 8-16s when restarting the web server.

**What I’ve tried so far:**  
I’ve read most of the existing threads on this issue. In no particular order:

- There are no errors in the console, and no issue serving assets, so [this thread](https://discourse.openondemand.org/t/dashboard-slow-to-load-after-cas-ondemand-selinux/2178/3) doesn’t seem to address the problem.
- [This isn’t the issue](https://discourse.openondemand.org/t/long-load-time-on-log-in-and-restart-of-web-server/1636), because we have SELinux disabled.
- I’ve applied the recommended [performance-boosting SSSD configs mentioned in this thread](https://discourse.openondemand.org/t/slow-dashboard-load-time/1616). Time to authenticate and retrieve ID information from AD does not seem to be the issue.
- [Loading time for individual assets](https://discourse.openondemand.org/t/dashboard-page-load-performance-troubleshooting/1780/3) doesn’t seem to be the issue, since in all cases, loading the dashboard itself takes almost all of the total request time. After the dashboard loads, all other assets load almost instantly.

I’ve also attempted to debug the process by manually stopping and restarting the PUN from the terminal on the OOD server:

```auto
/opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -u ndusek
/opt/ood/nginx_stage/sbin/nginx_stage pun -u ndusek

```

In those cases, load times are \<1s, consistent with a reload of the dashboard when the PUN is already running.

So at this point, it seems to be that the dashboard is doing something that takes longer if a PUN is not already running, and that the time seems to increase if you have more apps to serve. In other words, I’m wondering, with a new PUN, does the dashboard have to enumerate all the apps a user has access to, but after that initial load, assets/apps are cached and served more quickly?

I hope what I’m saying makes sense or rings a bell with the OOD developers. Let me know if I can provide any more information.

Thanks,  
Nick

---

<div class="post-metadata">

**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://discourse.openondemand.org/u/ndusek)\
**Post date:** [August 17, 2023, 3:37pm UTC](https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908/2 "2023-08-17T15:37:21Z")

</div>

Another data point:

We found that clearing out `$OOD_DATAROOT/sys` and `$OOD_DATAROOT/dev` reduced the load time for accounts with the slowest responses from 8-16s down to ~4s. So that accounts for the difference between “slow” and “really slow”.

This implies that more active users running interactive apps more often will have slower load times. What are some ways to address this? I seem to remember a setting that sets the card lifetime for past interactive jobs, but I don’t know if this actually clears out the output directories for the sessions, does it?

Also, that still leaves unaccounted the +3s delay when restarting the web server or reloading the dashboard after several minutes of inactivity. Any idea where that could be coming from?

Nick

---

<div class="post-metadata">

**Author:** ![travert](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/travert/32/3074_2.png) [@travert](https://discourse.openondemand.org/u/travert)\
**Post date:** [August 17, 2023, 4:03pm UTC](https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908/3 "2023-08-17T16:03:21Z")

</div>

Hello and thanks for the info!

We aren’t sure off the top of our heads what this might be and will have to check into this more.

I’ve opened an issue for this for us to track and dig into what is happening though. Sorry not to have a better answer right away. The issue can be seen here. Please feel free to add any more information and data you see fit.

> <https://github.com/OSC/ondemand/issues/2975>
>
> We had a user create a Discourse that gathered some information for us around sl…ow load times for PUNs being restarted with many apps to reload. 
> 
> The Discourse can be seen here:
> https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908
> 
> This is possibly an issue in the dashboard code in the routers where there may be some regexes that could be sped up, or the 'nav\_bar.rb\` in the '::items' or some of the longer class methods there which are doing quite a bit of work and may need some attention.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [February 13, 2024, 4:04pm UTC](https://discourse.openondemand.org/t/help-troubleshooting-poor-dashboard-load-performance/2908/4 "2024-02-13T16:04:00Z")

</div>

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.