> ### Check whether the server is low on resources > > Maybe the server is currently so low on resources that all the work that needed to be done, could not finish within the given time limit. Please inspect the server resource utilization statistics in the _diagnostics_ section to verify whether server is indeed low on resources. > > If so, then either increase the spawn timeout (currently configured at 90 sec), or find a way to lower the server’s resource utilization. > > ### Still no luck? > > Please try troubleshooting the problem by studying the _diagnostics_ reports. > > ”, > “summary” : “A timeout occurred while spawning an application process.” > }, > “journey” : { > “steps” : { > “SPAWNING\_KIT\_FINISH” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SPAWNING\_KIT\_FORK\_SUBPROCESS” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.36493100000001, > “timestamp” : 1580749283.0021019 > }, > “duration” : 0.001, > “end\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.36393100000001, > “timestamp” : 1580749283.0031021 > }, > “state” : “STEP\_PERFORMED” > }, > “SPAWNING\_KIT\_HANDSHAKE\_PERFORM” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.36393100000001, > “timestamp” : 1580749283.0031021 > }, > “duration” : 90.049999, > “end\_time” : { > “local” : “Mon Feb 3 11:02:53 2020”, > “relative” : “1m 30s ago”, > “relative\_timestamp” : -90.313931999999994, > “timestamp” : 1580749373.0531011 > }, > “state” : “STEP\_ERRORED” > }, > “SPAWNING\_KIT\_PREPARATION” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.36593099999999, > “timestamp” : 1580749283.001102 > }, > “duration” : 0.001, > “end\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.36493100000001, > “timestamp” : 1580749283.0021019 > }, > “state” : “STEP\_PERFORMED” > }, > “SUBPROCESS\_APP\_LOAD\_OR\_EXEC” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SUBPROCESS\_BEFORE\_FIRST\_EXEC” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.35193100000001, > “timestamp” : 1580749283.0151019 > }, > “duration” : 0.0, > “end\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.35193100000001, > “timestamp” : 1580749283.0151019 > }, > “state” : “STEP\_PERFORMED” > }, > “SUBPROCESS\_EXEC\_WRAPPER” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SUBPROCESS\_FINISH” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SUBPROCESS\_LISTEN” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SUBPROCESS\_OS\_SHELL” : { > “state” : “STEP\_NOT\_STARTED” > }, > “SUBPROCESS\_SPAWN\_ENV\_SETUPPER\_AFTER\_SHELL” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:02:23 2020”, > “relative” : “2m 0s ago”, > “relative\_timestamp” : -119.995931, > “timestamp” : 1580749343.3711021 > }, > “state” : “STEP\_IN\_PROGRESS” > }, > “SUBPROCESS\_SPAWN\_ENV\_SETUPPER\_BEFORE\_SHELL” : { > “begin\_time” : { > “local” : “Mon Feb 3 11:01:23 2020”, > “relative” : “3m 0s ago”, > “relative\_timestamp” : -180.35193100000001, > “timestamp” : 1580749283.0151019 > }, > “duration” : 60.343000000000004, > “end\_time” : { > “local” : “Mon Feb 3 11:02:23 2020”, > “relative” : “2m 0s ago”, > “relative\_timestamp” : -120.008931, > “timestamp” : 1580749343.3581021 > }, > “state” : “STEP\_PERFORMED” > }, > “SUBPROCESS\_WRAPPER\_PREPARATION” : { > “state” : “STEP\_NOT\_STARTED” > } > }, > “type” : “SPAWN\_DIRECTLY” > }, > “program\_name” : “Phusion Passenger”, > “short\_program\_name” : “Passenger” > } > ; ---

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\ **Post date:** [February 4, 2020, 10:14pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/2 "2020-02-04T22:14:42Z")

Doesn’t look like anything to me. Indeed you’re at 100% idle and ~0.25 load average says it’s not a resource issue. I saw you say it’s local storage, but do you have _any_ NFS mounts enabled? At this point (where you’re failing at) you’ve already authenticated, so how different AD accounts behave differently is strange. Maybe it’s a timing thing? Like do you ever use the test account (the one that works flawlessly) first? Or… do you use you’re own a few times, _then_ use the test account? Maybe it’s not the different in accounts, but in time - meaning something warmed up and got cached and that’s why the test user is less flaky. More information is required I’m afraid. I think a `pstack` may while it idles may help a lot. Also, there could be something in dmesg or /var/log/messages or journalctl. Some system level log may have something? Try to start a session, login to the machine and get a pstack of that process. Here are the processes on my machine for reference. You may not see all of them because you’re still in the boot process, but I think we’re looking for 311 (Passenger watchdog) in this example, but if you have 314 (Passenger Core) get it too. Hopefully that will shed a some light on what it’s waiting on. ```bash 4 S jeff 311 1 0 80 0 - 91556 - 17:13 ? 00:00:00 Passenger watchdog 0 S jeff 314 311 0 80 0 - 389004 - 17:13 ? 00:00:01 Passenger core 5 S root 333 1 0 80 0 - 25556 - 17:13 ? 00:00:00 nginx: master process (jeff) -c /var/lib/ondemand-nginx/config/puns/jeff.conf 5 S jeff 334 333 0 80 0 - 25557 - 17:13 ? 00:00:00 nginx: worker process ``` ---

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\ **Post date:** [February 5, 2020, 2:30pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/3 "2020-02-05T14:30:53Z")

Actually, thinking about this a second time maybe it does have something to do with LDAP? You could try switching your LDAP auth to use the users’ credentials instead of the test ones. It may or may not affect anything or have any difference, but it’s at least an avenue to test and rule AD caching out. If there is something there you could turn mod\_authnz\_ldap debug logs on to see timings better. use ```bash AuthLDAPSearchAsUser on ``` instead of ```bash AuthLDAPBindDN someTestUser AuthLDAPBindPassword someTestPassword ``` ---

**Author:** ![rgas20](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rgas20/32/309_2.png) [@rgas20](https://discourse.openondemand.org/u/rgas20)\ **Post date:** [March 4, 2020, 8:48pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/4 "2020-03-04T20:48:46Z")

Hi Jeff, My apologies for the huge delay with getting back to you, I have not looked at this problem in a while and am just getting back into it. I really appreciate your suggestions. I was playing around a bit with this today. I started with a fresh reboot of the server and starting up the Apache service. Then, I signed in with a known “good” user - I just created a new user under our Research Computing OU today. I am logged in immediately. When I flip over to htop and take a look at the processes, I see multiple Passenger Core and Watchdog processes for “good.user” appear immediately. Then, as a test, I rebooted the server again to gain a fresh slate, and signed in with a user I know I have trouble with (My AD account, for one). I signed in then watched htop. It took upwards of 30 seconds-1 minute for the Passenger Core and Watchdog processes to appear under my username. Once they do appear in htop, I’m still waiting for OnDemand to load in my browser. I can wait for it in another window for a few minutes and it will eventually return with the above error. When I tested it with the bad account, the only related processes I see are: ``` root id -nG bad.user root ruby -I/opt/ood/nginx_stage/lib -rnginx_stage -e NginxStage::Application.start -- pun -u bad.user -a https%3a%2f%2fondemand.domain.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri root sudo /opt/ood/nginx_stage/sbin/nginx_stage pun -u bad.user -a https%3a%2f%2fondemand.domain.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri root sh -c sudo /opt/ood/nginx_stage/sbin/nginx_stage pun -u 'bad.user' -a 'https%3a%2f%2fondemand.domain.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri' 2>&1 ``` I tried looking in the logs and I don’t see anything out of the ordinary but I will go back and comb through them again. I ran a tcpdump and I don’t know if it’s much relevance but on the “bad.user” accounts I see a lot of jumping between our ldap servers whereas with the “good” account it contacts our ldap server, returns immediately and works. However, like you said the issues seem to be happening after I’ve authenticated already, so it’s odd. I thought it might have been an LDAP referral issue but now I’m not so sure. When I have colleagues try it, the same thing happens but when I try it with fresh test accounts it seems to always work just fine. I didn’t return anything with pstack when I tried so that’s why I flipped over to htop. When I do an ldapwhoami or ldapsearch I am able to return the info on the “bad” accounts immediately so I don’t think it’s an issue with my LDAP config. I had trouble with the SearchAsUser option but I tried the ldap search binding with my own credentials and was successsful. There are no NFS mounts on the server. I did have /home mounted via NFS at one point, but since I’ve run into this issue I’ve put that on the back burner for now and I’ve been using /home directly on OnDemand. I’m looking at getting this tied in with another auth method like CAS and hopefully that resolves the problem. And we have been going in under the “good” account to do app development. Everything else is set up and looks great. Once we get past this issue I think we’ll be in business and can move it to production. ---

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\ **Post date:** [March 13, 2020, 4:40pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/5 "2020-03-13T16:40:44Z")

Hi, I’m sorry for my delay as well. The fact that `id -nG bad.user` is in your notes gives me pause. That hits ldap too. I’d be interested in seeing how long it takes to complete when you run that command from a shell (try as a regular user a couple times first then as root if it’s not slow) . That command should come back in almost no time, if it doesn’t then that’s our ticket. If it does take a long time, you can strace it like this and you’ll see where all your time is being taken. `strace -ttt id -nG johrstrom > strace.log 2>&1` ---

**Author:** ![rgas20](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rgas20/32/309_2.png) [@rgas20](https://discourse.openondemand.org/u/rgas20)\ **Post date:** [March 18, 2020, 1:36pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/6 "2020-03-18T13:36:06Z")

@jeff.ohrstrom, thanks a lot for your reply, we have been looking around at this more and have definitely found that we are running into problems with these accounts that have several group memberships. As you mentioned, ‘id’ returns immediately for the newer accounts and is pretty slow for the established accounts we’ve been having issues with. It looks like we have some more tuning to do and we’ll be on our way to getting this resolved. ---

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\ **Post date:** [May 26, 2022, 3:36pm UTC](https://discourse.openondemand.org/t/latency-issues-and-timeout-in-web-application/703/7 "2022-05-26T15:36:21Z")