Latency Issues with OOD and Jupyter

[apelkey (Alexander Pelkey)]

Hey everyone!

I am new to Open OnDemand, and have been working on deploying/configuring OOD for the institution I work at.

I have configured and deployed OOD to our test network, as well as Jupyter Lab within OOD. The issue at hand is that intermittently (but usually most of the time) the “Connect to Jupyter” function within the Jupyter app will launch a new window and then hang indefinitely (stays at a blank white screen). I have been trying to track down the root cause of this, but have not been able to identify anything in the logs that could be the smoking gun. For this issue, I have been checking the logs “error.log” and “access.log” under /var/log/ondemand-nginx/$user/ (happy to share the output if people think it would be pertinent). Furthermore, navigating around the portal (such as selecting the “Files” or “Clusters” tabs) also brings with it long load times. Has anyone experienced this before when deploying OOD, and if so what modifications were made to improve this?

For information regarding the configuration of Jupyter I have attached the form.yml and the submit.yml.erb from the /var/www/ood/apps/sys/Jupyter/ directory. Also, I have attached the cluster.d configuration file for the cluster from the path /etc/ood/config/clusters.d/.

Please let me know if there is any other information that I can provide. Thanks for all the help!
ood_form_yml.txt (1.2 KB)
submit_ym_erb.txt (1.2 KB)
clusterd_config.txt (438 Bytes)

[mjbludwig (Morgan - Cambridge Computer)]

Check the logs generated for the specific sessions that are hanging, default location is ~/ondemand/data/sys/dashboard/batch_connect/sys/<app>/output/<session id>/output.lo. This is where you will find the Jupyter output and based on the symptoms, I am guessing this is related to Jupyter and not OOD itself.

It would also be helpful to see how you are launching Jupyter so can you share the $APPROOT/template/script.sh.erb file?

[jeff.ohrstrom]

Hi and welcome!

I’d also check the apache/httpd logs. Those should have some timing information about how long things take. The network tab of your browser may also indicate something.

Beyond that, you can try with a simple curl from the machine OnDemand is on to replicate what apache/httpd is doing (it’s just proxying the request to the compute node at this point). If it’s (generally) fast, then it could be the case that apache/httpd is slow to proxy things. If it’s slow too, then there’s some network latency somewhere between the two machines.

[apelkey (Alexander Pelkey)]

Thanks for the responses!

I checked the output.log under a session ID which hung (see attached). The interesting bit from this log is the following:

[I 2025-07-30 08:38:41.494 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2025-07-30 08:38:41.494 ServerApp] http://localhost:49468/node/hub10/49468/lab
[I 2025-07-30 08:38:41.494 ServerApp] http://127.0.0.1:49468/node/hub10/49468/lab
[I 2025-07-30 08:38:41.494 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2025-07-30 08:38:41.503 ServerApp] No web browser found: Error(‘could not locate runnable browser’).
Discovered Jupyter Notebook server listening on port 49468!
TIMING - Wait ended at: Wed Jul 30 08:38:41 PDT 2025

I am not sure if the “No web browser found” error is a red herring or not? It seems that Jupyter is able to start immediately after that line. I have also attached the entire log output below.

regarding the curl testing, I tried executing the following from the host where OOD is running:

[apelkey@login01 ~]$ time curl -u $username:$password https://scu-ood.wave.scu.edu:443/node/hub10/49468/lab

real 0m7.156s
user 0m0.005s
sys 0m0.009s

Was this the correct test that you were recommending?

Inspecting the logs under /var/log/httpd/error.log (watching them with tail -f) I see the same block of code being repeated, but I could not identify any timing information (see attached).

I have never really played around with the network tab in Firefox, but hovering over the grey turtles reports a “slow server response time”. Im curious if you might know what would be the cause of these turtles. Is the server in question the Apache server?

Lastly, I have attached the script.sh.erb under the /templates directory.
httpd_error_log.txt (1.3 KB)
output_log.txt (5.1 KB)
script_sh_erb.txt (621 Bytes)

[jeff.ohrstrom]

@apelkey

Inspecting the logs under /var/log/httpd/error.log (watching them with tail -f) I see the same block of code being repeated, but I could not identify any timing information (see attached).

You’re looking for URLs like /node and /rnode. Or given your curl command example, /node. This is a request for a different URL.

@apelkey

I have never really played around with the network tab in Firefox, but hovering over the grey turtles reports a “slow server response time”. Im curious if you might know what would be the cause of these turtles. Is the server in question the Apache server?

These are not the right URLs. The URLs you’re looking for start with /node . I think you need to keep the link in the same tab (get rid of the target=“_blank” in the view.html.erb to keep it in the same tab) or use Chrome as it allows you to open the development tools when you open a new tab.

Given that curl is fairly quick, it may be that the original HTML is being returned quickly but there’s some javascript or something that’s taking a long time. The network tab from the client browser I think will tell us more, but again, you’ve got to see traffic to the URL in question. The traffic we’re seeing there is the ‘My interactive Sessions’ page which is not Jupyter.

[apelkey (Alexander Pelkey)]

Got it. Thank you very much for the explanation.

I was able to find the correct entries in the /var/log/httpd/error.log, and it looks to be executing at similar speeds to the curl test (see attached). Mainly, I saw this bit of data from the log:

time_proxy=“4.493”

Regarding the network tab, I believe I was able to get it recording the correct information following your steps. The screenshot below is from the Jupyter url itself:

I saw the red “error” icon at the top, so I figured I would expand that as well to include as much information as possible. Regrettably, I did not sort the Time column, and so I dont believe this screenshot reflects what took the longest. The screenshot is interesting, however, because it showcases the issue that has been occurring. The screen on the left is the blank screen that the Jupyter session ends up in.

Because I did not filter the time column I ran another test, and this time the Jupyter page was able to load (but after quite some time.. ~7.5 min it looks like?). Here is the resulting data from the network tab of that run:

httpd_correct_error_log.txt (1.3 KB)

[apelkey (Alexander Pelkey)]

Continuing down the rabbit hole, as I mentioned the behavior is intermittent. After gathering the data from the previous screenshots I deleted the Jupyter instance in OOD and started another. This time, the software kicked off in less than a second. I figured I would capture the network tab information for this “successful” run and leave it here as well for analysis.

[jeff.ohrstrom]

OK - looks like frrom your screenshot of a blank page and the network tab, the actual network seems OK. That is you’re able to connect, get the HTML page (the very first entry) then download all the assets.

This then leads me to believe there’s an issue on the client side. Like the javascript on the client gets stuck somewhere or something. Instead of looking at the network tab, maybe there’s something in the console tab. I’d also ask if you have any browser extensions that could be causing an issue. Lastly I’d suggest trying a different browser altogether and seeing if that makes any difference.

[apelkey (Alexander Pelkey)]

I went ahead and watched the console tab for the Jupyter URL for all three web browsers installed on my machine (Firefox, Chrome, and Safari). The screenshots below show which browser is being used in the top left of the image (sorry if its small). It appears that each time Jupyter fails there is an error associated with a promise (I am assuming this is the Javascript issue you were referring too).

Furthermore, after these failed attempts, Jupyter/OOD started working flawlessly again (no latency). I went ahead and got some console output from sessions which started instantly:

What is even more interesting is that while writing this post I left the “successful” Jupyter session open, and the console began to throw errors. Once the errors popped up, the session became pretty much useless (nothing would open/return).

I went ahead and tried to refresh the page to see what would happen, and a slew of network errors popped up. Once these errors started to appear, the Jupyter session refreshed into a blank white screen.

Lastly, regarding browser extensions, I checked and in both Firefox and Chome and I only have a “PrinterLogic Extension”. I can not seem to remove or disable this, and it seems to have been installed by the university that gave me this laptop. Safari had no browser extensions installed. My work device does have “ClearPass OnGuard” and “SentinelOne” (lol) installed which made me wonder if this was posing a threat. To be definite, I accessed the OOD instance from my personal laptop (with no browsers extensions installed or OnGuard or SentinelOne), and the Jupyter issue was present.

[jeff.ohrstrom]

OK, maybe it’s an issue of apache/httpd performance. For example you cannot make so many requests at a time currently. Maybe apache/httpd is becoming unresponsive sporadically. In an early image of the network tab you can see many requests in flight. Maybe some are begin queued beyond a certain timeout resulting in a failure.

This is at least the only guess I have at the moment.

Apache httpd tips — Open OnDemand 4.0.0 documentation

[apelkey (Alexander Pelkey)]

Hey Jeff!

I went ahead and toyed with the httpd performance settings under the mpm.conf file (specifically /etc/httpd/conf.modules.d/00-mpm.conf). I tried quite a few settings, and ultimately landed around here:

<IfModule mpm_event_module>

  # ServerLimit is MaxRequestWorkers / ThreadsPerChild
  ServerLimit            64
  StartServers           8
  MaxRequestWorkers      250
  MinSpareThreads        75
  MaxSpareThreads        250
  ThreadsPerChild        25
  MaxRequestsPerChild    0
  ThreadLimit            256
  ListenBacklog          255
</IfModule>

OOD is running on one of our login nodes, which has the follow specs:

[root@login01 conf.modules.d]# lscpu | grep -i "cpu(s)"
CPU(s):                               64

[root@login01 conf.modules.d]# free -g
               total        used        free      shared  buff/cache   available
Mem:             125           8         117           5           5         116
Swap:              0           0           0

Unfortunately modifying this file did not resolve the intermittent latency issues (let me know if I was way off with the parameters).

I have been doing some more digging, however, and am beginning to wonder if the latency issues are due to authentication.

I changed the LogLevel for httpd to debug, and begain tailing the file /var/log/httpd/error.log.

Within the OOD portal, when I would select another tab, the logs would hang at this point:

[Tue Aug 05 17:30:53.679694 2025] [ssl:debug] [pid 63093:tid 63233] ssl_engine_kernel.c(415): [client 172.31.32.132:63440] AH02034: Initial (No.1) HTTPS request received for child 1809 (server scu-ood.wave.scu.edu:443), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/
[Tue Aug 05 17:30:53.679838 2025] [authz_core:debug] [pid 63093:tid 63233] mod_authz_core.c(815): [client 172.31.32.132:63440] AH01626: authorization result of Require valid-user : denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/
[Tue Aug 05 17:30:53.679852 2025] [authz_core:debug] [pid 63093:tid 63233] mod_authz_core.c(815): [client 172.31.32.132:63440] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/
[Tue Aug 05 17:30:54.713600 2025] [authnz_pam:info] [pid 63093:tid 63233] [client 172.31.32.132:63440] PAM authentication passed for user apelkey, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/

Furthermore, regarding the JupyterHub application, when the app would hang or fail to launch, the following logs would be repeated:

[Tue Aug 05 17:08:27.547285 2025] [lua:info] [pid 62909:tid 63096] [client 172.31.32.132:62554] req_accept_language="en-us,en;q=0.9" req_method="POST" remote_user="apelkey" req_origin="https://scu-ood.wave.scu.edu" req_content_type="application/x-www-form-urlencoded" res_content_length="0" req_server_name="scu-ood.wave.scu.edu" res_location="/node/cpu04/38655/" req_hostname="scu-ood.wave.scu.edu" log_time="2025-08-05T17:08:27.547226.0Z" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" res_content_type="text/html; charset=utf-8" req_protocol="HTTP/1.1" res_content_encoding="" req_filename="proxy:http://cpu04:38655/node/cpu04/38655/login" req_status="302" res_content_language="" local_user="apelkey" allowed_hosts="scu-ood.wave.scu.edu" req_uri="/node/cpu04/38655/login" req_handler="proxy:http://cpu04:38655" res_content_disp="" res_content_location="" log_id="aJI6i6iWHmbGgJ1bxi6ixgAABAw" req_cache_control="max-age=0" time_user_map="0.002" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_is_https="true" req_referer="https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions" log_hook="ood" req_port="443" req_user_ip="172.31.32.132" time_proxy="3.753" req_accept_charset="" req_is_websocket="false" req_accept_encoding="gzip, deflate, br, zstd", referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.552674 2025] [ssl:debug] [pid 62909:tid 63097] ssl_engine_kernel.c(415): [client 172.31.32.132:62554] AH02034: Subsequent (No.2) HTTPS request received for child 1037 (server scu-ood.wave.scu.edu:443), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.552734 2025] [authz_core:debug] [pid 62909:tid 63097] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of Require valid-user : denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.552739 2025] [authz_core:debug] [pid 62909:tid 63097] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.580092 2025] [authnz_pam:info] [pid 62909:tid 63097] [client 172.31.32.132:62554] PAM authentication passed for user apelkey, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582120 2025] [authz_core:debug] [pid 62909:tid 63097] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of Require valid-user : granted, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582126 2025] [authz_core:debug] [pid 62909:tid 63097] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of <RequireAny>: granted, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582426 2025] [proxy:debug] [pid 62909:tid 63097] mod_proxy.c(1467): [client 172.31.32.132:62554] AH01143: Running scheme http handler (attempt 0), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582435 2025] [proxy_ajp:debug] [pid 62909:tid 63097] mod_proxy_ajp.c(785): [client 172.31.32.132:62554] AH00894: declining URL http://cpu04:38655/node/cpu04/38655/, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582440 2025] [proxy_fcgi:debug] [pid 62909:tid 63097] mod_proxy_fcgi.c(1091): [client 172.31.32.132:62554] AH01076: url: http://cpu04:38655/node/cpu04/38655/ proxyname: (null) proxyport: 0, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582444 2025] [proxy_fcgi:debug] [pid 62909:tid 63097] mod_proxy_fcgi.c(1096): [client 172.31.32.132:62554] AH01077: declining URL http://cpu04:38655/node/cpu04/38655/, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582447 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(2797): AH00942: http: has acquired connection for (*:80)
[Tue Aug 05 17:08:27.582451 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(3242): [client 172.31.32.132:62554] AH00944: connecting http://cpu04:38655/node/cpu04/38655/ to cpu04:38655, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582495 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(3450): [client 172.31.32.132:62554] AH00947: connecting /node/cpu04/38655/ to 10.160.7.104:38655 (cpu04:38655), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.582711 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(3956): AH02824: http: connection established with 10.160.7.104:38655 (cpu04:38655)
[Tue Aug 05 17:08:27.582718 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(4144): AH00962: http: connection complete to 10.160.7.104:38655 (cpu04)
[Tue Aug 05 17:08:27.584165 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(2813): AH00943: *: has released connection for (*:80)
[Tue Aug 05 17:08:27.584171 2025] [proxy:debug] [pid 62909:tid 63097] proxy_util.c(4068): [remote 10.160.7.104:38655] AH02642: proxy: connection shutdown
[Tue Aug 05 17:08:27.584426 2025] [lua:info] [pid 62909:tid 63097] [client 172.31.32.132:62554] req_user_ip="172.31.32.132" req_referer="https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions" res_content_type="text/html; charset=utf-8" res_content_disp="" remote_user="apelkey" log_id="aJI6i6iWHmbGgJ1bxi6ixwAABA0" req_cache_control="max-age=0" req_port="443" req_content_type="" res_content_length="0" req_protocol="HTTP/1.1" req_handler="proxy:http://cpu04:38655" res_content_encoding="" req_origin="" res_content_location="" allowed_hosts="scu-ood.wave.scu.edu" req_is_websocket="false" log_time="2025-08-05T17:08:27.584370.0Z" log_hook="ood" req_status="302" req_accept_charset="" time_user_map="0.001" req_filename="proxy:http://cpu04:38655/node/cpu04/38655/" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" local_user="apelkey" time_proxy="1.995" res_location="/node/cpu04/38655/lab?" res_content_language="" req_accept_language="en-us,en;q=0.9" req_uri="/node/cpu04/38655/" req_is_https="true" req_server_name="scu-ood.wave.scu.edu" req_hostname="scu-ood.wave.scu.edu" req_method="GET" req_accept_encoding="gzip, deflate, br, zstd" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.589688 2025] [ssl:debug] [pid 62909:tid 63100] ssl_engine_kernel.c(415): [client 172.31.32.132:62554] AH02034: Subsequent (No.3) HTTPS request received for child 1038 (server scu-ood.wave.scu.edu:443), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.589783 2025] [authz_core:debug] [pid 62909:tid 63100] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of Require valid-user : denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.589789 2025] [authz_core:debug] [pid 62909:tid 63100] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.614805 2025] [authnz_pam:info] [pid 62909:tid 63100] [client 172.31.32.132:62554] PAM authentication passed for user apelkey, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.616891 2025] [authz_core:debug] [pid 62909:tid 63100] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of Require valid-user : granted, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.616897 2025] [authz_core:debug] [pid 62909:tid 63100] mod_authz_core.c(815): [client 172.31.32.132:62554] AH01626: authorization result of <RequireAny>: granted, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617204 2025] [proxy:debug] [pid 62909:tid 63100] mod_proxy.c(1467): [client 172.31.32.132:62554] AH01143: Running scheme http handler (attempt 0), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617211 2025] [proxy_ajp:debug] [pid 62909:tid 63100] mod_proxy_ajp.c(785): [client 172.31.32.132:62554] AH00894: declining URL http://cpu04:38655/node/cpu04/38655/lab?, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617215 2025] [proxy_fcgi:debug] [pid 62909:tid 63100] mod_proxy_fcgi.c(1091): [client 172.31.32.132:62554] AH01076: url: http://cpu04:38655/node/cpu04/38655/lab? proxyname: (null) proxyport: 0, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617218 2025] [proxy_fcgi:debug] [pid 62909:tid 63100] mod_proxy_fcgi.c(1096): [client 172.31.32.132:62554] AH01077: declining URL http://cpu04:38655/node/cpu04/38655/lab?, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617222 2025] [proxy:debug] [pid 62909:tid 63100] proxy_util.c(2797): AH00942: http: has acquired connection for (*:80)
[Tue Aug 05 17:08:27.617226 2025] [proxy:debug] [pid 62909:tid 63100] proxy_util.c(3242): [client 172.31.32.132:62554] AH00944: connecting http://cpu04:38655/node/cpu04/38655/lab? to cpu04:38655, referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions
[Tue Aug 05 17:08:27.617269 2025] [proxy:debug] [pid 62909:tid 63100] proxy_util.c(3450): [client 172.31.32.132:62554] AH00947: connecting /node/cpu04/38655/lab? to 10.160.7.104:38655 (cpu04:38655), referer: https://scu-ood.wave.scu.edu/pun/sys/dashboard/batch_connect/sessions

Apologies for the block of code, I have attached the output file below for further inspection.

I took a look at the OOD architecture referenced here and I am wondering if the issue might be between the apache server and the authentication, whether it be from apache itself or the auth provider. When OOD does run “fast” I see no hang time in the logs around the authentication portion, and in regard to Jupyter I see PAM authentication marked as successful and then the application launches.

Could this be a plausible cause for the behavior we are seeing? If so, would have any advice as to how I can debug this further?

Lastly, authentication for this setup was handled via PAM on a Rocky9 host provisioned via WareWulf. I am currently talking with the universities infrastructure team to see if we can tie in OIDC with Dex to the existing infrastructure, but there is no ETA on that. Here is the configuration that was used for auth, as well as the docs which were referenced. Add PAM Authentication — Open OnDemand 1.8.12 documentation

auth:
  - 'AuthType Basic'
  - 'AuthName "Open OnDemand"'
  - 'AuthBasicProvider PAM'
  - 'AuthPAMService ood'
  - 'Require valid-user'

# Lua regular expression used to map authenticated-user to system-user
# This configuration is ignored if user_map_cmd is defined
# Example:
#     user_map_match: '^([^@]+)@.*$'
# Default: '.*'
user_map_match: '.*'

httpd_error.log.txt (325.9 KB)

[jeff.ohrstrom]

Sadly, I’m at a bit of a loss here. Maybe it is an issue with PAM/Basic auth given some are denied?

But that may be a red-herring, as you’re timing out. If it were unauthenticated, you’d directly get a 403 Forbiddenerror.

I would try to trace one single request for one single asset that failed. Find something unique in the network tab (a js file or css file or similar that timed out) and try to trace it through the httpd logs, filtering out all the other requests. You should be able to trace it through the tid values here. I believe pid is the PID of the server, where tid is more like task id, for all the child processes.

[apelkey (Alexander Pelkey)]

Hey Jeff!

I believe I was able to resolve the issue.

After seeing the logs hang at the authentication section I reached out to our campus infrastructure team which manages the AD/kerberos DC’s. Our OOD instance is being served out from our login nodes (we have no other nodes available) which have PAM configured to use sssd to reach out to the kerberos realm for auth.

The team responded that the default realm consists of DC’s in GCP, and this was likely getting backed up with the requests. They gave me the realm for an on-campus DC, and once the switch was made in sssd the OOD portal is working flawlessly!

Apologies to give you the run around when the root cause was a local issue. Nevertheless, the help and troubleshooting has given me a much better understanding of OOD and Apache, and so I appreciate all the help. Thanks!

[jeff.ohrstrom]

Nice! All’s well that ends well to me!