We have an Open OnDemand portal (OOD) set up which uses Shibboleth for authentication. This is working fine from a functional perspective. However, our institution’s uses Splunk monitoring and the monitoring team has reached out to my team because there’s a very high number of OPTIONS requests (preflight requests) coming from our Open OnDemand server.
Has anyone run into this issue and found a way to optimize/reduce the number of preflight requests? I’ve read that preflight requests can be optimized if one is able to set some web server parameters such as Access-Control-Max-Age, Access-Control-Allow-Headers, Access-Control-Allow-Methods, and/or Access-Control-Allow-Credentials. I’ve looked at the OOD code some, but am unsure where I would update/add a value for these parameters for OOD.
Where can these parameters be set? Also, we are having a difficult time monitoring the OPTIONS traffic from the browser end. If anyone has suggestions for how, we’d be interested in any tips.
Thanks in advance!
Michele Co
Research Computing
University of Virginia
No issues on the section you put it in (though you may have moved it).
You know it’s from the OnDemand server because of the http referrer or similar? Can you give me the URL that’s making the client issue an OPTIONS call?
I’m struggling to see where we’d make this call. In the XDMoD integration perhaps? That’s the only one that comes to mind.
We’ve been looking at the Splunk logs and it looks like the OPTIONS calls are coming from web browsers that are using applications provided by our Open OnDemand server, but not directly from our Open OnDemand server. Currently, it looks like our IdP servers are receiving between 30,000-50,000 OPTIONS requests per hour. I selected an IP address that comprised the highest number of OPTIONS requests (in my sample period) and tried to cross reference it with our httpd logs and for this particular instance, it seems to be a user’s JupyterLab session. I have not been able yet to investigate whether these requests are primarily associated with JupyterLab sessions.
I have been trying to find a way to see the HTTP OPTIONS request from the browser end when I start a sample OOD session, but it appears that for security reasons, most browser developer tools no longer provide the ability to track this sort of network traffic from the browser side.
From what I understand an OPTIONS request is a request which asks which operations are allowed. I’m able to see records like the following from our Splunk instance (I’ve obscured the IP address with X.X.X.X), but they are all of this form:
When I decode the SAML request from the Splunk records, they are of the form:
<samlp:AuthnRequest xmlns:samlp=“urn:oasis:names:tc:SAML:2.0:protocol” AssertionConsumerServiceURL=“https://ood.hpc.virginia.edu/Shibboleth.sso/SAML2/POST” Destination=“NetBadge Message” ID=“_1bbbd07bc099da23a5d7e2e5fed757c8” IssueInstant=“2025-01-27T19:29:53Z” ProtocolBinding=“urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST” Version=“2.0”><saml:Issuer xmlns:saml=“urn:oasis:names:tc:SAML:2.0:assertion”>https://ood.hpc.virginia.edu/shibboleth</saml:Issuer><samlp:NameIDPolicy AllowCreate=“1”/></samlp:AuthnRequest>
I’m hoping to get some ideas for how to continue to track down what might be happening, so any ideas you have would be helpful.
Someone else who runs Shibboleth may need to jump in here. I admit I’m a bit out of my depths here.
I just tried on OSC systems and I see no OPTIONS calls being made (we use OIDC authentication). What’s really puzzling to me is that it continues to make calls to Shibboleth. What I mean is, after you login, why would the user be interacting with Shibboleth? Could be something I’m missing with Shib - OIDC doesn’t work that way. Once you login you don’t have to contact the IdP until the token expires (or httpd restarts).
I tried to play around with this setting, but no luck so far.
If you want to add headers like Access-Control-Allow-Methods or similar you can use these configurations to apply any apache configurations to the OOD apache configuration.
Thank you for the pointers. We have not yet tried making any header changes while we have been trying to better understand what might be happening.
For high-level context, our Open OnDemand server is set up to use Shibboleth authentication and we use Slurm for job submission to our HPC cluster. We’ve found a number of examples of JupyterHub sessions served through our Open OnDemand displaying the pattern described below. We’re not sure yet if this pattern applies to other apps.
We’ve been looking through our http access logs and have been noticing the following access pattern.
A user will submit a job request to slurm with a requested time limit.
At the job start time, we see in our http logs that there are many http requests that are returning 200 OK.
At the Slurm job end time, we start seeing requests that return 503 Service Unavailable. This corresponds to Slurm terminating the Open OnDemand session. The 503s continue for roughly 4 hours from the examples we’ve found.
After the 503s, we see requests that returning 302 which we believe might be “temporary redirect”. We believe these requests might be triggering the high number of OPTIONS requests from the user’s web browser to our Shibboleth IdP.
After an Open OnDemand interactive app is terminated, how is the browser notified that the session is terminated?
Michele Co
HPC Engineer
ITS - Research Computing
University of Virginia
The OPTIONS requests are coming from the browser to the Shibboleth IDP. This happens when the Shibboleth session lifetime expires. The browser is automatically polling OOD with success, but when the Shib session expires, the browser is no longer authenticated, and the Shibboleth Apache module starts redirecting the browser to the Shibboleth IDP for login. Since this is an automatic poll from the browser (javascript), login cannot occur. But the browser just keeps on polling, gets redirected, and sends the OPTIONS requests.
Is there any way to reduce the frequency of the browser polling? We are running OOD 3.1 and we found the POLL_DELAY environment variable. We were able to reduce some (but not all) of the polling from every 10 sec to every minute. I don’t think we can eliminate the problem, but reducing the frequency of the OPTIONS requests would help. Besides POLL_DELAY are there any other configurations that affect browser polling? Looking at the httpd access log, POLL_DELAY controls the polling of URLs that begin with /pun/ but we keep seeing frequent polling of URLs that begin with /node/ or /rnode/. Any way to reduce those?
Unfortunately, I don’t think this was fixed until 4.0. In 4.0 when it encounters an error - it’ll stop polling and puts an alert on the page with something to the effect of you're no longer logged in, please refresh the page.
Actually I left my browser open yesterday, so I now have the message waiting for me when I opened my browser this morning, so I can show you directly the error users will see when they’re no longer authenticated.
This looks like exactly the behavior we need. I hope it works with Shibboleth. I think the only clue that the browser (javascript) sees is a 302 redirect to the Shibboleth IDP, so I hope that triggers the browser alert.
I should also note that 4.0.1 should be out around the beginning of next month and it has some quality of life bug fixes that you may run into in 4.0.0.
