OpenOnDemand server generating too many OPTIONS/preflight requests

Hi OOD Forum,

We have an Open OnDemand portal (OOD) set up which uses Shibboleth for authentication. This is working fine from a functional perspective. However, our institution’s uses Splunk monitoring and the monitoring team has reached out to my team because there’s a very high number of OPTIONS requests (preflight requests) coming from our Open OnDemand server.

Has anyone run into this issue and found a way to optimize/reduce the number of preflight requests? I’ve read that preflight requests can be optimized if one is able to set some web server parameters such as Access-Control-Max-Age, Access-Control-Allow-Headers, Access-Control-Allow-Methods, and/or Access-Control-Allow-Credentials. I’ve looked at the OOD code some, but am unsure where I would update/add a value for these parameters for OOD.

Where can these parameters be set? Also, we are having a difficult time monitoring the OPTIONS traffic from the browser end. If anyone has suggestions for how, we’d be interested in any tips.

Thanks in advance!
Michele Co
Research Computing
University of Virginia

I’ve just realized that I posted in the wrong section of the Discourse forum, but don’t know how to move it.

No issues on the section you put it in (though you may have moved it).

You know it’s from the OnDemand server because of the http referrer or similar? Can you give me the URL that’s making the client issue an OPTIONS call?

I’m struggling to see where we’d make this call. In the XDMoD integration perhaps? That’s the only one that comes to mind.

Hi Jeff,

Thank you for your response.

We’ve been looking at the Splunk logs and it looks like the OPTIONS calls are coming from web browsers that are using applications provided by our Open OnDemand server, but not directly from our Open OnDemand server. Currently, it looks like our IdP servers are receiving between 30,000-50,000 OPTIONS requests per hour. I selected an IP address that comprised the highest number of OPTIONS requests (in my sample period) and tried to cross reference it with our httpd logs and for this particular instance, it seems to be a user’s JupyterLab session. I have not been able yet to investigate whether these requests are primarily associated with JupyterLab sessions.

I have been trying to find a way to see the HTTP OPTIONS request from the browser end when I start a sample OOD session, but it appears that for security reasons, most browser developer tools no longer provide the ability to track this sort of network traffic from the browser side.

From what I understand an OPTIONS request is a request which asks which operations are allowed. I’m able to see records like the following from our Splunk instance (I’ve obscured the IP address with X.X.X.X), but they are all of this form:

jetty_access: X.X.X.X - - [27/Jan/2025:19:29:53 +0000] “OPTIONS /idp/profile/SAML2/Redirect/SSO?SAMLRequest=fZLRToMwFIZfhfR%2BFFgI0gwS3C5cMpUM9MIbU%2BjZaAIt9pSpby8bU7eb3bZ%2Fv3P%2BL10g79qeZYNt1BY%2BBkDrfHWtQna6SMhgFNMcJTLFO0Bma1ZkjxsWuB7rjba61i1xMkQwVmq11AqHDkwB5iBreNluEtJY2yOjVGvhNn3tHqTZSyW5C2KgRSOrSrdgGxdR0yM7oPlzURJnNS4jFT9i%2FyE45qXoXWnxGjQe0nGhnWzhTNmCkAZqS4vimTjrVULe%2FaqqhBdVtRfHggdzHooIAgh3IKIwqu%2FGGOIAa4WWK5uQwAvCmefPgqj0YxbELJy%2FESc%2F976XSki1vy2pmkLIHsoyn03NXsHgqdUYIOniqJqdBpsL%2Bbex%2FNc4SW%2F6xT%2B%2FC3oxZxras6cRvF7lupX1t5O1rf5cGuAWEuITmk5Prr9H%2BgM%3D&RelayState=ss%3Amem%3A615a75527ad07fd9c56bad04f878a1cdc428fd2ea360e75affc5a409a268911c&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256&Signature=Pkvtv8ZIhc2gadSZjvEUW89qGVwuxAm9jLzGizFg%2F46LiQWg9NvNyYKmpUZQeuxL%2FyY%2FH5ICMqCExD3jGt9abU073NmJHSBTysFMJK0OlCf40U9DvI7SK%2FYJK61Lp0AiUZOycctzQxVJG3oJJupqsq1U0dmP27WhTm%2B1zl%2F3uBbDkeX3T5KUeYJhk016RfZsbGM13u3Fk5b52OXiOn8tHp6un8cB7jsPs0mYtVQt5X9eLerl9OICo38FpHv7Y6qauipC3K3Db70vgZjQFt7fKfQT1l9GmZXql%2B0pTlfCxkm%2Bux657KsrsNXsyPmIpgYt15FFPsNjpVhGYDvC9zjHw11lcx06gfnHfIJJLtC4ZSDWPD3dME%2BuUnfdfm%2BTFprAOptl0j5TPmdV2uQ6FAIjF4ZXP5p%2F%2F1Omc80F6bdsbr4q2AulgE0COSMfr6T9UcQmL555T6ynQEAFUAzj6pD%2FE%2BqQB595KJwUCSU%2Fexgwtgs5MjaNvJoMCMkUY36j8xAq HTTP/1.1” 403 20 “https://ood.hpc.virginia.edu/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36”

When I decode the SAML request from the Splunk records, they are of the form:
<samlp:AuthnRequest xmlns:samlp=“urn:oasis:names:tc:SAML:2.0:protocol” AssertionConsumerServiceURL=“https://ood.hpc.virginia.edu/Shibboleth.sso/SAML2/POST” Destination=“NetBadge Message” ID=“_1bbbd07bc099da23a5d7e2e5fed757c8” IssueInstant=“2025-01-27T19:29:53Z” ProtocolBinding=“urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST” Version=“2.0”><saml:Issuer xmlns:saml=“urn:oasis:names:tc:SAML:2.0:assertion”>https://ood.hpc.virginia.edu/shibboleth</saml:Issuer><samlp:NameIDPolicy AllowCreate=“1”/></samlp:AuthnRequest>

I’m hoping to get some ideas for how to continue to track down what might be happening, so any ideas you have would be helpful.

Michele

Someone else who runs Shibboleth may need to jump in here. I admit I’m a bit out of my depths here.

I just tried on OSC systems and I see no OPTIONS calls being made (we use OIDC authentication). What’s really puzzling to me is that it continues to make calls to Shibboleth. What I mean is, after you login, why would the user be interacting with Shibboleth? Could be something I’m missing with Shib - OIDC doesn’t work that way. Once you login you don’t have to contact the IdP until the token expires (or httpd restarts).

I tried to play around with this setting, but no luck so far.

If you want to add headers like Access-Control-Allow-Methods or similar you can use these configurations to apply any apache configurations to the OOD apache configuration.

For the entire virtual host
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html#custom-vhost-directives

or attach it to the locations
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html#custom-location-directives

Hi Jeff,

Thank you for the pointers. We have not yet tried making any header changes while we have been trying to better understand what might be happening.

For high-level context, our Open OnDemand server is set up to use Shibboleth authentication and we use Slurm for job submission to our HPC cluster. We’ve found a number of examples of JupyterHub sessions served through our Open OnDemand displaying the pattern described below. We’re not sure yet if this pattern applies to other apps.

We’ve been looking through our http access logs and have been noticing the following access pattern.

  • A user will submit a job request to slurm with a requested time limit.
  • At the job start time, we see in our http logs that there are many http requests that are returning 200 OK.
  • At the Slurm job end time, we start seeing requests that return 503 Service Unavailable. This corresponds to Slurm terminating the Open OnDemand session. The 503s continue for roughly 4 hours from the examples we’ve found.
  • After the 503s, we see requests that returning 302 which we believe might be “temporary redirect”. We believe these requests might be triggering the high number of OPTIONS requests from the user’s web browser to our Shibboleth IdP.

After an Open OnDemand interactive app is terminated, how is the browser notified that the session is terminated?

Michele Co
HPC Engineer
ITS - Research Computing
University of Virginia