Home page of ondemand redirecting to compute node

So, what’s happening is, sometimes when some of our users try to view the homepage running on the headnode,
We are receiving a proxy error (screenshot below), it seems to be redirecting us to a compute node

Sometimes, we see this on clicking the domain name, before it even redirects to the dashboard

This is redirecting a user to a compute node running another user’s jupyter job

We believe both are the same issue although not quite sure,

There’s nothing specific in the httpd logs that says why this is happening, other than just the log of a different URL, just like you see in the browser.

This doesn’t happen all the time, happens rarely to some random user at a time, becoming hard to figure out why this is happening. Has anyone else faced the same problem?

We faced this with v2.0.29 as well as with v2.0.31.

I think this can happen when the origin server (in this case Jupyter) is first booting up. We’re using apache to proxy back to it and sometimes it’s not ready (even though the card says it is).

The after.sh script is supposed to catch this. It may not be enough simply for the port to be open. Maybe you could enhance the script to curl and expect a 200 response. Once the after.sh script exists, the card will be updated with a ‘connect to’ button.

So I’d say having stricter checks in the after.sh may help this.

Thank you for looking into it, I’m unsure if I explained it correctly, so rephrasing it here.
So both the scenarios mentioned above happens when the user is not attempting anything with jupyter, he’s just trying to see the dashboard and he gets a 404: Not Found with the header reading jupyter or a proxy error, where as other users are able to use the application correctly and launch jobs.

And sure, it makes sense to have stricter checks in after.sh, do you think this will help with the above situation? If so can you elaborate, thanks.

I’m not sure how you’d get the Jupyter error from dashboard pages. But let me see. I’m not sure how to resolve the issue for regular dashboard pages. We’re cleaning up our old ondemand-gem RPMs so that may help the first image’s issue (getting a 404 from URLs that are not /rnode & /node)

If you get error pages for URLs that are /rnode & /node and return Jupyter pages - that’s a Jupyter issue. I’m not sure how/if that’s even possible to get Jupyter pages from URLs that start with /pun.

If a user reports the Jupyter error page - they’re attempting to access Jupyter through a /rnode or /node URL, there’s just no way we’d return that page from a URL that starts with /pun.

Yes, that’s the same doubt we are having, we are not sure why jupyter even shows up, this happens way before even redirecting to dashboard happens.
Usually, domain name rediects to <domain_name>/pun/sys/dashboard.
We get the jupyter error even before that and logs weren’t helpful either, that’s why we wanted to see if anyone else has faced anything similar to this.

On a sidenote, we are also seeing this issue,

We found that this issue resolves, once we restart apache but this issue has been persistent, i.e it keeps happening to some user or the other.
Looking for any other possible root causes for this one.
@jeff.ohrstrom Happen to come across anything similar, after our conversation?