I can sometimes replicate this simply by disconnecting from the session and reloading the browser window. Then when I try to reconnect I see this error. The “view only” link works to connect to the session and see what is running.
I’ve been unable to find a fix for this and the users have to cancel their jobs and start over. Does anyone have any suggestions?
When you say “they try to reconnect” are you saying they try to connect on the same page/refresh that page or are they trying to reconnect through the button Launch <cluster> Desktop>? The former won’t work because these passwords are one time use. Every time you connect using a given password, it regenerates a new one for the next time. The latter should continue to work because the card should be able to pick up the newer password.
When I refresh the page once, I just get authentication failed. But if I continue to refresh the page - I get that message - authentication failed. client temporarily disabled.
However if I navigate back to OnDemand, the Launch <cluster> Desktop button still works using the new password.
So I think it may be a user error - refreshing that page won’t work, but using the Launch <cluster> Desktop button in OnDemand should continue to work.
If that button stops working, let me know and we’ll continue to look at it.
No problem! We were looking into things on our end too. I am able to replicate this on demand (no pun intended!) now. I start up a session (desktop, app, doesn’t matter) and click the button to launch it. Once that new tab or window with the session is open, it works. If I close the tab and click the button to launch it again, I get the connection error. We thought it might be a firewall issue but confirmed in our dev environment that isn’t the case. There isn’t anything in the user nginx logs or the job logs indicating a problem. Are there other logs we could be looking at that might indicate what might be causing the problem?
You should see messages like this in your output.log as you generate new passwords. Every time you successfully connect it should regenerate a new password. Maybe you’re clicking the card too soon as it hasn’t picked up the new password yet? Or maybe there’s some NFS lag where the file is being updated on the compute node, but the web node is still reading the old file?
When I click the button on our 4.2 instance, it disables it for a bit so I can’t click it that fast.
(polkit-mate-authentication-agent-1:3387467): polkit-mate-1-WARNING **: 10:57:01.157: Unable to determine the session we are in: No session for pid 3387467
Setting VNC password...
Generating connection YAML file...
Setting VNC password...
Generating connection YAML file...
Setting VNC password...
Generating connection YAML file...
Setting VNC password...
Generating connection YAML file...
Or maybe the page is failing to update the card. If you open your network tab on your ‘my interactive sessions’ page you should see something like this where the browser is querying the backend for new information (including a new password).
Doesn’t appear to be browser based. It’s happening in Chrome, Firefox, and Edge
I can’t replicate it immediately with interactive apps, including VNC based ones like the Matlab app. It’s only desktops. Eventually I can get the Matlab app to break by closing and reopening the browser tab with the session a few times.
When it’s working, I’m seeing the output.log file indicate that a new VNC password is generated every time I click the button to launch the app. When it’s not working, I only see this with the initial launch of the desktop but not afterwards
The non-VNC apps (Rstudio, Jupyter) work fine with no issues. Even if the session times out, the user can click the button and get a new one.
The view-only links work fine for all the apps
We’re not seeing anything in the user logs, job logs, or the network info in the browser. We do have plans to update to 4.x in the next few months but aren’t prepared to do that right now. This is causing an issue for users that start something running in their session and leave it. Then they can’t connect back to it and have to resubmit their jobs which is a big issue for our GPU users who wait a long time to get a node. We’re just unsure what else we can check or what could be causing this.
Is it specific users or are you able to consistently replicate it?
Here’s the PID I’m waiting for in my example (3662923) - you see it’s the script.sh. Odd that the tail on the PID would fail, but I’m guessing the actual PID/script.sh is still alive?
I tried to take a screen recording and I can’t replicate it consistently now. Very odd! I can even login to OOD from another browser and connect to a running session without issue. Right up until it stops working. They used to be immediately but now I have to connect a bunch of times before it finally fails. We’ll keep looking to see what might be causing it!
Do you have availability to meet later this week? I think I’d like to see it in person if you can replicate consistently, which I guess you can’t anymore. I’ll still think on it from my side.
Just tested quickly with turbovnc 3.1.2 and it seemed OK.
Thinking about this a little more - I wonder if a simple page refresh doesn’t fix it? That is when it starts happening, just refresh the ‘My Interactive Sessions’ page to pickup the new password would work.
I have tried to replicate in a 3.1 instance in development mode, but haven’t got much luck.
I’d say if you can replicate pretty consistently then we should set up a time to meet and debug it interactively since you’ve got a support subscription.
