Shell app changes since v3.1 (esp timeout disconnections)

Get Help
1

First off, if you have end-users reporting their OOD shell app is disconnecting far more quickly than it used to, with the message…

Your connection to the remote server has been terminated.

…then you should know that there were a couple changes to how OOD handled inactive shell connections (over a couple of 3.1.x versions) and if you want backgrounded shell tabs to last longer than 5 (?) minutes, then you’ll have to make some configuration changes. TL;DR: you may need to create a config file for your shell app (at /etc/ood/config/apps/shell/env) and drop in the magic timeout string to specify your timeout, and may also (?) need to enable shell-ping-pong.

IF I read the following release notes and announcements correctly, “the new system” was probably (?) introduced in v3.1.0, then sorta nerfed in v3.1.9 to default to a safer security posture due to some unforeseen issues:

v3.1 release-notes: improved shell connections

The shell app has been updated so that shell sessions can last longer. The app is better at keeping the connection alive, meaning users will maintain shell sessions for longer.

This, however, is disabled by default since 3.1.9. You can see more details in Enable and configure Shell Ping Pong.

v3.1.0 announcement

Better shell connectivity.

v3.1.9 announcement

Ping Ponging. Ping ponging in the shell application has been disabled by default. When ping ponging is enabled, the shell connection can stay active for longer than the authentication session. Open OnDemand version 3.1 added this feature, but after unanticipated findings of the shell session continuing after the authentication token expired, we have decided to turn ping ponging off by default.

I think that decision was fair but I don’t think it was clear to many of us that this series of changes could add up to complaints from users. But before I make too many assumptions, I want to ask for some clarifications:

  • am I correct in concluding that idle shell apps are effectively treated differently since v3.1, at least from an end-user’s perspective?
  • when pingpong is disabled: will v3.1 neither use “the new system”, nor failover to “the old system”?
    (i.e. will Apache appear to close connections more aggressively than in v3.0?)
  • is enabling OOD_SHELL_PING_PONG indeed always required for OOD_SHELL_INACTIVE_TIMEOUT_MS & OOD_SHELL_MAX_DURATION_MS to have any effect? (testing inactive_timeout actually seemed to help my v3.1.15 system just now, even though pingponging is disabled, which confuses me)
  • are both of the two points above true for both v3.1 and v4.0?

I’m asking because in some other related posts, Jeff mentions one needs to enable pingpongs otherwise INACTIVE_TIMEOUT_MS & MAX_DURATION_MS are disabled entirely. But I don’t think this is made clear by the shell-ping-pong docs. (To my ear, “In addition to enabling or disabling ping pongs, there are other settings you may wish to change” sounds like pingpong is optional.)

It also sounds like, since v3.1, nothing will actively keep idle shells active when pingpong is disabled because OOD does not fall-back to whatever was done in “the old system”…

Ood shell inactive timeout

[from /etc/ood/config/apps/shell/env] OOD_SHELL_INACTIVE_TIMEOUT_MS=600000 [but] The connection to the shell session has been terminated in < 1 minute

…

You have to enable ping pongs by setting the OOD_SHELL_PING_PONG environment variable. Otherwise, they’re disabled entirely.

oct 2024 Shell access timeout after 60secs

Upgraded to 3.1.9 and now the shell timeout less than a minute.

…

Is there any way to completely disable this behavior and return to the old style that didn’t disconnect?

…

No, and you can blame me personally for this if you like. I made the decision to disable it by default because I figured we (the developers/maintainers) need to be conservative in the defaults

…

Also, I’ve implemented …/shell/env file, with the two setting, but without enabling ping pongs. I’ve had two sessions fail under these conditions.

…

You need to enable ping pongs to keep the connection alive. Otherwise you’d need some trick like backgrounding something that prints stuff at regular intervals or similar. apache’s going to disconnect the connection after 60 seconds if there’s no traffic (pings and pongs are the traffic that will keep the connection alive).

Thanks in advance for any clarifications here!

(And sorry for any bad conclusions above! I’ll fix up any misdirection as I’m corrected!)

-jason

2

Yes. With ping pongs enabled they can be idle for much longer. Without it I’m pretty sure the experience is the same as pre 3.1.0 versions.

I’m pretty sure it falls back to the “the old system” which is/was to do nothing. OOD_SHELL_INACTIVE_TIMEOUT_MS is still in the mix, but the default of 5 minutes is much longer than apache’s default of 60 seconds, so apache is going to time you out in this case as it had done before.

No. Users can do something like this in a shell to effectively have ping pongs. They themselves create the TCP activity that will keep the connection alive. They only have to do this more often than OOD_SHELL_INACTIVE_TIMEOUT_MS or apache’s timeout setting of 60 seconds to keep it alive, but they cannot keep the shell open for longer than OOD_SHELL_MAX_DURATION_MS regardless of how much TCP activity they create.

while true; do echo ''; sleep 5; done &

Yes for 3.1.9+ and 4.0.0+ and for the next versions and beyond.

I could have been wrong here, not thinking about the issue deeply enough. Without OOD_SHELL_PING_PONG enabled, the user has to generate TCP traffic by themselves, which generally they won’t/don’t, but I can now imagine a sophisticated user who backgrounds something that creates TCP traffic themselves to keep the connection alive.

So, in general it wouldn’t apply because apache times you out well before we will in most circumstances. Most users won’t do this, so it mostly won’t apply, but some sophisticated users may do this and find these guardrails.

I’m pretty sure “the old system” was to do nothing and let apache time you out which is what will happen if you don’t enable ping pongs.

No issues on having to fix up anything - this forum is for asking questions so it’s all good.