The ttyd not working in OOD 4.1.1

Get Help
1

Hi developers,

After upgrading OOD from 4.0.8 to 4.1.1, an interactive application using ttyd no longer works. The ttyd application was developed based on the following URL:

https://code.oscer.ou.edu/OnDemand/ood-apps-nmsu/-/tree/main/shell

As you can see in view.html.erb, the application connects to ttyd running on the compute node using a URL with embedded user credentials:

https://<username>:<password>@<host>

After upgrading to OOD 4.1.1, the connection fails and the ttyd session cannot be established.

Was any security enhancement introduced in OOD 4.1.1 that affects URLs containing embedded credentials ?

Any guidance would be appreciated.

2

No changes off the top of my head - but I’ll look into it further.

3

I’m seeing the same issue on our 4.1.1 instance but still unsure of the cause.

4

I have received information from a business associate regarding this issue.

It appears to be related to the following changes:

This change appears to have caused Basic Authentication to no longer work in OOD 4.1.1.

Thanks,

1 Like
5

Thanks for finding that. I was looking at workarounds and alternatives yesterday. I still like ttyd but the maintainer doesn’t seem to be interested in adding alternative auth options. I’m already making a slight modification when building ttyd to add the basic auth credential via an environment variable instead of the -c flag.

I was looking at this pull request that adds URL parameter authentication - add URL parameter authentication support by tantara · Pull Request #1464 · tsl0922/ttyd · GitHub

Also considering if using the --auth-header option would work well here and be secure but it only currently seems to check for the existence of the header and not the value. So possibly with some more updates to ttyd an additional parameter could be passed via the script.sh.erb with the owner’s userid.


--auth-header "X-Forwarded-User"
--auth-header-value "${USER}"
1 Like
6

I made some changes in my fork here but haven’t fully tested them yet. GitHub - hansen-m/ttyd: Share your terminal over the web

I can also share any parts of my app or apptainer build script if those would be helpful.

1 Like
7

Did you all update OnDemand on the same machine? I’ve got a dev instance running with trace7 turned on, but it’s unclear where/how these values are being parsed out and then passed downstream.

So I’m wondering if this came from OnDemand updates or apache updates. Googling around, putting the username & password in the URL seems to be deprecated and indeed browsers themselves may not even be passing this information.

In any case, I’m still trying to track down where and how this is being parsed in apache and passed to the origin.

8

:man_facepalming: I think I figured out why this isn’t working. Asking the correct question in google provides the answer: The browser strips this from the URL and sends it in an Authorization header.

For additional security 4.1 started to strip these headers when making requests downstream to origin servers.

strip_proxy_headers in ood_portal.yml is now stripping the Authorization header to the origins to actually prevent this exact situation. It’s in the default configuration, so updating this configuration to allow Authorization to pass through should fix your issue.

See Apache proxy passing sensitive headers to origins · Advisory · OSC/ondemand · GitHub for more details.

1 Like
9

Turns out I need to write the documentation for strip_proxy_headers and strip_proxy_cookies so I’ll get on that shortly.

In the interim you can see the config and the defaults here:

10

Thanks for digging into this issue. This was already a fragile setup for ttyd auth, so it feels like the better long-term fix is to make the changes to ttyd. I’ve already updated our instances to my new fork and have heard no complaints from users yet.

Although it doesn’t seem to get a ton of updates, I’m just not sure how difficult it will be to maintain longer term and the maintainer appears to be unwilling to incorporate the changes.

11

Harder than you think believe me. Good luck!

12

Sorry for the delayed response. Due to a system power outage, my verification was delayed.

I have now confirmed that ttyd works if I configure strip_proxy_headers to remove only the Authorization header. However, I understand that this is not a recommended configuration from a security perspective.

Therefore, either:

  • Using the fork maintained by hansen-m, or
  • Finding an alternative approach

will be necessary for a proper solution.

Since the original purpose of this ticket (confirming that the default ttyd does not work as-is) has been achieved, I will finish this ticket.

Thank you all for your consideration and support.

13

I was able to get ttyd working without modifying strip_proxy_headers. I will explain the approach based on the following implementation:
https://code.oscer.ou.edu/OnDemand/ood-apps-nmsu/-/tree/main/shell/

Originally, ttyd was started like this:

ttyd -b "/node/${HOST}/${PORT}" -c "ttyd:${PASSWORD}" "$OOD_STAGED_ROOT/tmux.sh"

I changed it to:

ttyd -b "/node/${HOST}/${PORT}/${PASSWORD}" -W -m 1 "$OOD_STAGED_ROOT/tmux.sh"
  • -b: Specifies the expected base path for requests coming from a reverse proxy. By including /${PASSWORD} in the base path, the one-time password becomes part of the URL.
  • -c: Enables BASIC authentication. Since we are not using BASIC authentication in this approach, this option is removed.
  • -W: Enables write access.
  • -m 1: Limits the maximum number of clients to 1.

Instead of using BASIC authentication (-c), access control is handled by embedding a random token directly in the URL.

Next, I define the password in template/before.sh, as follows:

password="$(create_passwd 24)"
PASSWORD="$password"
export password PASSWORD

The default length was 16 characters, which felt short, so I increased it to 24 characters to improve entropy.

Finally, I updated view.html.erb as follows:

<script type="text/javascript">
  function get_<%= form_id %>_action(form) {
    form.action = "https://" + window.location.host.split(":")[0] + "/node/<%= host %>/<%= port %>/<%= password %>"
}
</script>
<form id="<%= form_id %>" method="post" target="_blank" onsubmit="get_<%= form_id %>_action(this)">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-eye"></i> Connect
  </button>
</form>

Clicking the button generates a URL that includes the session-specific password.

This approach does NOT use BASIC authentication. However:

  • A session-specific random token is embedded in the URL.
  • The token is sufficiently long and random.
  • Access is limited to a single client (-m 1).

In practice, this behaves similarly to a token-based authentication mechanism. Note that this is essentially the same as BASIC authentication.

Thanks,

14

Perhaps I’m missing something but doesn’t this just result in an obscured base path but still open URL for access to the session? Other users who discovered (fromps output) or were provided the link would have full access to that user’s shell. Only the client limit of -m 1 would prevent them from connecting.

15

You’re correct. This behaves the same as the original ttyd -c (Basic Auth): it does not truly protect the session.

So it’s only safe on systems with hidepid enabled or on compute nodes that are not shared by multiple users.

16

I came up with a slightly more secure approach.

When I checked the ttyd GitHub repository, I found some PRs that aim to improve security. However, as you mentioned, it doesn’t seem likely to be merged anytime soon.

So I went ahead and forked it myself:

The modification to src/server.c is very minimal - just 16 lines in total. Instead of using the -b option, it introduces an environment variable, TTYD_BASE_PATH.

With this change, it works as follows:

# template/script.sh.erb
export TTYD_BASE_PATH="/node/${HOST}/${PORT}/${PASSWORD}"
ttyd -W -m 1 "$OOD_STAGED_ROOT/tmux.sh"

Thanks,