Shell Access not working

Get Help
1

Dear developers,

After updating to OOD 3.1.9, Shell Access stopped working.

The error message in /var/log/ondemand-nginx/ is as follows:

App 3453282 output: Connection established
App 3453282 output: /var/www/ood/apps/sys/shell/app.js:161
App 3453282 output:   token = req.url.match(/csrf=([^&]*)/)[1];

The value of “req.url” is the string “/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp”. The expected string would be the above string followed by “?csrf=XXXXX”.

Do you know how to solve this problem ?

Best,

2

We just updated ours to 3.1.9 and are running into the same issue.

3

I’m not able to replicate, but I am aware of other reports for the same. I’ve given more description in the bug ticket below. I’d ask what do you see in the network tab of your browser similar to what I saw (and posted on github).

Additionally, while looking into that issue, it seems to build the URL for the websocket based off of the URL of the webpage. Can you share the URL of the webpage you’re trying to access?

Lastly given the changelog, I’d pin this on updates we had to make to our lua code to support the latest version of httpd (or apache2 as the case may be). I’d ask what apache2/httpd version you’re on.

4

@jeff.ohrstrom Thank you for your reply.

I think our issue is the same issue as described in the GitHub link.

screenshot
screenshot1992×608 109 KB

I use httpd-2.4.37-56 on RedHat 8.8. The URL when I launch the Shell Access is https://ondemand.fugaku.r-ccs.riken.jp/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp.

Another information is that the above environment is a production environment, and we also have a test environment, where Shell Access works, and the configuration files are almost the same in both environments, but the software environment is a bit different. On the test environment httpd-2.4.37-65 on Rocky Linux 8.10 are used. The URL is https://ondemand-test.fugaku.r-ccs.riken.jp/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp.

Thanks,

5

Unfortunately I can’t replicate on Rocky 8.8. Even 8.8 has httpd-2.4.37-65. I’ll try to track down a system that has httpd-2.4.37-56, but I’m guessing the fixes we put in place for 2.4.62 somehow broke `httpd-2.4.37-56. I’ll also have to look into the difference between those 56 and 65 releases.

6

I can replicate on patch version 56. But I had to get the version from Rocky’s vault, meaning it’s not even available anymore.

It seems like you’ll need to update httpd or downgrade to OOD 3.1.7. There will be a 3.1.10, so I’m looking into alternatives now.

7

Thank you very much for your help.

I downgraded to OOD 3.1.7 (# dnf downgrade ondemand). But I have a different issue. When I run Shell Access, it says Permissin Denied. The following error appears in /var/log/ondemand-nginx/[user]/error.log:

App 1831557 output: Listening on 3000
App 1831557 output: Connection established
App 1831557 output: Opened terminal: 1831604
App 1831557 output: /var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:263
App 1831557 output:         pty.resize(this._fd, cols, rows);
App 1831557 output:             ^
App 1831557 output:
App 1831557 output: Error: ioctl(2) failed, EBADF
App 1831557 output:     at UnixTerminal.resize (/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:263:13)
App 1831557 output:     at WebSocket.<anonymous> (/var/www/ood/apps/sys/shell/app.js:187:28)
App 1831557 output:     at WebSocket.emit (node:events:513:28)
App 1831557 output:     at Receiver.receiverOnMessage (/var/www/ood/apps/sys/shell/node_modules/ws/lib/websocket.js:1209:20)
App 1831557 output:     at Receiver.emit (node:events:513:28)
App 1831557 output:     at /var/www/ood/apps/sys/shell/node_modules/ws/lib/receiver.js:608:16
App 1831557 output:     at node:internal/process/task_queues:140:7
App 1831557 output:     at AsyncResource.runInAsyncScope (node:async_hooks:204:9)
App 1831557 output:     at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
App 1831557 output:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
App 1831557 output:
App 1831557 output: Node.js v18.14.2

Is the way I downgrade correct?

Best,

8

Should be the way to downgrade yes. I’ve never seen that before. A quick google search shows GitHub · Where software is built which the developer says it’s harmless?

When you say it says Permissin Denied. You’re getting the apache page for this error message? Maybe when you downgrade you also need to restart all the PUNs so they boot up with the correct code?

9

I’m sorry I don’t have enough words. The “Permissin denied” message is displayed from Shell Access.

b
b882×156 14.6 KB

I executed following commands and clicked “Restart Web Server” button in “Help” of navigation bar.

# systemctl try-restart httpd
# /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -f

However, the Shell Access message remains unchanged.

Of cause, I have confirmed that I can log in to the node without a password after logging in normally via SSH from OOD server.

Best,

10

Odd that it says it tried hostbased authentication and failed. Even more odd that you can ssh manually.

I wonder if it’s an selinux denial? Is there anything in /var/log/audit/ for the same?

11

Sorry. The issue of Shell Access in OOD 3.1.7 is solved.

I had been changing various settings, and forgot to restore the PATH setting for ssh on OOD (OOD_SSH_WRAPPER in apps/shell/env).

Best,