# Shell Access not working

**URL:** <https://discourse.openondemand.org/t/shell-access-not-working/3826>\
**Category:** Get Help\
**Tags:** question\
**Created:** [October 22, 2024, 7:27am UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826 "2024-10-22T07:27:42Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![mnakao](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mnakao/32/1001_2.png) [@mnakao](https://discourse.openondemand.org/u/mnakao)\
**Post date:** [October 22, 2024, 7:27am UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/1 "2024-10-22T07:27:42Z")

</div>

Dear developers,

After updating to OOD 3.1.9, Shell Access stopped working.

The error message in /var/log/ondemand-nginx/ is as follows:

```auto
App 3453282 output: Connection established
App 3453282 output: /var/www/ood/apps/sys/shell/app.js:161
App 3453282 output: token = req.url.match(/csrf=([^&]*)/)[1];

```

The value of “req.url” is the string “/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp”. The expected string would be the above string followed by “?csrf=XXXXX”.

Do you know how to solve this problem ?

Best,

---

<div class="post-metadata">

**Author:** ![brad.traver](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@brad.traver](https://discourse.openondemand.org/u/brad.traver)\
**Post date:** [October 22, 2024, 3:04pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/2 "2024-10-22T15:04:41Z")

</div>

We just updated ours to 3.1.9 and are running into the same issue.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 22, 2024, 8:48pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/3 "2024-10-22T20:48:08Z")

</div>

I’m not able to replicate, but I am aware of other reports for the same. I’ve given more description in the bug ticket below. I’d ask what do you see in the network tab of your browser similar to what I saw (and posted on github).

> <https://github.com/OSC/ondemand/issues/3894>
>
> After updating from 3.1.7 to 3.1.9, the shell app immediately crashes with the m…essage:
> \`\`\`
> Your connection to the remote server has been terminated.
> \`\`\`
> 
> Looking at the logs I found
> \`\`\`
> App 4042244 output: Listening on 3000
> App 4042244 output: Connection established
> App 4042244 output: /var/www/ood/apps/sys/shell/app.js:161
> App 4042244 output: token = req.url.match(/csrf=(\[^&\]\*)/)\[1\];
> App 4042244 output: ^
> App 4042244 output:
> App 4042244 output: TypeError: Cannot read properties of null (reading '1')
> App 4042244 output: at WebSocketServer.connection (/var/www/ood/apps/sys/shell/app.js:161:40)
> App 4042244 output: at WebSocketServer.emit (node:events:517:28)
> App 4042244 output: at done (/var/www/ood/apps/sys/shell/app.js:233:9)
> App 4042244 output: at WebSocketServer.completeUpgrade (/var/www/ood/apps/sys/shell/node\_modules/ws/lib/websocket-server.js:435:5)
> App 4042244 output: at WebSocketServer.handleUpgrade (/var/www/ood/apps/sys/shell/node\_modules/ws/lib/websocket-server.js:343:10)
> App 4042244 output: at Server.upgrade (/var/www/ood/apps/sys/shell/app.js:232:7)
> App 4042244 output: at Server.emit (node:events:517:28)
> App 4042244 output: at onParserExecuteCommon (node:\_http\_server:939:14)
> App 4042244 output: at onParserExecute (node:\_http\_server:825:3) 
> \`\`\`
> 
> I made it print out \`req.url\` and it's the bare URL to the shell app without a CSRF token.
> 
> 3.1.7 does not have this issue since downgrading fixes it.

Additionally, while looking into that issue, it seems to build the URL for the websocket based off of the URL of the webpage. Can you share the URL of the webpage you’re trying to access?

Lastly given the changelog, I’d pin this on updates we had to make to our lua code to support the latest version of httpd (or apache2 as the case may be). I’d ask what apache2/httpd version you’re on.

---

<div class="post-metadata">

**Author:** ![mnakao](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mnakao/32/1001_2.png) [@mnakao](https://discourse.openondemand.org/u/mnakao)\
**Post date:** [October 23, 2024, 12:41am UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/4 "2024-10-23T00:41:39Z")

</div>

@jeff.ohrstrom Thank you for your reply.

I think our issue is the same issue as described in the GitHub link.

 ![screenshot](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/f/f98d3b6aa9b67a760874d22da56a68d48b684a42.png)

I use `httpd-2.4.37-56 on RedHat 8.8`. The URL when I launch the Shell Access is `https://ondemand.fugaku.r-ccs.riken.jp/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp`.

Another information is that the above environment is a production environment, and we also have a test environment, where Shell Access works, and the configuration files are almost the same in both environments, but the software environment is a bit different. On the test environment `httpd-2.4.37-65 on Rocky Linux 8.10` are used. The URL is `https://ondemand-test.fugaku.r-ccs.riken.jp/pun/sys/shell/ssh/login.fugaku.r-ccs.riken.jp`.

Thanks,

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 23, 2024, 1:41pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/5 "2024-10-23T13:41:52Z")

</div>

Unfortunately I can’t replicate on Rocky 8.8. Even 8.8 has `httpd-2.4.37-65`. I’ll try to track down a system that has `httpd-2.4.37-56`, but I’m guessing the fixes we put in place for `2.4.62` somehow broke `httpd-2.4.37-56. I’ll also have to look into the difference between those 56 and 65 releases.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 23, 2024, 3:36pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/6 "2024-10-23T15:36:29Z")

</div>

I can replicate on patch version 56. But I had to get the version from Rocky’s vault, meaning it’s not even available anymore.

It seems like you’ll need to update httpd or downgrade to OOD `3.1.7`. There will be a 3.1.10, so I’m looking into alternatives now.

---

<div class="post-metadata">

**Author:** ![mnakao](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mnakao/32/1001_2.png) [@mnakao](https://discourse.openondemand.org/u/mnakao)\
**Post date:** [October 24, 2024, 12:06pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/7 "2024-10-24T12:06:17Z")

</div>

Thank you very much for your help.

I downgraded to OOD 3.1.7 (# dnf downgrade ondemand). But I have a different issue. When I run Shell Access, it says Permissin Denied. The following error appears in /var/log/ondemand-nginx/[user]/error.log:

```auto
App 1831557 output: Listening on 3000
App 1831557 output: Connection established
App 1831557 output: Opened terminal: 1831604
App 1831557 output: /var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:263
App 1831557 output: pty.resize(this._fd, cols, rows);
App 1831557 output: ^
App 1831557 output:
App 1831557 output: Error: ioctl(2) failed, EBADF
App 1831557 output: at UnixTerminal.resize (/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:263:13)
App 1831557 output: at WebSocket.<anonymous> (/var/www/ood/apps/sys/shell/app.js:187:28)
App 1831557 output: at WebSocket.emit (node:events:513:28)
App 1831557 output: at Receiver.receiverOnMessage (/var/www/ood/apps/sys/shell/node_modules/ws/lib/websocket.js:1209:20)
App 1831557 output: at Receiver.emit (node:events:513:28)
App 1831557 output: at /var/www/ood/apps/sys/shell/node_modules/ws/lib/receiver.js:608:16
App 1831557 output: at node:internal/process/task_queues:140:7
App 1831557 output: at AsyncResource.runInAsyncScope (node:async_hooks:204:9)
App 1831557 output: at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
App 1831557 output: at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
App 1831557 output:
App 1831557 output: Node.js v18.14.2

```

Is the way I downgrade correct?

Best,

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 24, 2024, 1:14pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/8 "2024-10-24T13:14:14Z")

</div>

Should be the way to downgrade yes. I’ve never seen that before. A quick google search shows [GitHub · Where software is built](https://github.com/microsoft/vscode/issues/100457) which the developer says it’s harmless?

When you say it says `Permissin Denied.` You’re getting the `apache` page for this error message? Maybe when you downgrade you also need to restart all the PUNs so they boot up with the correct code?

---

<div class="post-metadata">

**Author:** ![mnakao](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mnakao/32/1001_2.png) [@mnakao](https://discourse.openondemand.org/u/mnakao)\
**Post date:** [October 24, 2024, 1:51pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/9 "2024-10-24T13:51:33Z")

</div>

I’m sorry I don’t have enough words. The “Permissin denied” message is displayed from Shell Access.

 ![b](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/b8d1f8cb461130f56013b8512501c86e52056368.png)

I executed following commands and clicked “Restart Web Server” button in “Help” of navigation bar.

```auto
# systemctl try-restart httpd
# /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -f

```

However, the Shell Access message remains unchanged.

Of cause, I have confirmed that I can log in to the node without a password after logging in normally via SSH from OOD server.

Best,

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 24, 2024, 1:55pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/10 "2024-10-24T13:55:51Z")

</div>

Odd that it says it tried `hostbased` authentication and failed. Even more odd that you can ssh manually.

I wonder if it’s an selinux denial? Is there anything in `/var/log/audit/` for the same?

---

<div class="post-metadata">

**Author:** ![mnakao](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mnakao/32/1001_2.png) [@mnakao](https://discourse.openondemand.org/u/mnakao)\
**Post date:** [October 24, 2024, 2:20pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/11 "2024-10-24T14:20:12Z")

</div>

Sorry. The issue of Shell Access in OOD 3.1.7 is solved.

I had been changing various settings, and forgot to restore the PATH setting for ssh on OOD (OOD\_SSH\_WRAPPER in apps/shell/env).

Best,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [April 22, 2025, 2:20pm UTC](https://discourse.openondemand.org/t/shell-access-not-working/3826/12 "2025-04-22T14:20:28Z")

</div>

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.