We upgraded several installations from 3.0.x to 3.1.15 on RHEL8.8 to address security issues. We are having a few failures now on the desktop app, file downloads, and shell app.
When I try to run a shell I get the following in the log:
App 3902435 output: Listening on 3000
App 3902435 output: Connection established
App 3902435 output: /var/www/ood/apps/sys/shell/app.js:203
App 3902435 output: token = req.url.match(/csrf=([^&]*)/)[1];
App 3902435 output: ^
App 3902435 output:
App 3902435 output: TypeError: Cannot read properties of null (reading ‘1’)
App 3902435 output: at WebSocketServer.connection (/var/www/ood/apps/sys/shell/app.js:203:40)
App 3902435 output: at WebSocketServer.emit (node:events:517:28)
App 3902435 output: at done (/var/www/ood/apps/sys/shell/app.js:281:9)
App 3902435 output: at WebSocketServer.completeUpgrade (/var/www/ood/apps/sys/shell/node_modules/ws/lib/websocket-server.js:435:5)
App 3902435 output: at WebSocketServer.handleUpgrade (/var/www/ood/apps/sys/shell/node_modules/ws/lib/websocket-server.js:343:10)
App 3902435 output: at Server.upgrade (/var/www/ood/apps/sys/shell/app.js:280:7)
App 3902435 output: at Server.emit (node:events:517:28)
App 3902435 output: at onParserExecuteCommon (node:_http_server:915:14)
App 3902435 output: at onParserExecute (node:_http_server:809:3)
App 3902435 output:
App 3902435 output: Node.js v18.18.2
I have found some similar errors in the forums but the solutions don’t seem to apply to us; our httpd is updated (httpd-2.4.37-56.module+el8.8.0+19808+379766d6.7.x86_64) and earlier versions of ondemand are vulnerable to the security issues we were trying to remedy.
I have not yet tried removing and reinstalling the ondemand packages, wondering if there’s anything else I should look for/try before attempting that.
Before troubleshooting too much, I wanted to ask if there is there a specific reason you’re upgrading to 3.1.15 rather than the latest stable release 4.1.x?
If you’re upgrading to address security vulnerabilities, I’d strongly recommend going to OnDemand 4.1 instead of 3.1.15. Version 4.1 includes all the security fixes from 3.1.x and likely more, and it’s supported on RHEL 8.
For the CSRF error: the Shell app is failing because it’s unable to parse the CSRF token from the websocket URL, which the error is just telling us.
This could be a configuration issue introduced during the upgrade, this could be something custom in apache stripping query parameters somehow, it’s a bit tricky to nail down. Did you rebuild the portal config after you did the upgrade or change anything in the ood_portal.yml at all?
We wanted to address the security issues now and do a major version upgrade later since things were working acceptably. We’ve often had issues with upgrades (as we are now) so we wanted to make minimal changes.
We ran sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal as detailed in the instructions.
Update: it looks like the newest version of httpd available to us on RHEL 8.8 is 2.4.37-56, I am looking into how we can install 2.4.37-65 as it looks like that may be the cause of our issues.
Enabling upgrades to a newer version of rhel 8 and upgrading httpd to httpd-2.4.37-65 resolved our issue with the shell and file downloads.
There is still an issue with the desktop that seems to stem from a change in the way the account field is passed from the startup form, still investigating that one.