403 Forbidden on Ubuntu: Passenger Core SIGABRT / "Directory Index Forbidden" in PUN logs

Get Help
1

Hello OOD Community,

I am experiencing a persistent “403 Forbidden” error on a fresh installation of Open OnDemand 3.1 on Ubuntu 24.04 (Single Node, H100 GPU server).

The Problem: After a successful authentication (using htpasswd), I am redirected to /pun/sys/dashboard/, but I receive a “403 Forbidden” page served by nginx.

The Logs: Checking the Per-User Nginx (PUN) error log at /var/log/ondemand-nginx/yahav/error.log, I see that Passenger is crashing immediately upon startup with a SIGABRT, followed by Nginx failing to serve the directory:

2026/03/11 20:03:05 [alert] 98990#0: Unable to start the Phusion Passenger(R) watchdog because it encountered the following error during startup: Unable to start the Passenger core: it seems to have been killed with signal SIGABRT during startup (-1: Unknown error)
2026/03/11 20:03:54 [error] 99020#0: *2 directory index of “/var/www/ood/apps/sys/dashboard/public/” is forbidden, client: unix:, server: localhost, request: “GET /pun/sys/dashboard/ HTTP/1.1”, host: “”

What I have tried so far:

  • File Limits: Increased nofile limits to 65535 via limits.conf and systemd overrides for both apache2 and ondemandservices. Verified ulimit -n for the user is 65535.

  • Permissions: Set home directory permissions to 751 and added www-data to the user’s group.

  • Passenger Path: Linked /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini to /etc/ood/locations.ini.

  • Reinstallation: Completely purged ondemand, apache2, and /var/lib/ondemand-nginx before reinstalling.

  • Manual PUN Start: Attempting to start the PUN manually with nginx_stage pun -u yahav results in the same crash log.

System Details:

  • OS: Ubuntu 24.04 LTS (Kernel 6.8.0)

  • OOD Version: 3.1

  • Auth: Basic Auth (htpasswd)

It appears that Passenger is aborting before it can initialize the Ruby application. Has anyone seen this specific SIGABRTon newer Ubuntu kernels, or is there a specific dependency I might be missing for the Passenger watchdog?

Any guidance would be much appreciated!

2

Hmmmm that’s an odd one. I wonder if you have AppArmor on and if there are any relevant logs/denials from it?

3

Also - Hi and welcome!

4

Thanks for the suggestion regarding AppArmor. I’ve checked the logs using journalctl -ke | grep -i "apparmor=\"DENIED\"".

The system is definitely active with AppArmor (I see denials for ubuntu_pro_apt_news and python3), but interestingly, there are no entries specifically mentioning ondemand-nginx or the PUN at the time of the crash.

I even tried putting the Nginx profile into complain mode with aa-complain /usr/sbin/ondemand-nginx, but the SIGABRT persists. It seems the Passenger watchdog is aborting before it even hits an AppArmor restriction, or it’s failing on something else…

5

OK strange. I guess I’d also ask if you have selinux running with apparmor. We provide selinux packages, but no apparmor equivalent.

Then I guess just as a sanity check can the yahav user actually run ls -l /var/www/ood/apps/sys/dashboard/public/ command?

Beyond that, you can add this config in /etc/ood/config/nginx_stage.yml to wrap nginx with strace and get much more detailed information about what system calls it’s trying to make and where exactly it’s trying to fail.

# /etc/ood/config/nginx_stage.yml
nginx_bin: 'strace -o /tmp/nginx.out /opt/ood/ondemand/root/usr/sbin/nginx'
7 
yahav@enics-h100:~$ ls -l /var/www/ood/apps/sys/dashboard/public/
total 56
-rw-r–r-- 1 root root  1564 Mar  3 17:01 404.html
-rw-r–r-- 1 root root  1547 Mar  3 17:01 422.html
-rw-r–r-- 1 root root  1477 Mar  3 17:01 500.html
drwxr-xr-x 7 root root 32768 Mar 11 20:43 assets
-rw-r–r-- 1 root root     0 Mar  3 17:01 favicon.ico
drwxr-xr-x 7 root root  4096 Mar 11 20:43 noVNC-1.3.0
-rw-r–r-- 1 root root   202 Mar  3 17:01 robots.txt

Should the folder contain an index.html file?

8

I saw in this discussion:
https://discourse.openondemand.org/t/403-forbidden-error/3635
that the issue was caused by the open file limit on the system being too low.

I believe I might be experiencing the same problem.

See below:

enics@enics-h100:/etc/systemd/system/apache2.service.d$ pid=$(sudo cat /var/run/ondemand-nginx/testuser/passenger.pid)
enics@enics-h100:/etc/systemd/system/apache2.service.d$ cat /proc/$pid/limits | grep "Max open files"
Max open files            1024                 1048576              files

I’m not sure which commands I should run to resolve this.
Any guidance or assistance would be greatly appreciated.

Thanks.

9

You need to edit /etc/security/limits.conf for the affects to be permanent.