403 Forbidden on Ubuntu: Passenger Core SIGABRT / "Directory Index Forbidden" in PUN logs

Get Help
1

Hello OOD Community,

I am experiencing a persistent “403 Forbidden” error on a fresh installation of Open OnDemand 3.1 on Ubuntu 24.04 (Single Node, H100 GPU server).

The Problem: After a successful authentication (using htpasswd), I am redirected to /pun/sys/dashboard/, but I receive a “403 Forbidden” page served by nginx.

The Logs: Checking the Per-User Nginx (PUN) error log at /var/log/ondemand-nginx/yahav/error.log, I see that Passenger is crashing immediately upon startup with a SIGABRT, followed by Nginx failing to serve the directory:

2026/03/11 20:03:05 [alert] 98990#0: Unable to start the Phusion Passenger(R) watchdog because it encountered the following error during startup: Unable to start the Passenger core: it seems to have been killed with signal SIGABRT during startup (-1: Unknown error)
2026/03/11 20:03:54 [error] 99020#0: *2 directory index of “/var/www/ood/apps/sys/dashboard/public/” is forbidden, client: unix:, server: localhost, request: “GET /pun/sys/dashboard/ HTTP/1.1”, host: “”

What I have tried so far:

  • File Limits: Increased nofile limits to 65535 via limits.conf and systemd overrides for both apache2 and ondemandservices. Verified ulimit -n for the user is 65535.

  • Permissions: Set home directory permissions to 751 and added www-data to the user’s group.

  • Passenger Path: Linked /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini to /etc/ood/locations.ini.

  • Reinstallation: Completely purged ondemand, apache2, and /var/lib/ondemand-nginx before reinstalling.

  • Manual PUN Start: Attempting to start the PUN manually with nginx_stage pun -u yahav results in the same crash log.

System Details:

  • OS: Ubuntu 24.04 LTS (Kernel 6.8.0)

  • OOD Version: 3.1

  • Auth: Basic Auth (htpasswd)

It appears that Passenger is aborting before it can initialize the Ruby application. Has anyone seen this specific SIGABRTon newer Ubuntu kernels, or is there a specific dependency I might be missing for the Passenger watchdog?

Any guidance would be much appreciated!

2

Hmmmm that’s an odd one. I wonder if you have AppArmor on and if there are any relevant logs/denials from it?

3

Also - Hi and welcome!

4

Thanks for the suggestion regarding AppArmor. I’ve checked the logs using journalctl -ke | grep -i "apparmor=\"DENIED\"".

The system is definitely active with AppArmor (I see denials for ubuntu_pro_apt_news and python3), but interestingly, there are no entries specifically mentioning ondemand-nginx or the PUN at the time of the crash.

I even tried putting the Nginx profile into complain mode with aa-complain /usr/sbin/ondemand-nginx, but the SIGABRT persists. It seems the Passenger watchdog is aborting before it even hits an AppArmor restriction, or it’s failing on something else…

5

OK strange. I guess I’d also ask if you have selinux running with apparmor. We provide selinux packages, but no apparmor equivalent.

Then I guess just as a sanity check can the yahav user actually run ls -l /var/www/ood/apps/sys/dashboard/public/ command?

Beyond that, you can add this config in /etc/ood/config/nginx_stage.yml to wrap nginx with strace and get much more detailed information about what system calls it’s trying to make and where exactly it’s trying to fail.

# /etc/ood/config/nginx_stage.yml
nginx_bin: 'strace -o /tmp/nginx.out /opt/ood/ondemand/root/usr/sbin/nginx'
7 
yahav@enics-h100:~$ ls -l /var/www/ood/apps/sys/dashboard/public/
total 56
-rw-r–r-- 1 root root  1564 Mar  3 17:01 404.html
-rw-r–r-- 1 root root  1547 Mar  3 17:01 422.html
-rw-r–r-- 1 root root  1477 Mar  3 17:01 500.html
drwxr-xr-x 7 root root 32768 Mar 11 20:43 assets
-rw-r–r-- 1 root root     0 Mar  3 17:01 favicon.ico
drwxr-xr-x 7 root root  4096 Mar 11 20:43 noVNC-1.3.0
-rw-r–r-- 1 root root   202 Mar  3 17:01 robots.txt

Should the folder contain an index.html file?

8

I saw in this discussion:
https://discourse.openondemand.org/t/403-forbidden-error/3635
that the issue was caused by the open file limit on the system being too low.

I believe I might be experiencing the same problem.

See below:

enics@enics-h100:/etc/systemd/system/apache2.service.d$ pid=$(sudo cat /var/run/ondemand-nginx/testuser/passenger.pid)
enics@enics-h100:/etc/systemd/system/apache2.service.d$ cat /proc/$pid/limits | grep "Max open files"
Max open files            1024                 1048576              files

I’m not sure which commands I should run to resolve this.
Any guidance or assistance would be greatly appreciated.

Thanks.

9

You need to edit /etc/security/limits.conf for the affects to be permanent.

10

I am using Ubuntu 24.04 and OOD 4.2.2, and I am experiencing the same issue. I modified /etc/security/limits.conf as following,

* soft nofile 65536
* hard nofile 65536
# End of file

After rebooting, I checked the following values, but they were unchanged from before the change.

$ pid=$(sudo cat /var/run/ondemand-nginx/<USER>/passenger.pid)
$ cat /proc/$pid/limits | grep "Max open files"
Max open files     1024     1048576     files

As another test, I also edited the systemd service limit directly by creating/editing:

# /etc/systemd/system/apache2.service.d/override.conf
[Service]
LimitNOFILE=65536

However, the result was the same.

Could there be another place where Passenger/PUN gets its open file limit on Ubuntu 24.04?

Thanks,

11

I finally found a workaround for this issue.

The fix was to add the following line to /etc/sudoers.d/ood:

Defaults:www-data !pam_session

My final /etc/sudoers.d/ood looks like this (I only added the second line):

Defaults:www-data !requiretty, !authenticate
Defaults:www-data !pam_session
Defaults:www-data env_keep += "NGINX_STAGE_* OOD_*"
www-data ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage

After making this change, OOD started correctly on Ubuntu 24.04.

Interestingly, my Rocky Linux 9.8 system does not exhibit this problem, even though the effective soft limit is also 1024.

I should also note that, in an earlier post, I mentioned modifying /etc/security/limits.conf and /etc/systemd/system/apache2.service.d/override.conf. However, I have since reverted those changes, and OOD continues to work correctly with only the Defaults:www-data !pam_session change in /etc/sudoers.d/ood.

At this point, I am no longer convinced that the file descriptor limit itself is the root cause. Instead, it seems that the issue is related to the PAM session created by sudo, although I do not yet understand the exact mechanism.

Thanks.