Hey there
I am trying to get OnDemand 4.0 running in Ubuntu 24.04 but currently experience an error which I’m having trouble to pinpoint. I used the ood-ansible role for the installation of 4.0.3, which runs successfully.
The setup uses ondemand-dex for LDAP authentication (probably unrelated to the issue). I can access the login page of OnDemand and can successfully login via dex / LDAP (indicated by successful auth messages from dex in the system log). After the login, I receive a “HTTP 403 Forbidden” error from nginx.
The nginx log shows the following error:
[ N 2025-05-05 15:36:55.6629 1390982/T1 age/Wat/WatchdogMain.cpp:1370 ]: Starting Passenger watchdog...
2025/05/05 15:36:55 [alert] 1390979#0: Unable to start the Phusion Passenger(R) watchdog because it encountered the following error during startup: Cannot change the directory '/tmp/passenger.VUI5t5y/apps.s' its UID to 1000109 and GID to 100: Operation not permitted (errno=1) (-1: Unknown error)
2025/05/05 15:36:55 [error] 1390986#0: *1 directory index of "/var/www/ood/apps/sys/dashboard/public/" is forbidden, client: unix:, server: localhost, request: "GET /pun/sys/dashboard/ HTTP/1.1", host: "<hostname>"
So it seems that Passenger can’t set the correct permissions for the dashboard, causing nginx to not being able to access it. But I don’t understand what permissions Passenger tries set exactly and why it fails. UID 1000109 is the correct user ID of the ldap-based user i logged into. I can manually set the UID:GID of the directory to the values from the error message.
I am quite confident that the error is not related to AppArmor as I created a ‘complain’ profile for /opt/ood/nginx_stage/sbin/nginx_stage (equivalent to PERMISSIVE in SELinux) and also see no denied requests in the system log.
As there is no further hint in the logs (neither system logs nor nginx) what permissions Passenger actually tries to set there, I’m at a loss.
Is there a way to run the Passenger watchdog manually to debug what fails?
Or does anyone has another idea how to tackle this issue?
Thank you already.
Jens
I noticed that the folder /tmp/passenger./apps.s/ is empty. Maybe the error comes from the fact that Passenger tries to set permissions on files which are not there? Should it contain files?
As an off the top guess - I’d guess you have some PAM issue with the GID being so low. There are rules around IDs and IDs lower than 1000 get treated differently. This is what I suspect - it’s not letting you do this because you have a GID of 100.
Hey Jeff
Thanks for getting back to me. I agree that GID 100 is a bit odd. It is the default users group, but not my users main group so I’m wondering where it comes from.
Nevertheless, I can set the permissions to this UID / GID fine as root, but when I run the nginx_stage command as root (which from my understanding OnDemand does via sudo) it results in the same error. And all the folders inside the /tmp/passenger./ directory already have the correct UID/GID set. This leads me to believe that either Passenger is trying to set additional permissions (e.g. some special permissions / ACLs, although there are no restrictions in /tmp/) which fails or that the Passenger commands are from some reason not run with sudo.
I am currently trying to figure out which exact commands Passanger executes so I can debug which part causes the error. But I’m at a loss where to look as I have a system engineer background and little knowledge about web applications.
I’m not so familiar with the PAM stack as maybe I should be. But if I grep -ru uid /etc/pam.d I can see my center has rules around UIDs less than 500.
Searching for gid across all /etc files I found GID_MIN 1000 in /etc/login.defs.
I don’t exactly know the google search you can search for, but I’m 100% sure the GID value of 100 is the issue. I just don’t know the appropriate etc file or subsystem where this is attached to.
You could try to override the nginx_bin binary the nginx_stage.yml with some wrapper that uses strace or similar, though you may have to modify the sudo rules.
Or, as root, you can maybe try to start nginx with strace -o /tmp/debug.out nginx -c /var/lib/ondemand-nginx/config/puns/the_100_gid_user.conf (you have to find the right .conf file).
But even a strace is likely to tell you what you already know - it’s going to try to chown and fail. It may help you find what libraries you’ve loaded and maybe that’ll point to the right /etc file to inspect.