I am trying to get OnDemand 4.0 running in Ubuntu 24.04 but currently experience an error which I’m having trouble to pinpoint. I used the ood-ansible role for the installation of 4.0.3, which runs successfully.
The setup uses ondemand-dex for LDAP authentication (probably unrelated to the issue). I can access the login page of OnDemand and can successfully login via dex / LDAP (indicated by successful auth messages from dex in the system log). After the login, I receive a “HTTP 403 Forbidden” error from nginx.
The nginx log shows the following error:
[ N 2025-05-05 15:36:55.6629 1390982/T1 age/Wat/WatchdogMain.cpp:1370 ]: Starting Passenger watchdog...
2025/05/05 15:36:55 [alert] 1390979#0: Unable to start the Phusion Passenger(R) watchdog because it encountered the following error during startup: Cannot change the directory '/tmp/passenger.VUI5t5y/apps.s' its UID to 1000109 and GID to 100: Operation not permitted (errno=1) (-1: Unknown error)
2025/05/05 15:36:55 [error] 1390986#0: *1 directory index of "/var/www/ood/apps/sys/dashboard/public/" is forbidden, client: unix:, server: localhost, request: "GET /pun/sys/dashboard/ HTTP/1.1", host: "<hostname>"
So it seems that Passenger can’t set the correct permissions for the dashboard, causing nginx to not being able to access it. But I don’t understand what permissions Passenger tries set exactly and why it fails. UID 1000109 is the correct user ID of the ldap-based user i logged into. I can manually set the UID:GID of the directory to the values from the error message.
I am quite confident that the error is not related to AppArmor as I created a ‘complain’ profile for /opt/ood/nginx_stage/sbin/nginx_stage (equivalent to PERMISSIVE in SELinux) and also see no denied requests in the system log.
As there is no further hint in the logs (neither system logs nor nginx) what permissions Passenger actually tries to set there, I’m at a loss.
Is there a way to run the Passenger watchdog manually to debug what fails?
Or does anyone has another idea how to tackle this issue?
I noticed that the folder /tmp/passenger./apps.s/ is empty. Maybe the error comes from the fact that Passenger tries to set permissions on files which are not there? Should it contain files?
As an off the top guess - I’d guess you have some PAM issue with the GID being so low. There are rules around IDs and IDs lower than 1000 get treated differently. This is what I suspect - it’s not letting you do this because you have a GID of 100.
Hey Jeff
Thanks for getting back to me. I agree that GID 100 is a bit odd. It is the default users group, but not my users main group so I’m wondering where it comes from.
Nevertheless, I can set the permissions to this UID / GID fine as root, but when I run the nginx_stage command as root (which from my understanding OnDemand does via sudo) it results in the same error. And all the folders inside the /tmp/passenger./ directory already have the correct UID/GID set. This leads me to believe that either Passenger is trying to set additional permissions (e.g. some special permissions / ACLs, although there are no restrictions in /tmp/) which fails or that the Passenger commands are from some reason not run with sudo.
I am currently trying to figure out which exact commands Passanger executes so I can debug which part causes the error. But I’m at a loss where to look as I have a system engineer background and little knowledge about web applications.
I’m not so familiar with the PAM stack as maybe I should be. But if I grep -ru uid /etc/pam.d I can see my center has rules around UIDs less than 500.
Searching for gid across all /etc files I found GID_MIN 1000 in /etc/login.defs.
I don’t exactly know the google search you can search for, but I’m 100% sure the GID value of 100 is the issue. I just don’t know the appropriate etc file or subsystem where this is attached to.
You could try to override the nginx_bin binary the nginx_stage.yml with some wrapper that uses strace or similar, though you may have to modify the sudo rules.
Or, as root, you can maybe try to start nginx with strace -o /tmp/debug.out nginx -c /var/lib/ondemand-nginx/config/puns/the_100_gid_user.conf (you have to find the right .conf file).
But even a strace is likely to tell you what you already know - it’s going to try to chown and fail. It may help you find what libraries you’ve loaded and maybe that’ll point to the right /etc file to inspect.
Seems like you got it almost right. The main issue seems to be the high UID (1000109). When I define another user for Passenger in my users pun config (e.g. ‘passenger_default_user ondemand-nginx;’) then the Passenger watchdog starts and successfully sets the folder permissions. But of course this messes up the app permissions leading to a different error (this time from Passenger).
Is there a way to temporarily disable per-user permissions (so that I can use the same local user for every LDAP user while still having the authentication)? This way I could continue / complete the installation and worry about this issue later.
There are hacks you can do to user mapping, but I don’t know if I’d suggest it. Sooner or later you’re going to run into the same issue so I can’t say I’d advise mapping all users to a single user just to move forward.
I don’t know if we can change the behavior of passenger beyond these configurations.
Thanks.
I may made some progress on this issue. I figured out that Passenger doesn’t try to set the permissions (which already belong to the correct user and group) if I enable sandboxing (passenger_user_switching on;). With my user set as user to run the app under (passenger_user <myuser>; ), it seems that nginx can now run the dashboard app.
I had to manually set the GEM_PATH variable as it seems to not have included the /opt/ood/gems/ path on my Ubuntu system. Now Passenger is no longer complaining about permission issues or missing gems. Instead, it fails when reading the config files (and also when trying to process the error):
Error: The application encountered the following error: undefined method `error' for nil:NilClass (NoMethodError)
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:454:in `rescue in block in read_config'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:449:in `block in read_config'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:448:in `each'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:448:in `each_with_object'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:448:in `read_config'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:412:in `config'
/var/www/ood/apps/sys/dashboard/config/configuration_singleton.rb:524:in `block (2 levels) in add_string_configs'
/var/www/ood/apps/sys/dashboard/config/application.rb:56:in `<class:Application>'
/var/www/ood/apps/sys/dashboard/config/application.rb:23:in `<module:Dashboard>'
/var/www/ood/apps/sys/dashboard/config/application.rb:22:in `<top (required)>'
/var/www/ood/apps/sys/dashboard/config/environment.rb:4:in `require_relative'
/var/www/ood/apps/sys/dashboard/config/environment.rb:4:in `<top (required)>'
config.ru:5:in `require_relative'
config.ru:5:in `block in <main>'
/opt/ood/gems/gems/rack-2.2.10/lib/rack/builder.rb:125:in `instance_eval'
/opt/ood/gems/gems/rack-2.2.10/lib/rack/builder.rb:125:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:in `eval'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:96:in `load_app'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:116:in `block in <module:App>'
/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:399:in `run_block_and_record_step_progress'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:115:in `<module:App>'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:28:in `<module:PhusionPassenger>'
/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:27:in `<main>
I’m not sure if this is related to the sandboxing (and I didn’t made much progress) or another issue which is caused by my ubuntu system configuration.
I also find it odd that Passenger loaded the logger gem (according to it’s error page), but does not know about its ‘error’ method.
Do you have any idea what I’m missing here? Let me know if I can provide additional information or if I should move that issue to a seperate thread so it can be found easier at a later time.
Sorry about all that. I think you can patch this line with warn instead of Rails.logger.error. It appears we forgot Rails isn’t available yet while this runs.
Thank you for the quick reply. Changing the ‘Rails.logger.error’ to ‘warn’ did the trick, now the dashboard is loading as intended. I can’t tell you how happy I am to finally see the dashboard
Now I would be glad to reflect the fixes I needed to do in the upstream code, so it doesn’t get overwritten by updates.
I’d suggest to move that to 2 issues in the repo:
UID error: From my (limited) understanding of Passenger, there should be no difference between ‘passenger_user_switching off’ with ‘passenger_default_user <user>’ (the current implementation) and ‘passenger_user_switching on’ with ‘passenger_user <user>’ (which is the fix on my side). Both runs the Passenger app in the context of the provided user.
I don’t see any immediate errors on my Ubuntu system, but don’t have a RHEL system for thorough testing. Do you have a system to verify if this option also works as expected under RHEL? If so, I’m offering creating an issue for this change.
The Rails.logger.error case: I’m a bit surprised that this doesn’t seem to occur on other systems during the start. I’ll create another issue for it, but probably can’t provide a good fix for this.
