I am a new Open OnDemand user attempting to make it work for our clusters. I have installed the RPMs, launched the services, and reached step Start Services' (https://osc.github.io/ood-documentation/latest/installation/start-apache.html). I created a local user ood’ and tried to connect. However, I received message
Home directory not found
Your home directory appears to be missing. The home directory mount may be unavailable, or your home directory may need to still be created. Please contact support for help and attempt to restart your web server by clicking below when the problem has been fixed.
The home directory for user `ood’ does exist on a local file system on the Open OnDemand server.
In file /var/log/ondemand-nginx/ood/error.log I found the following:
[ E 2022-05-06 15:51:53.1138 238677/T5 age/Cor/SecurityUpdateChecker.h:507 ]: Security update check failed: SSL connect error while connecting to https://securitycheck.phusionpassenger.com/v1/check.json (if this error persists check your connection security or try upgrading Passenger) (next check in 24 hours)
I suspect that this is the problem. If I should look elsewhere, please tell me.
We are a healthcare institution and are very strict about security (HIPAA violations are subject to criminal penalties). By default, all access by data center servers to external sites is blocked. Firewall exceptions can be requested but are reviewed by our information security department, and anything that provides a whiff of data exfiltration is examined closely and must be justified. I think my best approach is to disable this check.
I found the URL embedded in binary file
/opt/ood/ondemand/root/usr/lib64/passenger/support-binaries/PassengerAgent
This is not the issue and to disable it in 2.0 you’d have to edit this file that we distribute. You can see other passenger settings there, simply add the config to disable it.
/opt/ood/nginx_stage/templates/pun.conf.erb
If you do update this template remove all the files here: (I’m not 100% sure about that glob, but it’s basically any conf file you find here).
/var/lib/ondemand-nginx/config/apps/*/**.conf
Back to your issue - it really means what it says. It can’t find the home directory. Did you create it after you hit that page? ‘restarting the webserver’ can restart the state and retry. Some systems won’t create a home directory until the user logs in. Then of course if you made a local user, usermod need to specify home creation.
In any case - If the user does indeed have a valid home directory, then you should restart things until starts to work. Start with the link the page gives you then try the entire machine maybe.
Thank you for the quick reply, particularly given that you must have sent it late in your day.
After more experimenting this morning, I finally have traced this to being an SELinux issue of some kind; switching SELInux from Enforcing to Permissive mode made the difference. I had installed the ondemand-selinux RPM, so I guess that something about the way our servers are built must differ from the assumptions in that package. I will try to get more specific information and post it to this thread.
Like the other problem that you posted, we are running RHEL 8 with SELinux. We will be using the institution-controlled Active Directory, but right now, I am trying to make the basics work.
I ran this two times, once with SELinux in Permissive mode and a second time with SELinux in Enforcing mode. I performed a reboot between the two to ensure as much as possible that I was starting from the same conditions. From each attempt, I captured lines from /var/log/audit.log. In each case, I saw a series of 14 lines beginning with type-USER_ACCT' and ending with type=CRED_DISP’. Ignoring differences in timestamps and PID numbers, those lines looked mostly the same except for two lines — type=USER_START' and type=USER_END’ just before the ending CRED_DISP line:
The difference is in section msg=', subsection grantors=’:
Permissive mode includes `pam_systemd’
Enforcing mode this specification is absent
The man page for pam_systemd says ``Register user sessions in the systemd login manager’’. I confess that I tend to get lost in PAM, so I hope this means something to you.
Those audit logs don’t look like denials. There are likely “dontaudit” denials taking place which don’t get logged. It’s a very annoying “feature” of SELinux.
If that boolean doesn’t help, you may have to turn off the “dontaudit” in SELinux, but only leave it off during the shortest timespan possible and then perform the operation that’s blocked then re-enable “dontaudit” as it can floor the logs and auditd might rotate logs too quickly.
Disable “dontaudit”
semodule -DB
Turn back on “dontaudit”
semodule -B
After you perform the operation with dontaudit disabled run this:
cat /var/log/audit/audit.log | audit2allow
And past the output here. There might already be denials you can pull out so might be good to run the above command now just to check.
That was it. The output from audit2allow included the following:
#============= ood_pun_t ==============
#!!!! This avc can be allowed using the boolean 'ondemand_manage_user_home_dir'
allow ood_pun_t user_home_dir_t:dir getattr;
and that setting is described at the page linked by Trey. So I entered
setsebool -PV ondemand_manage_user_home_dir on
and tried again. The first attempt still returned the “Home directory not found” message, but hitting “Restart Web Server” connected.
Of course, in a final deployment, all the user home directories will be mounted by NFS, so this was something of detour into the weeds.
Here is another issue I found earlier that may be helpful to others. Like the person Jeff referenced who was having similar problems a few months ago, we are connecting to an institutional Active Directory for authentication, and we are doing this through sssd. I had to add user apache to file /etc/security/access.conf to allow it to invoke its sudo rule to launch nginx_stage. That may be a consequence of how our data center people told us to configure authentication rather than something inherent in sssd or PAM, but it did allow me to get past the following error:
There maybe something with SELinux blocking. If there is nothing in SELinux then might be something specific to your PAM stack for sudo so show the contents of /etc/pam.d/sudo
I have removed apache from access.conf and rebooted the server. I can connect to the web login page, enter the userid and password. After that I received a page with the following text:
The output from audit2allow contains the following:
#============= httpd_t ==============
#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t chkpwd_t:process { noatsecure rlimitinh siginh };
#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t initrc_var_run_t:file read;
#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t self:capability net_admin;
#============= system_dbusd_t ==============
#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t self:capability net_admin;
#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t setroubleshootd_t:process { noatsecure rlimitinh siginh };
#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t unconfined_service_t:process { noatsecure rlimitinh siginh };
File /etc/pam.d/sudo is very simple:
#%PAM-1.0
auth include system-auth
account include system-auth
password include system-auth
session include system-auth
File system-auth is a symlink to “/etc/authselect/system-auth”, which is provided by RPM authselect-libs and has the following contents:
So it looks like we have some of those httpd_t items allowed with the ondemand_use_kubernetes boolean so try this:
setsebool -PV ondemand_use_kubernetes on
If that works, then we just need to move some of the items to be not gated behind the Kubernetes boolean. If that doesn’t work, try this:
mkdir /tmp/selinux
cd /tmp/selinux
cat /var/log/audit/audit.log | audit2allow -M ood
# Edit ood.te to remove system_dbusd_t allow lines
make -f /usr/share/selinux/devel/Makefile
semodule -i ood.pp
If that Makefile path doesn’t exist, you’ll need to install selinux-policy-devel.
Note the comment in those commands, I don’t think the system_dbusd_t items are related to OnDemand so first let’s just try allowing the httpd_t contexts.
If those httpd_t items work for you I can update the OnDemand SELinux policies to handle those when ondemand-selinux package is installed. If those work also please include contents of ood.pp that you applied
I reset the audit.log file before running this test, but the output from the audit2allow step for that file was
Nothing to do
I grabbed the previous audit.log file, and that did generate an ood.te file. I modified that to remove the allow system_dbusd_t … lines, installed the RPM, ran make, and ran the semodule step, which took several seconds to return. However, the result is the same; I still receive the message above.
Here is the final ood.te file
module ood 1.0;
require {
type unconfined_service_t;
type setroubleshootd_t;
type system_dbusd_t;
type httpd_t;
type fsdaemon_t;
class capability net_admin;
class process { noatsecure rlimitinh siginh };
}
#============= fsdaemon_t ==============
#!!!! This avc has a dontaudit rule in the current policy
allow fsdaemon_t self:capability net_admin;
#============= httpd_t ==============
#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t self:capability net_admin;
That leads me to believe the issue is with either sudo configuration or PAM.
Do you have the file /etc/sudoers.d/ood that allows the Apache user to execute /opt/ood/nginx_stage/sbin/nginx_stage. This is what ours looks like, can ignore Kubernetes parts if not using:
# cat /etc/sudoers.d/ood
# This file is managed by Puppet; changes may be overwritten
Defaults:apache !requiretty, !authenticate
Defaults:apache env_keep += "NGINX_STAGE_* OOD_*"
apache ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage
Cmnd_Alias KUBECTL = /usr/local/bin/kubectl
Defaults!KUBECTL !syslog
So one thing we do different with PAM is we don’t put pam_access into the main system-auth and password-auth files because those files are used by things we don’t want to use pam_access with, like sudo. We only put pam_access.so into /etc/pam.d/sshd since that’s the only access we care about. If you do need to put pam_access in the system-auth you might try adding this to /etc/security/access.conf:
+ : apache : ALL
I’m not sure if sudo is an option in access.conf, I only ever fill in cron/crond and hostnames for SSH access.
to /etc/security/access.conf. That was the extra step I mentioned yesterday, earlier in this thread, that I had to perform to make things work.
Just now, I tried specifying sudo in place of ALL, but that did not work.
I am reluctant to modify PAM configurations because that is something that we do very rarely, if at all, whereas modifying access.conf is a basic configuration action (typically by specifying Active Directory groups), so it is a typical place to look for changes.
I guess we have reached an end stage here. Thank you for your time looking into this.
Based on the man page for access.conf I don’t think sudo is recognized by pam_access. So you’re likely going to have to stick with ALL. You might be able to say LOCAL but not sure if that applies to the way sudo is using pam_access.