How to disable Passenger security check update

Greetings,

I am a new Open OnDemand user attempting to make it work for our clusters. I have installed the RPMs, launched the services, and reached step Start Services' (https://osc.github.io/ood-documentation/latest/installation/start-apache.html). I created a local user ood’ and tried to connect. However, I received message

Home directory not found

Your home directory appears to be missing. The home directory mount may be unavailable, or your home directory may need to still be created. Please contact support for help and attempt to restart your web server by clicking below when the problem has been fixed.

The home directory for user `ood’ does exist on a local file system on the Open OnDemand server.

In file /var/log/ondemand-nginx/ood/error.log I found the following:

[ E 2022-05-06 15:51:53.1138 238677/T5 age/Cor/SecurityUpdateChecker.h:507 ]: Security update check failed: SSL connect error while connecting to https://securitycheck.phusionpassenger.com/v1/check.json (if this error persists check your connection security or try upgrading Passenger) (next check in 24 hours)

I suspect that this is the problem. If I should look elsewhere, please tell me.

We are a healthcare institution and are very strict about security (HIPAA violations are subject to criminal penalties). By default, all access by data center servers to external sites is blocked. Firewall exceptions can be requested but are reviewed by our information security department, and anything that provides a whiff of data exfiltration is examined closely and must be justified. I think my best approach is to disable this check.

I found the URL embedded in binary file
/opt/ood/ondemand/root/usr/lib64/passenger/support-binaries/PassengerAgent

Documentation at
Configuration reference - for Passenger Standalone - Passenger Library
says that the Passenger security check update can be disabled at a command line, within a configuration file, or via an environment variable.

What is the simplest and most effective method to disable the Passenger security check update, and what is the procedure to do that?

Regards,
Eric Sisson

This is not the issue and to disable it in 2.0 you’d have to edit this file that we distribute. You can see other passenger settings there, simply add the config to disable it.

/opt/ood/nginx_stage/templates/pun.conf.erb

If you do update this template remove all the files here: (I’m not 100% sure about that glob, but it’s basically any conf file you find here).

/var/lib/ondemand-nginx/config/apps/*/**.conf

Back to your issue - it really means what it says. It can’t find the home directory. Did you create it after you hit that page? ‘restarting the webserver’ can restart the state and retry. Some systems won’t create a home directory until the user logs in. Then of course if you made a local user, usermod need to specify home creation.

In any case - If the user does indeed have a valid home directory, then you should restart things until starts to work. Start with the link the page gives you then try the entire machine maybe.

Greetings, Jeff,

Thank you for the quick reply, particularly given that you must have sent it late in your day.

After more experimenting this morning, I finally have traced this to being an SELinux issue of some kind; switching SELInux from Enforcing to Permissive mode made the difference. I had installed the ondemand-selinux RPM, so I guess that something about the way our servers are built must differ from the assumptions in that package. I will try to get more specific information and post it to this thread.

Regards,
Eric

Thank you for the update, here’s the TE file for reference.

The SELinux thing I should have caught before. There could be an issue here that we’ve yet to identify. This user had the same issue with HOME.

Greetings, Jeff,

Here is a follow-up.

Like the other problem that you posted, we are running RHEL 8 with SELinux. We will be using the institution-controlled Active Directory, but right now, I am trying to make the basics work.

I ran this two times, once with SELinux in Permissive mode and a second time with SELinux in Enforcing mode. I performed a reboot between the two to ensure as much as possible that I was starting from the same conditions. From each attempt, I captured lines from /var/log/audit.log. In each case, I saw a series of 14 lines beginning with type-USER_ACCT' and ending with type=CRED_DISP’. Ignoring differences in timestamps and PID numbers, those lines looked mostly the same except for two lines — type=USER_START' and type=USER_END’ just before the ending CRED_DISP line:


Permissive mode

type=USER_START msg=audit(1652112985.483:130): pid=2419 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”

type=USER_END msg=audit(1652112987.049:131): pid=2419 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”


Enforcing mode

type=USER_START msg=audit(1652114316.986:127): pid=2461 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”

type=USER_END msg=audit(1652114318.477:128): pid=2461 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”


The difference is in section msg=', subsection grantors=’:

  • Permissive mode includes `pam_systemd’
  • Enforcing mode this specification is absent
    The man page for pam_systemd says ``Register user sessions in the systemd login manager’’. I confess that I tend to get lost in PAM, so I hope this means something to you.

Regards,
Eric

No idea. @tdockendorf any ideas?

Those audit logs don’t look like denials. There are likely “dontaudit” denials taking place which don’t get logged. It’s a very annoying “feature” of SELinux.

If you’re still using local $HOME then might need to enable the ondemand_manage_user_home_dir SELinux boolean: 3. Modify System Security — Open OnDemand 2.0.20 documentation

If that boolean doesn’t help, you may have to turn off the “dontaudit” in SELinux, but only leave it off during the shortest timespan possible and then perform the operation that’s blocked then re-enable “dontaudit” as it can floor the logs and auditd might rotate logs too quickly.

Disable “dontaudit”

semodule -DB

Turn back on “dontaudit”

semodule -B

After you perform the operation with dontaudit disabled run this:

cat /var/log/audit/audit.log | audit2allow

And past the output here. There might already be denials you can pull out so might be good to run the above command now just to check.

Greetings, Trey and Jeff,

That was it. The output from audit2allow included the following:

#============= ood_pun_t ==============

#!!!! This avc can be allowed using the boolean 'ondemand_manage_user_home_dir'
allow ood_pun_t user_home_dir_t:dir getattr;

and that setting is described at the page linked by Trey. So I entered

setsebool -PV ondemand_manage_user_home_dir on

and tried again. The first attempt still returned the “Home directory not found” message, but hitting “Restart Web Server” connected.

Of course, in a final deployment, all the user home directories will be mounted by NFS, so this was something of detour into the weeds.

Here is another issue I found earlier that may be helpful to others. Like the person Jeff referenced who was having similar problems a few months ago, we are connecting to an institutional Active Directory for authentication, and we are doing this through sssd. I had to add user apache to file /etc/security/access.conf to allow it to invoke its sudo rule to launch nginx_stage. That may be a consequence of how our data center people told us to configure authentication rather than something inherent in sssd or PAM, but it did allow me to get past the following error:

sudo[234158]:  apache : PAM account management error: Permission denied ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u ood -a ...

Thank you all, again, and I look forward to exercising Open OnDemand further.

Regards,
Eric

Try running this command and paste output:

cat /var/log/audit/audit.log | audit2allow

There maybe something with SELinux blocking. If there is nothing in SELinux then might be something specific to your PAM stack for sudo so show the contents of /etc/pam.d/sudo

Greetings, Trey,

I have removed apache from access.conf and rebooted the server. I can connect to the web login page, enter the userid and password. After that I received a page with the following text:

Error -- sudo: PAM account management error: Permission denied

File /var/log/secure contains the following:

... sudo[2689]:  apache : PAM account management error: Permission denied ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u ood -a ...

The output from audit2allow contains the following:

#============= httpd_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t chkpwd_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t initrc_var_run_t:file read;

#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t self:capability net_admin;

#============= system_dbusd_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t self:capability net_admin;

#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t setroubleshootd_t:process { noatsecure rlimitinh siginh };

#!!!! This avc has a dontaudit rule in the current policy
allow system_dbusd_t unconfined_service_t:process { noatsecure rlimitinh siginh };

File /etc/pam.d/sudo is very simple:

#%PAM-1.0
auth       include      system-auth
account    include      system-auth
password   include      system-auth
session    include      system-auth

File system-auth is a symlink to “/etc/authselect/system-auth”, which is provided by RPM authselect-libs and has the following contents:

# Generated by authselect on Tue Apr 12 07:55:13 2022
# Do not modify this file manually.

auth        required                                     pam_env.so
auth        required                                     pam_faildelay.so delay=2000000
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        [default=1 ignore=ignore success=ok]         pam_localuser.so
auth        sufficient                                   pam_unix.so nullok
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        sufficient                                   pam_sss.so forward_pass
auth        required                                     pam_deny.so

account     required                                     pam_access.so
account     required                                     pam_unix.so
account     sufficient                                   pam_localuser.so
account     sufficient                                   pam_usertype.so issystem
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required                                     pam_permit.so

password    requisite                                    pam_pwquality.so local_users_only
password    sufficient                                   pam_unix.so sha512 shadow nullok use_authtok
password    sufficient                                   pam_sss.so use_authtok
password    required                                     pam_deny.so

session     optional                                     pam_keyinit.so revoke
session     required                                     pam_limits.so
-session    optional                                     pam_systemd.so
session     optional                                     pam_oddjob_mkhomedir.so
session     [success=1 default=ignore]                   pam_succeed_if.so service in crond quiet use_uid
session     required                                     pam_unix.so
session     optional                                     pam_sss.so

Regards,
Eric

So it looks like we have some of those httpd_t items allowed with the ondemand_use_kubernetes boolean so try this:

setsebool -PV ondemand_use_kubernetes on

If that works, then we just need to move some of the items to be not gated behind the Kubernetes boolean. If that doesn’t work, try this:

mkdir /tmp/selinux
cd /tmp/selinux
cat /var/log/audit/audit.log | audit2allow -M ood
# Edit ood.te to remove system_dbusd_t allow lines
make -f /usr/share/selinux/devel/Makefile
semodule -i ood.pp

If that Makefile path doesn’t exist, you’ll need to install selinux-policy-devel.

Note the comment in those commands, I don’t think the system_dbusd_t items are related to OnDemand so first let’s just try allowing the httpd_t contexts.

If those httpd_t items work for you I can update the OnDemand SELinux policies to handle those when ondemand-selinux package is installed. If those work also please include contents of ood.pp that you applied

Greetings, Trey,

Setting the ondemand_use_kubernetes boolean to on made no difference. I still receive the page with message

Error -- sudo: PAM account management error: Permission denied

I reset the audit.log file before running this test, but the output from the audit2allow step for that file was

Nothing to do

I grabbed the previous audit.log file, and that did generate an ood.te file. I modified that to remove the allow system_dbusd_t … lines, installed the RPM, ran make, and ran the semodule step, which took several seconds to return. However, the result is the same; I still receive the message above.

Here is the final ood.te file

module ood 1.0;

require {
	type unconfined_service_t;
	type setroubleshootd_t;
	type system_dbusd_t;
	type httpd_t;
	type fsdaemon_t;
	class capability net_admin;
	class process { noatsecure rlimitinh siginh };
}

#============= fsdaemon_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow fsdaemon_t self:capability net_admin;

#============= httpd_t ==============

#!!!! This avc has a dontaudit rule in the current policy
allow httpd_t self:capability net_admin;

Regards,
Eric

That leads me to believe the issue is with either sudo configuration or PAM.

Do you have the file /etc/sudoers.d/ood that allows the Apache user to execute /opt/ood/nginx_stage/sbin/nginx_stage. This is what ours looks like, can ignore Kubernetes parts if not using:

# cat /etc/sudoers.d/ood 
# This file is managed by Puppet; changes may be overwritten
Defaults:apache !requiretty, !authenticate
Defaults:apache env_keep += "NGINX_STAGE_* OOD_*"
apache ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage
Cmnd_Alias KUBECTL = /usr/local/bin/kubectl
Defaults!KUBECTL !syslog

So one thing we do different with PAM is we don’t put pam_access into the main system-auth and password-auth files because those files are used by things we don’t want to use pam_access with, like sudo. We only put pam_access.so into /etc/pam.d/sshd since that’s the only access we care about. If you do need to put pam_access in the system-auth you might try adding this to /etc/security/access.conf:

+ : apache : ALL

I’m not sure if sudo is an option in access.conf, I only ever fill in cron/crond and hostnames for SSH access.

Greetings, Trey,

In fact, what I had done was to add line

+ : apache : ALL

to /etc/security/access.conf. That was the extra step I mentioned yesterday, earlier in this thread, that I had to perform to make things work.

Just now, I tried specifying sudo in place of ALL, but that did not work.

I am reluctant to modify PAM configurations because that is something that we do very rarely, if at all, whereas modifying access.conf is a basic configuration action (typically by specifying Active Directory groups), so it is a typical place to look for changes.

I guess we have reached an end stage here. Thank you for your time looking into this.

Regards,
Eric

Greetings, Trey,

I should have mentioned that our /etc/sudoers.d/ood looks almost identical to yours except that it adds a couple of paths for KUBECTL.

Regards,
Eric

Based on the man page for access.conf I don’t think sudo is recognized by pam_access. So you’re likely going to have to stick with ALL. You might be able to say LOCAL but not sure if that applies to the way sudo is using pam_access.

Greetings, Trey,

Good shot! Changing to LOCAL worked. Earlier I had tried specifying 127.0.0.1 but had no luck with that.

Thanks for the persistence and the follow-up.

Regards,
Eric