$Home Directory Doesn't Exist (CAS, SELinux, & University Controlled AD)

JPGlaser · January 20, 2022, 5:55pm

Hey All,

So we are currently setting up an instance of Open OnDemand on our department’s new HPC cluster’s headnode. We have everything set up pretty stock at the moment (install from the Selinux RPM for Redhat 8) via following the docs. The only difference is setting up CAS based on the fixes posted, which is now working just fine.

However, after CAS is confirmed, we are getting the following error (which has come up before on here but never had an answered solution):

The following comes from our logs:

[Thu Jan 20 11:01:53.708583 2022] [lua:info] [pid 672902:tid 140540842596096] [client 67.163.164.11:52591] req_cache_control="" req_hostname="link.phys.wvu.edu" req_status="302" remote_user="jpg00017" res_content_disp="" req_accept_encoding="gzip, deflate" req_user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" req_origin="" req_port="80" log_time="2022-01-20T16:01:53.708473.0Z" req_user_ip="67.163.164.11" req_server_name="link.phys.wvu.edu" req_is_websocket="false" req_handler="" req_referer="http://link.phys.wvu.edu/pun/sys/dashboard" res_content_length="0" req_is_https="false" req_uri="/nginx/stop" res_content_language="" log_hook="ood" req_method="GET" time_proxy="0" req_protocol="HTTP/1.1" res_content_encoding="" req_filename="/var/www/html/nginx" res_content_location="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" time_user_map="0.003" res_location="/pun/sys/dashboard" res_content_type="" req_accept_charset="" req_accept_language="en-us,en;q=0.9" req_content_type="" local_user="jpg00017", referer: http://link.phys.wvu.edu/pun/sys/dashboard
[Thu Jan 20 11:01:53.734294 2022] [proxy:error] [pid 672902:tid 140540834203392] (2)No such file or directory: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/ondemand-nginx/jpg00017/passenger.sock (*) failed
[Thu Jan 20 11:01:53.734311 2022] [proxy_http:error] [pid 672902:tid 140540834203392] [client 67.163.164.11:52591] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://link.phys.wvu.edu/pun/sys/dashboard

So when a user logs in, it should direct them to /home/$USER which is created via our University-run AD connected through realm. Suggestions on why this is coming up?

~ Joe G.

travert · January 21, 2022, 4:25pm

Hello and thanks for posting!

The relevant links that should help you clear the issue up can be found below.

Here’s the relevant part of the docs you will want:
https://osc.github.io/ood-documentation/latest/customization.html?highlight=home#custom-error-page-for-missing-home-directory-on-launch

And there is a separate Discourse that goes into more detail:

It essentially has to do with when that $HOME directory is created and trying to access it before that happens, and how to make sure it does get created for the user if they try to access it before it exists by providing an error page that allows the SSH session to start and create the $HOME for the user.

Hopefully that will help clear the issue but please let me know if you need any more help or guidance

JPGlaser · January 21, 2022, 5:36pm

Hey @travert !

This is actually not the direct issue, however VERY important for me to know as most users are probably going to use this to connect to Link directly. This is a pretty handy feature as I can actually add in some links to tutorials there as well.

What did solve it was:

setsebool ondemand_manage_user_home_dir=on
setsebool ondemand_manage_nfs=off

I’ll have to switch on nfs later when I move the home directory around to our JBOD (hopefully using the $HOME variable).

That said, I am running into an SELinux issue where I cannot write to my home directory via the UI (but I can via the Shell). When SELinux is turned off, it allows me to see the files, but does not grant me write permissions, with the following being displayed if I try to create a folder/file in the user’s home directory (which has user owned rwx).

I believe there is yet another boolean I should probably toggle somewhere. Any ideas where I can find a list of all of them?

~ Joe G.

EDIT:
@jeff.ohrstrom This thread might give better insight into the current issue.

jeff.ohrstrom · January 21, 2022, 7:49pm

Thanks. So you’re home directory is currently local storage? Can you share the audit.log entry where directory creation is being blocked?

JPGlaser · January 21, 2022, 7:56pm

Hey @jeff.ohrstrom ,

It is indeed on local storage for right now. Eventually, it will be transferred over to a JBOD running ZFS mounted over NTFS. Here is what is recorded when something gets created:

App 254617 output: [2022-01-21 14:51:44 -0500 ]  WARN "Can't verify CSRF token authenticity."
App 254617 output: [2022-01-21 14:51:44 -0500 ]  INFO "method=PUT path=/pun/sys/dashboard/files/fs/home/negarver-daniels/test format=json controller=FilesController action=update status=422 error='ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken' duration=0.85 view=0.00"
App 254617 output: [2022-01-21 14:51:44 -0500 ] FATAL ""
App 254617 output: [2022-01-21 14:51:44 -0500 ] FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):"
App 254617 output: [2022-01-21 14:51:44 -0500 ] FATAL ""

~ Joe G.

jeff.ohrstrom · January 21, 2022, 8:05pm

Sorry about that - this isn’t an selinux issue. I was looking for entries in /var/log/audit.log for denials, but the log you’ve provided is the actual issue.

I’m guessing you’re not behind SSL. You need to add an initializer to be able to pass your cookies insecurely. When you setup SSL - you should remove it.

You can see the instructions here

github.com/OSC/ondemand

HTTP 422 Error / InvalidAuthenticityToken

opened 09:43AM - 31 May 21 UTC

closed 02:38PM - 17 Jun 21 UTC

GloktarFR

bug

After a fresh installation of OOD-2.0.9 on CentOS-7.9, I'm unable to use batch c…onnect applications. The navigation through the dashboard (file explorer, shell, etc.) is working well though. But every time I try to submit a job in an interactive session, I'm getting a HTTP 422 Error. In the /var/log/ondemand-nginx/<user>/error.log, I'm seeing this error message: ``` App 19736 output: [2021-05-31 09:51:01 +0200 ] WARN "Can't verify CSRF token authenticity." App 19736 output: [2021-05-31 09:51:01 +0200 ] INFO "method=POST path=/pun/sys/dashboard/batch_connect/sys/bc_desktop_3d/session_contexts format=html controller=BatchConnect::SessionContextsController action=create status=422 error='ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken' duration=0.73 view=0.00" App 19736 output: [2021-05-31 09:51:01 +0200 ] FATAL "" App 19736 output: [2021-05-31 09:51:01 +0200 ] FATAL "ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):" App 19736 output: [2021-05-31 09:51:01 +0200 ] FATAL "" App 19736 output: [2021-05-31 09:51:01 +0200 ] FATAL "actionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:215:in `handle_unverified_request'\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:247:in `handle_unverified_request'\nactionpack (5.2.6) lib/action_controller/metal/request_forgery_protection.rb:242:in `verify_authenticity_token'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:426:in `block in make_lambda'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:199:in `block in halting'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `block in invoke_before'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `each'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:513:in `invoke_before'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:131:in `run_callbacks'\nactionpack (5.2.6) lib/abstract_controller/callbacks.rb:41:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/rescue.rb:22:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in `block in instrument'\nactivesupport (5.2.6) lib/active_support/notifications/instrumenter.rb:23:in `instrument'\nactivesupport (5.2.6) lib/active_support/notifications.rb:168:in `instrument'\nactionpack (5.2.6) lib/action_controller/metal/instrumentation.rb:32:in `process_action'\nactionpack (5.2.6) lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\nactionpack (5.2.6) lib/abstract_controller/base.rb:134:in `process'\nactionview (5.2.6) lib/action_view/rendering.rb:32:in `process'\nactionpack (5.2.6) lib/action_controller/metal.rb:191:in `dispatch'\nactionpack (5.2.6) lib/action_controller/metal.rb:252:in `dispatch'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:34:in `serve'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:52:in `block in serve'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:35:in `each'\nactionpack (5.2.6) lib/action_dispatch/journey/router.rb:35:in `serve'\nactionpack (5.2.6) lib/action_dispatch/routing/route_set.rb:840:in `call'\nrack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'\nrack (2.2.3) lib/rack/etag.rb:27:in `call'\nrack (2.2.3) lib/rack/conditional_get.rb:40:in `call'\nrack (2.2.3) lib/rack/head.rb:12:in `call'\nactionpack (5.2.6) lib/action_dispatch/http/content_security_policy.rb:18:in `call'\nrack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'\nrack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/cookies.rb:670:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\nactivesupport (5.2.6) lib/active_support/callbacks.rb:98:in `run_callbacks'\nactionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\nlograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'\nrailties (5.2.6) lib/rails/rack/logger.rb:26:in `block in call'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `block in tagged'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:28:in `tagged'\nactivesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `tagged'\nrailties (5.2.6) lib/rails/rack/logger.rb:26:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'\nrequest_store (1.5.0) lib/request_store/middleware.rb:19:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/request_id.rb:27:in `call'\nrack (2.2.3) lib/rack/method_override.rb:24:in `call'\nrack (2.2.3) lib/rack/runtime.rb:22:in `call'\nactivesupport (5.2.6) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\nactionpack (5.2.6) lib/action_dispatch/middleware/executor.rb:14:in `call'\nrack (2.2.3) lib/rack/sendfile.rb:110:in `call'\nrailties (5.2.6) lib/rails/engine.rb:524:in `call'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'\n/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'" ``` Any idea what's causing this ? I have this exact same installation with ondemand-1.8.20 and it's working fine.

JPGlaser · January 22, 2022, 1:17am

Hey @jeff.ohrstrom

No worries! I got the SSL all certified just fine via certbot. That said, we are still not able to access the home directory when SELinux is enabled.

[root@link jpg00017]# tail /var/log/ondemand-nginx/jpg00017/error.log
App 14500 output: [2022-01-21 20:12:34 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=14.93 view=11.29"
App 14500 output: [2022-01-21 20:12:36 -0500 ] ERROR "Permission denied @ rb_file_s_lstat - /home/jpg00017/.mozilla"
App 14500 output: [2022-01-21 20:12:36 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=5.92 view=2.29"
App 14500 output: [2022-01-21 20:13:30 -0500 ]  INFO "method=PUT path=/pun/sys/dashboard/files/fs/home/jpg00017/testing2 format=json controller=FilesController action=update status=200 duration=5.14 view=0.24"
App 14500 output: [2022-01-21 20:13:30 -0500 ] ERROR "Permission denied @ rb_file_s_lstat - /home/jpg00017/.mozilla"
App 14500 output: [2022-01-21 20:13:30 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=5.78 view=2.46"
[ N 2022-01-21 20:18:30.1402 12506/T4 age/Cor/CoreMain.cpp:1147 ]: Checking whether to disconnect long-running connections for process 14500, application /var/www/ood/apps/sys/dashboard (production)
App 15902 output: [2022-01-21 20:22:36 -0500 ] ERROR "Permission denied @ rb_file_s_lstat - /home/jpg00017/.mozilla"
App 15902 output: [2022-01-21 20:22:36 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=25.62 view=7.90"
App 15902 output: [2022-01-21 20:22:36 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/ format=html controller=DashboardController action=index status=200 duration=45.39 view=41.70"

Files do get created just fine, but do not show up in the UI:

# ls -lah /home/jpg00017/
total 4.6M
drwx------. 8 jpg00017 domain users 4.0K Jan 21 20:13 .
drwxr-xr-x. 7 root     root           94 Jan 19 01:54 ..
-rw-------. 1 jpg00017 domain users  434 Jan 21 19:14 .bash_history
-rw-------. 1 jpg00017 domain users   18 Jan 13 10:15 .bash_logout
-rw-------. 1 jpg00017 domain users  141 Jan 13 10:15 .bash_profile
-rw-------. 1 jpg00017 domain users  376 Jan 13 10:15 .bashrc
drwxr-xr-x. 6 jpg00017 domain users  162 Jan 19 11:58 .cache
drwx------. 4 jpg00017 domain users   33 Jan 21 18:08 .config
drwx------. 3 jpg00017 domain users   25 Jan 19 09:19 .dbus
-rw-------. 1 jpg00017 domain users   16 Jan 13 10:15 .esd_auth
-rw-r--r--. 1 jpg00017 domain users 170K Jan 21 20:01 Link_Phantom_Hourglass.png
drwx------. 6 jpg00017 domain users   81 Jan 19 09:19 .mozilla
drwxr-xr-x. 3 jpg00017 domain users   18 Jan 21 11:01 ondemand
-rw-r--r--. 1 jpg00017 domain users 4.4M Jan 21 18:02 SophosSetup.sh
drwx------. 2 jpg00017 domain users   25 Jan 21 11:56 .ssh
-rw-r--r--. 1 jpg00017 domain users    0 Jan 21 20:11 test
-rw-r--r--. 1 jpg00017 domain users    0 Jan 21 20:13 testing2
-rw-r--r--. 1 jpg00017 domain users    0 Jan 21 12:29 test.txt
-rw-r--r--. 1 jpg00017 domain users  207 Jan 21 18:02 .wget-hsts
-rw-------. 1 jpg00017 domain users   50 Jan 19 11:46 .Xauthority
-rw-------. 1 jpg00017 domain users  658 Jan 13 10:15 .zshrc

The certificate is verified as working just fine (certbot did its thing and then I made sure to add it to the ood-portal.yml and rebuild before resetting httpd):

Must be something I am just blanking on at this point. ^_^’

~ Joe G.

jeff.ohrstrom · January 24, 2022, 6:35pm

OK cool. So then we really will need to see relevant log entries about the denial from /var/log/audit/audit.log files.

JPGlaser · January 24, 2022, 7:22pm

Hey @jeff.ohrstrom , here is the tail from the audit.log

[root@link warewulf]# tail /var/log/audit/audit.log
type=USER_END msg=audit(1642813123.397:400): pid=10730 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=CRED_DISP msg=audit(1642813123.397:401): pid=10730 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=USER_END msg=audit(1642813124.543:402): pid=10673 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=CRED_DISP msg=audit(1642813124.543:403): pid=10673 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:httpd_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1642813125.035:404): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1642813125.035:405): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1642813125.524:406): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=USER_END msg=audit(1642813125.531:407): pid=11220 uid=0 auid=560378 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'UID="root" AUID="jpg00017"
type=CRED_DISP msg=audit(1642813125.531:408): pid=11220 uid=0 auid=560378 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'UID="root" AUID="jpg00017"
type=DAEMON_END msg=audit(1642813128.487:7725): op=terminate auid=0 pid=11934 subj=system_u:system_r:unconfined_service_t:s0 res=successAUID="root"

[root@link warewulf]# tail /var/log/ondemand-nginx/jpg00017/error.log
[ N 2022-01-22 18:00:08.5823 155401/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
[ N 2022-01-24 14:20:36.2431 445630/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2022-01-24 14:20:36.2668 445633/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2022-01-24 14:20:36.2669 445633/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2022-01-24 14:20:36.3765 445633/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 445633
App 445747 output: [2022-01-24 14:20:38 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/ format=html controller=DashboardController action=index status=200 duration=56.85 view=41.20"
[ N 2022-01-24 14:20:38.8557 445633/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
App 445747 output: [2022-01-24 14:20:48 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=24.38 view=18.49"
App 445747 output: [2022-01-24 14:20:48 -0500 ] ERROR "Permission denied @ rb_file_s_lstat - /home/jpg00017/.mozilla"
App 445747 output: [2022-01-24 14:20:48 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/jpg00017 format=html controller=FilesController action=fs status=200 duration=8.71 view=5.49"

~ Joe G.

jeff.ohrstrom · January 26, 2022, 2:29pm

I don’t see anything relevant in the audit log. Do you by chance have a /var/log/audit.log too?

Just to recap - these are set these both?

setsebool ondemand_manage_user_home_dir=on
setsebool ondemand_manage_nfs=off

@tdockendorf do you have any where this denail should be or things maybe the user can grep for to find the entires?

tdockendorf · January 26, 2022, 3:33pm

The best way to debug SELinux issues is clear out the audit log, perform the action being blocked, then dump audit log into audit2allow. So these steps:

> /var/log/audit/audit.log
# Perform operation that is having issues
cat /var/log/audit/audit.log | audit2allow

This will print a policy generated to allow whatever is being blocked. That output would be what we need along with the log lines from audit.log.

If nothing is produced and you still think the issue is SELinux, it might be hidden from audit logs by the “dontaudit” policies which are incredibly verbose but often contain some harder to debug denials. To disable the “dontaudit” do this:

semodule -DB
> /var/log/audit/audit.log
# Perform operation that is having issues
cat /var/log/audit/audit.log | audit2allow

semodule -B

When logging the “dontaudit” denials, it’s really important to be fast between the step of clearing log file and performing the problematic operation as dontaudit can be incredibly noisy for things that have nothing to do with what you are trying to debug.

JPGlaser · January 26, 2022, 6:49pm

Hey @tdockendorf and @jeff.ohrstrom ,

So yes, to recap the following are set as follows as for right now, I want to access the /home/$user directory which is stored locally:

setsebool ondemand_manage_user_home_dir=on
setsebool ondemand_manage_nfs=off

Running the commands that Trey suggested provided the following output:

[root@link jpg00017]# tail /var/log/audit/audit.log
[root@link jpg00017]# cat /var/log/audit/audit.log | audit2allow
Nothing to do
[root@link jpg00017]# tail /var/log/audit/audit.log
[root@link jpg00017]# semodule -DB
[root@link jpg00017]# cat /var/log/audit/audit.log | audit2allow
Nothing to do
[root@link jpg00017]# cat /var/log/audit/audit.log | audit2allow
Nothing to do

This is the error that comes up when loading the Home Directory via the Browser after going through CAS authentication (Duo) and landing on the server hosting this instance of OnDemand. Users are controlled by a centeralized AD, which automatically makes a user’s home upon first connection to the server and ensures it is owned properly.

The reason I thought this was a SELinux issue is that by disabling it, I was able to see files in the Web GUI, but uploading files returned an error (even though it was eventually successful).

This might be an issue with Group Permissions instead? This is what the average home directory looks like:

[root@link jpg00017]# ls -lah
total 5.4M
drwx------. 8 jpg00017 domain users 4.0K Jan 26 13:39  .
drwxr-xr-x. 7 root     root           94 Jan 19 01:54  ..
-rw-------. 1 jpg00017 domain users   50 Jan 26 13:37  .Xauthority
-rw-------. 1 jpg00017 domain users  442 Jan 26 11:57  .bash_history
-rw-------. 1 jpg00017 domain users   18 Jan 13 10:15  .bash_logout
-rw-------. 1 jpg00017 domain users  141 Jan 13 10:15  .bash_profile
-rw-------. 1 jpg00017 domain users  376 Jan 13 10:15  .bashrc
drwxr-xr-x. 6 jpg00017 domain users  162 Jan 19 11:58  .cache
drwx------. 4 jpg00017 domain users   33 Jan 21 18:08  .config
drwx------. 3 jpg00017 domain users   25 Jan 19 09:19  .dbus
-rw-------. 1 jpg00017 domain users   16 Jan 13 10:15  .esd_auth
drwx------. 6 jpg00017 domain users   81 Jan 19 09:19  .mozilla
drwx------. 2 jpg00017 domain users   25 Jan 21 11:56  .ssh
-rw-r--r--. 1 jpg00017 domain users  207 Jan 21 18:02  .wget-hsts
-rw-------. 1 jpg00017 domain users  658 Jan 13 10:15  .zshrc
-rw-r--r--. 1 jpg00017 domain users 940K Jan 22 16:51  LinkBanner-web.png
-rw-r--r--. 1 jpg00017 domain users 4.4M Jan 21 18:02  SophosSetup.sh
-rw-r--r--. 1 jpg00017 domain users  31K Jan 22 15:20 'favicon_package_v0.16 (1).zip'
drwxr-xr-x. 3 jpg00017 domain users   18 Jan 21 11:01  ondemand
-rw-r--r--. 1 jpg00017 domain users    0 Jan 26 13:39  sudo

~ Joe G.

JPGlaser · January 27, 2022, 6:02pm

Just bumping this because I’ve been banging my head against this brick wall for a bit now. I can’t seem to find any additional errors in any of the logs and full system restarts don’t fix it either. Something odd is going on …

~ Joe G.

bennet · January 27, 2022, 6:14pm

Joseph,

Do you have a way to check that the processes started by OnDemand are actually getting AD credentials? Does that make sense?

JPGlaser · January 27, 2022, 7:49pm

@bennet @jeff.ohrstrom @tdockendorf

I think I see what you are getting at and I am unsure. I think you might be on to something though …

So right now, logins on OnDemand are handled through CAS provided by my university (WVU). ood_portal.yml is set up with:

auth:
   - 'AuthType CAS'
   - 'Require valid-user'
   - 'RequestHeader unset Authorization'

And /etc/httpd/conf.d/auth_cas.conf reading as:

LoadModule auth_cas_module modules/mod_auth_cas.so
CASCookiePath /var/cache/httpd/mod_auth_cas/
CASLoginURL http://sso.wvu.edu/cas/login
CASValidateURL http://sso.wvu.edu/cas/serviceValidate

The cookie path is working as intended, I can see the CAS from my latest log-in sitting in the directory:

[root@link ood]# ls -lah /var/cache/httpd/mod_auth_cas/
total 12K
drwxr-xr-x. 2 apache apache  104 Jan 27 14:23 .
drwx------. 5 apache apache   50 Jan 19 14:21 ..
-rw-------. 1 apache apache 2.7K Jan 27 14:24 80599eb9a6d5d4a2012a8a05499943b1
-rw-------. 1 apache apache   32 Jan 27 14:23 .f071516d3eda1411c3dc4dadc47a72e6
-rw-------. 1 apache apache   17 Jan 27 14:23 .metadata

With the cookie being in the right format with the right info (obviously slight redacted):

<cacheEntry xmlns="http://uconn.edu/cas/mod_auth_cas">
<user>jpg00017</user>
[...]
<path>/pun/sys/</path>
[...]

I will note that for some reason, after CAS authentication, it takes sometimes several minutes for PUN to load up the instance, sometimes even resulting in the following:

Searching this code in the Phusion database produces no related error documented.

But refreshing it a few times eventually gets it to direct to the active PUN system just fine.

I am a bit at a loss for this one because our university has working instances of OnDemand running on RHEL 8 machines with AD and CAS managed exactly as depicted in this thread.

~ Joe G.

jeff.ohrstrom · January 27, 2022, 8:13pm

Could it be you didn’t boot with the domain users group?

Find this process for the relevant user and see what groups it started with (or without).

[root@4cf09e016e20 ~]# ps 471
    PID TTY      STAT   TIME COMMAND
    471 ?        Ssl    0:00 Passenger watchdog
[root@4cf09e016e20 ~]# grep Groups /proc/471/status
Groups:	1000

Passenger probably created some temp files or just referenced that ID in /var/log/ondemand-nginx logs.

In any case - clearly when you flip SELinux on is when the issue arises correct? Running in permissive mode things work OK right?

JPGlaser · January 27, 2022, 8:48pm

@jeff.ohrstrom

So the Phusion error seems to have been a timeout (the switch from the CAS to OnDemand loading the PUN takes some time to complete ~1-2 minutes which is very odd).

Looking into what you suggested @jeff.ohrstrom …

[root@link config]# pidof "Passenger watchdog"
917109
[root@link config]# ps 917109
    PID TTY      STAT   TIME COMMAND
 917109 ?        Ssl    0:00 Passenger watchdog
[root@link config]# grep Groups /proc/917109/status
Groups: 10 30000 1002942 1003282 1004819 1004990 1005407 1005408 1005668 1007095 1078732 1079001 1080058 1080290 1105354 1105587 1123885 1123999 1162598 1162631 1162645 1205619 1205620
[root@link config]# groups jpg00017
jpg00017 : domain users dept-as physics and astronomy its-rc-mamclaughlin librarygroup_proxy paa-cluster-access paa-cluster-admins acc-entitlespark rol-portal-staff o365-employee its-rc-thorny mymediasite rol-ep-staff rol-employee employee dept-arts and sciences as dept-academic affairs aca dialout adm-ais-crrb-519 administrators eng-compclass-inst sophosuser lock mymediasitequiz wheel

The process is executed by the UID and GID that matches jpg00017:domain users, but I can’t seem to find said group domain users in getent group …

And to clarify, turning SELinux off doesn’t fully resolve the issues. It allows me to read via the UI, but doesn’t allow me to write.

Just a note … why does the audit log say its trying to access with double slashes …
/pun/sys/dashboard/files/fs//home/jpg00017

~ Joe G.

jeff.ohrstrom · May 9, 2022, 3:53pm

Hi sorry I’m just picking this back up again. Are you still having issues writing to your HOME directory?

emsisson · May 19, 2022, 5:54pm

Greetings, all,

I see that this issue appears to be open, still, and I am seeing a similar problem.

Environment:

RHEL 8.6
Open OnDemand RPM ondemand-2.0.23-1.el8.x86_64
digital certificate installed (I do not think this is significant)
SELinux in Enforcing mode
server configured to use SSSD to authenticate users via institutional Active Directory
– however, user ood is a local account with a home directory on local storage
– ultimately, our users will authenticate via AD and have NFS-mounted directories

I can connect to Open OnDemand as ood@localhost / password and receive the dashboard.

When I selected menu path Files → Home Directory, I received a window displaying message

    Permission denied @ rb_file_s_lstat - /home/ood/.mozilla
    Error occurred when attempting to access /pub/sys/dashboard/files/fs//home/ood

(see attached screenshot below). Information in /var/log/messages, /var/log/audit/audit.log, and /var/log/ondemand-nginx/ood/error.log did not add much information.

I changed SELinux to Permissive mode, cleared the browser caches, and tried again. This time, I was able to display the directory contents. Also, I found the following in /var/log/messages:

May 19 10:38:26 r1trlhpcood /SetroubleshootPrivileged.py[52787]: failed to retrieve rpm info for /var/lib/selinux/targeted/active/modules/400/ondemand-selinux
May 19 10:38:26 r1trlhpcood setroubleshoot[52775]: SELinux is preventing utils.rb:110 from read access on the directory dir. For complete SELinux messages run: sealert -l 581096a2-c3ac-43dc-9fb3-304d7d3fffc7
May 19 10:38:26 r1trlhpcood setroubleshoot[52775]: SELinux is preventing utils.rb:110 from read access on the directory dir.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that utils.rb:110 should be allowed read access on the dir directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'utils.rb:110' --raw | audit2allow -M my-utilsrb110#012# semodule -X 300 -i my-utilsrb110.pp#012

I ran commands

# ausearch -c 'utils.rb:110' --raw | audit2allow -M my-utilsrb110
# semodule -X 300 -i my-utilsrb110.pp

(see attached file my-utilsrb110.te below) then reset SELinux to Enforcing mode. I cleared the browser caches, reconnected, and received the same ``Permission denied’’ message as before.

Here is the relevant output from /var/log/messages:

May 19 10:50:25 r1trlhpcood ondemand-dex[1765]: time="2022-05-19T15:50:25Z" level=info msg="login successful: connector \"local\", username=\"ood\", preferred_username=\"\", email=\"ood@localhost\", groups=[]"

Here is the relevant output from /var/log/audit/audit.log:

type=USER_AVC msg=audit(1652975375.794:3478): pid=1028 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  received setenforce notice (enforcing=1)  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"
type=SERVICE_START msg=audit(1652975403.590:3479): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1652975403.590:3480): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"

# date -ud @1652975403.590
Thu May 19 15:50:03 UTC 2022

Here is the relevant output from /var/log/ondemand-nginx/ood/error.log:

App 53029 output: [2022-05-19 10:50:28 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/ format=html controller=DashboardController action=index status=200 duration=71.58 view=51.97"
App 53029 output: [2022-05-19 10:50:33 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/ood format=html controller=FilesController action=fs status=200 duration=22.73 view=15.95"
App 53029 output: [2022-05-19 10:50:33 -0500 ] ERROR "Permission denied @ rb_file_s_lstat - /home/ood/.mozilla"
App 53029 output: [2022-05-19 10:50:33 -0500 ]  INFO "method=GET path=/pun/sys/dashboard/files/fs/home/ood format=html controller=FilesController action=fs status=200 duration=15.10 view=6.39"

Clearly, SELinux is affecting something although I do not know how, but I suspect that another factor is involved. What should I try next?

Regards,
Eric

ScreenShot
my-utilsrb110.te.txt (395 Bytes)

jeff.ohrstrom · May 19, 2022, 5:58pm

Did you enable any of the selinux booleans we provide?

https://osc.github.io/ood-documentation/latest/installation/modify-system-security.html

looks like you need to enable ondemand_manage_user_home_dir .

Topic		Replies	Views
User Home Directory Issues Get Help	4	633	January 21, 2022
Missing Home Directory Error Get Help	27	3438	May 26, 2022
Error -- Permission denied @ dir_s_mkdir - /var/run/ondemand-nginx Get Help	4	1521	May 26, 2022
Permission denied rb_file_s_lstat with SELinux and local home directories Get Help question	3	569	April 25, 2023
Fix "no home directory" issue when first time logging in Get Help	2	1177	May 26, 2022

$Home Directory Doesn't Exist (CAS, SELinux, & University Controlled AD)

Related topics