# Maintenance mode shown when you Restart Web Server from Help tab RHEL 8.3

**URL:** <https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473>\
**Category:** Get Help\
**Tags:** question\
**Created:** [May 6, 2021, 9:08pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473 "2021-05-06T21:08:37Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 6, 2021, 9:08pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/1 "2021-05-06T21:08:37Z")

</div>

I have a new test instance of Open OnDemand running version 1.8.20-1.el8 on RedHat 8.3. Everytime I use Help =\> Restart Web Server, the instance behaves as if it is set in maintenance mode. There is no /etc/ood/maintenance.enable file but it redirects as if there is; showing the Maintenance Mode html page. If I turn off the maintenance mode capability in the ood\_portal.yml file it then shows a 503 error. I cannot get it back out of maintenance mode without rebooting the server. I have tried restarting httpd but it keeps showing the Maintenance page.  
If I start a new browser window or use a different browser altogether, I am allowed to login (using DEX and ldap with DUO). It still shows the maintenance page.

How is this happening? How do I avoid it or at least get out of this perceived maintenance mode?

---

<div class="post-metadata">

**Author:** ![dugan](https://avatars.discourse-cdn.com/v4/letter/d/eada6e/32.png) [@dugan](https://discourse.openondemand.org/u/dugan)\
**Post date:** [May 7, 2021, 6:34pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/2 "2021-05-07T18:34:55Z")

</div>

I would look in the apache logs to see which request is resulting in the 503 error.  
Also check that the PUN actually restarted. I think apache will return a 503 error if it  
tries to proxy a request to the PUN and it isn’t available.

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 10, 2021, 6:38pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/3 "2021-05-10T18:38:02Z")

</div>

This is the sum total of the error logs from the instance.

[root@ondemand1 log]# tail -2 httpd/error.log  
[Mon May 10 12:17:38.524199 2021] [proxy:error] [pid 25998:tid 140111228401408] (111)Connection refused: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/ondemand-nginx/jasw8470/passenger.sock (\*) failed  
[Mon May 10 12:17:38.524255 2021] [proxy\_http:error] [pid 25998:tid 140111228401408] [client 172.21.35.69:39340] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: [http://ondemand1.rc.int.colorado.edu/pun/sys/dashboard](http://ondemand1.rc.int.colorado.edu/pun/sys/dashboard)

[root@ondemand1 log]# tail ondemand-nginx/error.log  
2021/05/10 12:17:38 [notice] 44723#0: signal process started

[root@ondemand1 log]# tail ondemand-nginx/jasw8470/error.log  
[N 2021-05-10 12:17:38.3858 44460/T8 Ser/Server.h:558]: [ServerThr.1] Shutdown finished  
[N 2021-05-10 12:17:38.3858 44460/Ta Ser/Server.h:902]: [ServerThr.2] Freed 0 spare client objects  
[N 2021-05-10 12:17:38.3858 44460/Ta Ser/Server.h:558]: [ServerThr.2] Shutdown finished  
[N 2021-05-10 12:17:38.3858 44460/Tc Ser/Server.h:902]: [ServerThr.3] Freed 0 spare client objects  
[N 2021-05-10 12:17:38.3858 44460/Tc Ser/Server.h:558]: [ServerThr.3] Shutdown finished  
[N 2021-05-10 12:17:38.3859 44460/Te Ser/Server.h:902]: [ServerThr.4] Freed 0 spare client objects  
[N 2021-05-10 12:17:38.3859 44460/Te Ser/Server.h:558]: [ServerThr.4] Shutdown finished  
[N 2021-05-10 12:17:38.3859 44460/Tg Ser/Server.h:902]: [ApiServer] Freed 0 spare client objects  
[N 2021-05-10 12:17:38.3859 44460/Tg Ser/Server.h:558]: [ApiServer] Shutdown finished  
[N 2021-05-10 12:17:40.0998 44460/T1 age/Cor/CoreMain.cpp:1325]: Passenger core shutdown finished

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 10, 2021, 7:06pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/4 "2021-05-10T19:06:16Z")

</div>

Hey sorry for the delay. I wonder if the maintenance page is just being shown by default for any 503. Can you turn that feature off and see what the behaviour is after that?

```auto
# /etc/ood/config/ood_portal.yml
use_maintenance: false

```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 10, 2021, 7:11pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/5 "2021-05-10T19:11:39Z")

</div>

Also here’s a similar topic where I describe the triaging steps.

I’d wonder off hand if this is a new VM, what your ulimits are like. I’ve shown in that topic what ours are.

> [@Nginx Error: Address already in use](https://discourse.openondemand.org/t/nginx-error-address-already-in-use/1238/7):
>
> here’s my best guess as to what’s happening: The first nginx starts and crashes, but doesn’t release the socket file or zombies. (1) I would try to find out what the process is still using it. lsof will give you this. See if that process is still running or what’s going on with that. [jeff@518bd97864ee /]$ lsof /var/run/ondemand-nginx/jeff/passenger.sock COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nginx 379 jeff 6u unix 0x00000000ff78144c 0t0 2523864 /var/…

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 10, 2021, 7:13pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/6 "2021-05-10T19:13:36Z")

</div>

> [@jasw8470](#):
>
> (111)Connection refused: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/ondemand-nginx/jasw8470/passenger.sock (\*) failed

OK last message in this burst I promise. This looks like it’s an selinux issue. If you have selinux enabled you can install the `ondemand-selinux` RPM to create a policy that’s compliant with all the stuff we need to do.

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 10, 2021, 8:35pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/7 "2021-05-10T20:35:55Z")

</div>

Already have ondemand-selinux installed and created policy to allow ondemand to function.

ood.te

module ood 1.0;

require {  
type var\_lib\_t;  
type sssd\_conf\_t;  
type unreserved\_port\_t;  
type selinux\_config\_t;  
type ood\_apps\_t;  
type rhsmcertd\_t;  
type ood\_pun\_t;  
type node\_t;  
type httpd\_t;  
type automount\_tmp\_t;  
type sshd\_key\_t;  
type var\_run\_t;  
type gpg\_exec\_t;  
type ssh\_keysign\_exec\_t;  
class tcp\_socket { name\_connect node\_bind };  
class file { append create execute execute\_no\_trans getattr map open read write };  
class sock\_file { create getattr setattr write };  
class netlink\_route\_socket { bind create getattr nlmsg\_read };  
class dir { add\_name create read setattr write };  
}

#============= httpd\_t ==============

#!!! This avc is allowed in the current policy  
allow httpd\_t sssd\_conf\_t:file read;  
allow httpd\_t sssd\_conf\_t:file open;  
allow httpd\_t var\_run\_t:sock\_file { getattr write };

#============= ood\_pun\_t ==============  
allow ood\_pun\_t automount\_tmp\_t:dir read;  
allow ood\_pun\_t ood\_apps\_t:file append;

#!!! This avc can be allowed using the boolean ‘ondemand\_use\_torque’  
allow ood\_pun\_t self:netlink\_route\_socket { bind create getattr nlmsg\_read };  
allow ood\_pun\_t selinux\_config\_t:dir read;

#!!! This avc can be allowed using the boolean ‘domain\_can\_mmap\_files’  
allow ood\_pun\_t ssh\_keysign\_exec\_t:file map;  
allow ood\_pun\_t ssh\_keysign\_exec\_t:file { execute execute\_no\_trans open read };  
allow ood\_pun\_t sshd\_key\_t:file { open read };

#!!! This avc can be allowed using the boolean ‘ondemand\_use\_torque’  
allow ood\_pun\_t unreserved\_port\_t:tcp\_socket name\_connect;  
allow ood\_pun\_t var\_run\_t:dir { add\_name create setattr write };  
allow ood\_pun\_t var\_run\_t:file { create open read write };  
allow ood\_pun\_t var\_run\_t:sock\_file { create setattr };

#============= rhsmcertd\_t ==============

#!!! This avc is allowed in the current policy  
allow rhsmcertd\_t gpg\_exec\_t:file execute;  
allow rhsmcertd\_t gpg\_exec\_t:file { open read };

#!!! This avc is allowed in the current policy  
allow rhsmcertd\_t node\_t:tcp\_socket node\_bind;  
allow rhsmcertd\_t var\_lib\_t:file { getattr read write };

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 10, 2021, 8:36pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/8 "2021-05-10T20:36:52Z")

</div>

If I turn off use\_maintenance I get a generic 503 error screen.

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 10, 2021, 8:47pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/9 "2021-05-10T20:47:22Z")

</div>

This is a brand new RHEL 8.3 VM

Using the troubleshooting steps from the article.

lsof before “Restarting the Web Server”

[root@ondemand1 ~]# lsof /var/run/ondemand-nginx/jasw8470/passenger.sock  
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME  
nginx 2682 root 8u unix 0xffff9b308c707980 0t0 37053 /var/run/ondemand-nginx/jasw8470/passenger.sock type=STREAM  
nginx 2683 jasw8470 8u unix 0xffff9b308c707980 0t0 37053 /var/run/ondemand-nginx/jasw8470/passenger.sock type=STREAM

After (when the error is occuring) no output  
[root@ondemand1 ~]# lsof /var/run/ondemand-nginx/jasw8470/passenger.sock  
[root@ondemand1 ~]#

ulimits match closely to the limits shown in your article with  
[root@ondemand1 ~]# ulimit -a  
core file size (blocks, -c) unlimited  
data seg size (kbytes, -d) unlimited  
scheduling priority (-e) 0  
file size (blocks, -f) unlimited  
pending signals (-i) 62900  
max locked memory (kbytes, -l) 64  
max memory size (kbytes, -m) unlimited  
open files (-n) 1024  
pipe size (512 bytes, -p) 8  
POSIX message queues (bytes, -q) 819200  
real-time priority (-r) 0  
stack size (kbytes, -s) 8192  
cpu time (seconds, -t) unlimited  
max user processes (-u) 62900  
virtual memory (kbytes, -v) unlimited  
file locks (-x) unlimited

Log from /var/log/ondemand-nginx/jasw8470/error.log seems to still point at selinux

2021/05/10 14:39:38 [alert] 2682#0: unlink() “/var/run/ondemand-nginx/jasw8470/passenger.pid” failed (13: Permission denied)  
2021/05/10 14:39:38 [emerg] 2682#0: unlink() /var/run/ondemand-nginx/jasw8470/passenger.sock failed (13: Permission denied)

[N 2021-05-10 14:39:38.5161 2670/T8 age/Cor/CoreMain.cpp:671]: Signal received. Gracefully shutting down… (send signal 2 more time(s) to force shutdown)  
[N 2021-05-10 14:39:38.5161 2670/T1 age/Cor/CoreMain.cpp:1246]: Received command to shutdown gracefully. Waiting until all clients have disconnected…  
[N 2021-05-10 14:39:38.5162 2670/T8 Ser/Server.h:902]: [ServerThr.1] Freed 0 spare client objects  
[N 2021-05-10 14:39:38.5162 2670/T8 Ser/Server.h:558]: [ServerThr.1] Shutdown finished  
[N 2021-05-10 14:39:38.5162 2670/Tc Ser/Server.h:902]: [ServerThr.3] Freed 0 spare client objects  
[N 2021-05-10 14:39:38.5162 2670/Ta Ser/Server.h:902]: [ServerThr.2] Freed 0 spare client objects  
[N 2021-05-10 14:39:38.5162 2670/Tc Ser/Server.h:558]: [ServerThr.3] Shutdown finished  
[N 2021-05-10 14:39:38.5162 2670/Ta Ser/Server.h:558]: [ServerThr.2] Shutdown finished  
[N 2021-05-10 14:39:38.5162 2670/Te Ser/Server.h:902]: [ServerThr.4] Freed 0 spare client objects  
[N 2021-05-10 14:39:38.5163 2670/Te Ser/Server.h:558]: [ServerThr.4] Shutdown finished  
[N 2021-05-10 14:39:38.5164 2670/Tg Ser/Server.h:902]: [ApiServer] Freed 0 spare client objects  
[N 2021-05-10 14:39:38.5164 2670/Tg Ser/Server.h:558]: [ApiServer] Shutdown finished  
[N 2021-05-10 14:39:39.2200 2670/T1 age/Cor/CoreMain.cpp:1325]: Passenger core shutdown finished

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 10, 2021, 9:08pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/10 "2021-05-10T21:08:21Z")

</div>

Regenerated selinux policy and have been able to restart the web server cleanly. If you get stuck in between you can simply remove the pid and sock files which will allow the server to restart.

New te file below:

module passenger\_sock2 1.0;

require {  
type sshd\_key\_t;  
type httpd\_t;  
type selinux\_config\_t;  
type sssd\_conf\_t;  
type var\_run\_t;  
type automount\_tmp\_t;  
type gpg\_exec\_t;  
type ood\_apps\_public\_t;  
type ood\_pun\_t;  
type node\_t;  
type var\_lib\_t;  
type ssh\_keysign\_exec\_t;  
type ood\_apps\_t;  
type unreserved\_port\_t;  
type rhsmcertd\_t;  
class tcp\_socket { name\_connect node\_bind };  
class file { append create execute execute\_no\_trans getattr map open read unlink write };  
class sock\_file { create getattr setattr unlink write };  
class netlink\_route\_socket { bind create getattr nlmsg\_read };  
class dir { add\_name create read remove\_name setattr write };  
}

#============= httpd\_t ==============

#!!! This avc is allowed in the current policy  
allow httpd\_t ood\_apps\_public\_t:file map;

#!!! This avc is allowed in the current policy  
allow httpd\_t sssd\_conf\_t:file { open read };

#!!! This avc is allowed in the current policy  
allow httpd\_t var\_run\_t:sock\_file { getattr write };

#============= ood\_pun\_t ==============

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t automount\_tmp\_t:dir read;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t ood\_apps\_t:file append;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t self:netlink\_route\_socket { bind create getattr nlmsg\_read };

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t selinux\_config\_t:dir read;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t ssh\_keysign\_exec\_t:file { execute execute\_no\_trans map open read };

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t sshd\_key\_t:file { open read };

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t unreserved\_port\_t:tcp\_socket name\_connect;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t var\_run\_t:dir { add\_name create remove\_name setattr write };  
allow ood\_pun\_t var\_run\_t:file unlink;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t var\_run\_t:file { create open read write };  
allow ood\_pun\_t var\_run\_t:sock\_file unlink;

#!!! This avc is allowed in the current policy  
allow ood\_pun\_t var\_run\_t:sock\_file { create getattr setattr write };

#============= rhsmcertd\_t ==============

#!!! This avc is allowed in the current policy  
allow rhsmcertd\_t gpg\_exec\_t:file { execute open read };

#!!! This avc is allowed in the current policy  
allow rhsmcertd\_t node\_t:tcp\_socket node\_bind;

#!!! This avc is allowed in the current policy  
allow rhsmcertd\_t var\_lib\_t:file { getattr open read write };

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 11, 2021, 1:37pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/11 "2021-05-11T13:37:42Z")

</div>

Do the audit logs have any denials about this in them? Quickly checking the logic of the code we’re in `sudo /opt/ood/nginx_stage/sbin/nginx_stage` trying to send a stop signal to the nginx that’s running. Those logs `[alert] 2682#0: unlink()` are from nginx itself. So we’re executing `nginx stop` (with some other args and so on) as root itself.

Here is what the permissions for these files look like for me.

```auto
[root@b524ac0c1d39 ~] ls /var/run/ondemand-nginx/jeff/ -lrt
total 4
srw-rw-rw-. 1 root root 0 May 11 13:35 passenger.sock
-rw-r--r--. 1 root root 4 May 11 13:35 passenger.pid

```

---

<div class="post-metadata">

**Author:** ![jasw8470](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@jasw8470](https://discourse.openondemand.org/u/jasw8470)\
**Post date:** [May 11, 2021, 3:31pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/12 "2021-05-11T15:31:09Z")

</div>

Audit logs were reporting denials until I created the new SElinx policy that handled those errors. See the above .te file for the policies. The ones that were added are the sock\_file entries

File permissions are the same

[root@ondemand1 ~]# ls -lrt /var/run/ondemand-nginx/jasw8470/  
total 4  
srw-rw-rw-. 1 root root 0 May 11 09:28 passenger.sock  
-rw-r–r–. 1 root root 6 May 11 09:28 passenger.pid

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 11, 2021, 3:39pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/13 "2021-05-11T15:39:39Z")

</div>

@jasw8470 thanks for figuring this out a bit!

Looks like this is the addition we need, to be able to unlink?

> [@jasw8470](#):
>
> class sock\_file { create getattr setattr unlink write };

I don’t quite know how to translate that into an selinux policy

I’m guessing it’s somewhere around here? @tdockendorf please advise.

> <https://github.com/OSC/ondemand/blob/d3a6461ac26a980d37b88431bfd30b1590f8ae57/packaging/ondemand-selinux.te#L261>

---

<div class="post-metadata">

**Author:** ![tdockendorf](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/tdockendorf/32/202_2.png) [@tdockendorf](https://discourse.openondemand.org/u/tdockendorf)\
**Post date:** [May 11, 2021, 7:03pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/14 "2021-05-11T19:03:39Z")

</div>

Something like `class sock_file { create getattr setattr unlink write };` just loads those things so they can be used by actual policy allows, it’s sort of like a Python import or Ruby require.

Based on the comments from generated policy it looks like only these items maybe not covered by existing policies but hard to say

> allow ood\_pun\_t var\_run\_t:file unlink;  
> allow ood\_pun\_t var\_run\_t:sock\_file unlink;

---

<div class="post-metadata">

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\
**Post date:** [May 26, 2022, 3:42pm UTC](https://discourse.openondemand.org/t/maintenance-mode-shown-when-you-restart-web-server-from-help-tab-rhel-8-3/1473/15 "2022-05-26T15:42:56Z")

</div>