Httpd24 fails after installing ondemand 1.6.22 (CentOS 7.6)

Hello, I recently installed ondemand 1.6.20 on our development cluster. Everything worked great there and I was able to create the edits for the ansible deployment in prod. However, after deploying in prod the httpd24-httpd service will not start after the ondemand install. The error messages are also not helpful as they reference a PID error from the httpd-scp-wrapper but nothing else. I took that to mean there was a permission problem but even changing the permissions or PID location doesn’t help. Any insight on what else to look at would be helpful.
Environment notes:
Currently running “bright cluster manager”
selinux is disabled
firewalld is disabled
iptables is disabled
no other applications are using the ports

Steps to reproduce:

  1. yum install centos-release-scl
  2. yum install https://yum.osc.edu/ondemand/1.6/ondemand-release-web-1.6-4.noarch.rpm
  3. yum install httpd24
  4. systemctl start httpd24-httpd
  5. Check to verify its working
    systemctl status httpd24-httpd
    ● httpd24-httpd.service - The Apache HTTP Server
    Loaded: loaded (/usr/lib/systemd/system/httpd24-httpd.service; disabled; vendor preset: disabled)
    Active: active (running) since Thu 2020-04-16 07:13:53 PDT; 2s ago
    Main PID: 58056 (httpd)
    Status: “Processing requests…”
    Tasks: 47
    CGroup: /system.slice/httpd24-httpd.service
    ├─58056 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
    ├─58059 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
    ├─58060 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
    ├─58061 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
    ├─58062 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
    └─58063 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND

Apr 16 07:13:53 pplhpc1ln1 systemd[1]: Starting The Apache HTTP Server…
Apr 16 07:13:53 pplhpc1ln1 systemd[1]: Started The Apache HTTP Server.

  1. yum install ondemand
  2. systemctl restart httpd24-httpd (it fails)
    ● httpd24-httpd.service - The Apache HTTP Server
    Loaded: loaded (/usr/lib/systemd/system/httpd24-httpd.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/httpd24-httpd.service.d
    └─ood.conf
    Active: failed (Result: exit-code) since Thu 2020-04-16 07:51:15 PDT; 8s ago
    Process: 63795 ExecStop=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
    Process: 63793 ExecStart=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
    Main PID: 63793 (code=exited, status=1/FAILURE)
    Status: “Reading configuration…”

Apr 16 07:51:15 pplhpc1ln1 systemd[1]: Starting The Apache HTTP Server…
Apr 16 07:51:15 pplhpc1ln1 systemd[1]: httpd24-httpd.service: main process exited, code=exited, status=1/FAILURE
Apr 16 07:51:15 pplhpc1ln1 httpd-scl-wrapper[63795]: httpd (no pid file) not running
Apr 16 07:51:15 pplhpc1ln1 systemd[1]: Failed to start The Apache HTTP Server.
Apr 16 07:51:15 pplhpc1ln1 systemd[1]: Unit httpd24-httpd.service entered failed state.
Apr 16 07:51:15 pplhpc1ln1 systemd[1]: httpd24-httpd.service failed.

  1. Apply configs that work in dev. Same problem.

Appreciate any assistance as to where I might be missing something.

Hi and welcome! I’ve found journalctl is a better place for error messages. You could also look in /var/log/httpd24. Hope that helps, link back logs from either places if you like.

Certainly! Sorry I should have done so but wasn’t sure how long people would want the post to be. Please find the ‘journalctl’ below and if needed I can provide the ‘strace’ if I can do attachments. (Don’t want this to be insanely long.)

journalctl -xe
Apr 16 11:35:08 pplhpc1ln1 polkitd[86114]: Registered Authentication Agent for unix-process:98744:794109691 (system bus name :1.167357 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

Apr 16 11:35:08 pplhpc1ln1 systemd[1]: Starting The Apache HTTP Server…

– Subject: Unit httpd24-httpd.service has begun start-up

– Defined-By: systemd

– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit httpd24-httpd.service has begun starting up.

Apr 16 11:35:08 pplhpc1ln1 systemd[1]: httpd24-httpd.service: main process exited, code=exited, status=1/FAILURE

Apr 16 11:35:08 pplhpc1ln1 httpd-scl-wrapper[98754]: httpd (no pid file) not running

Apr 16 11:35:08 pplhpc1ln1 systemd[1]: Failed to start The Apache HTTP Server.

– Subject: Unit httpd24-httpd.service has failed

– Defined-By: systemd

– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit httpd24-httpd.service has failed.

– The result is failed.

Apr 16 11:35:08 pplhpc1ln1 systemd[1]: Unit httpd24-httpd.service entered failed state.

Apr 16 11:35:08 pplhpc1ln1 systemd[1]: httpd24-httpd.service failed.

Apr 16 11:35:08 pplhpc1ln1 polkitd[86114]: Unregistered Authentication Agent for unix-process:98744:794109691 (system bus name :1.167357, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

It’s no issue if the post is long. I can’t tell if there is any issue there, if that polkitd error message is related to this.

As a quick spot check though, I’d check if selinux is enabled, that may be something that’s different between the two environments. Otherwise, if you want to keep selinux on, you can try the ondemand-selinux package for 1.6, but it was in beta for that version and has improvement s in the next.

selinux is not enabled.

Also I can post the log in a sec.

OK, yea again, there may be something relevant in /var/log/httpd24/error_log or /var/log/httpd24/<your server name>_error_log.

To save space i truncated it but here is what is in the ‘/var/log/httpd24/error_log’ the $servername_error_log was blank.

AH00016: Configuration Failed
AH00016: Configuration Failed

AH00016: Configuration Failed
AH00016: Configuration Failed

I’m sorry, there should be more somewhere. A super quick google search of ‘AH00016’ shows ssl certificate errors, but even then there should be a message somewhere about that.

Are you sure that selinux isn’t enabled, that’s the only think I can take way from Unregistered Authentication Agent for unix-process being so close to httpd failing. Maybe you should check /var/log/audit/audit.log to be sure.

Yes, there isn’t any security blocking it as far as the system is concerned.
I tried rebooting and everything but to no avail.
Ended up installing a whole new OS copying all the RPMs over and deploying a whole new image, which seems to have fixed the issue. Just very peculiar as all the same packages are installed.

I’ll keep an eye on it and let everyone know if there is something that becomes obvious later.

Strange, I guess all is well that ends well? Thanks for the heads up, just let us know if you run into more issues.

Sounds great, thanks for taking a look when you did.

Just as another heads up - I’ve run into this vague apache pid error not starting a couple times. It also was more easily fixed with another re-installation as opposed to trying to troubleshoot the little info you get from trying to troubleshoot apache.

1 Like