Anyone else httpd "client denied" errors in logs?

My httpd error logs are full of noise like
[Tue Jan 22 14:24:27.261723 2019] [access_compat:error] [pid 22974] [client 150.135.159.198:64310] AH01797: client denied by server configuration: /opt/rh/httpd24/root/var/www/html/pun, referer: https://ood.hpc.arizona.edu/pun/sys/dashboard/batch_connect/sessions
(about 30k of these per day!).

In spite of how dire that looks, OnDemand seems to work correctly. Is anyone else experiencing this? I’m not sure if this is a leftover from upgrading from OnDemand pre-rpm or a config I missed or ??

I have not seen this unfortunately. /opt/rh/httpd24/root/var/www/html/ is an empty directory. My guess is that this is the default directory Apache serves static files out of. Is it possible that a directive in your Apache config is set to try to serve static files on each request, like it tries /opt/rh/httpd24/root/var/www/html/ first, then falls back to default functionality? Is this occurring on every request? I assume so since you said 30k per day.

Just guessing. @tdockendorf knows a lot more than me about Apache and might be able to comment.

Also, maybe if you share the Apache config, or non-sensitive parts /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf. Of course feel free to email me directly anything you don’t want on Discourse.

Would need to upload redacted copies of .conf files in /opt/rh/httpd24/root/etc/httpd/conf and /opt/rh/httpd24/root/etc/httpd/conf.d. The default DocumentRoot for stock httpd24-httpd install points to /opt/rh/httpd24/root/var/www/html but the virtual host used by OnDemand should prevent that from having an effect I think, though one is not defined for OnDemand configs.

I’ve emailed the requested bits to Eric Franz directly and asked him to share with you; I don’t think anything in there is secret, but …

Try commenting out DocumentRoot from /opt/rh/httpd24/root/etc/httpd/conf/httpd.conf. Should be able to apply change with systemctl reload httpd24-httpd. I can’t reproduce the logs in vagrant environment that matches stock install. Our production and test environments all use Puppet and Puppet removes things like DocumentRoot from httpd.conf.

I tried mucking with DocumentRoot, but that didn’t do much useful. What did solve my problem with access_compat:error complaining about every /pun reference was
overloading lua_log_level as
lua_log_level: ‘info access_compat:crit’
to suppress the errors from access_compat.

This could also be the result of mixing Apache 2.2 directives like Order and Satisfy with Apache 2.4 directives like Require. The old directives are supplied by the module http://httpd.apache.org/docs/2.4/mod/mod_access_compat.html and since the errors you were seeing in the logs were from access_compat perhaps those would also go away if you avoided using those directives altogether. I only just now stumbled across http://httpd.apache.org/docs/2.4/upgrading.html which mentions changing from 2.2. to 2.4 but the examples it gave did not show our use case:

Order allow,deny
Satisfy Any

So I don’t know what the corresponding Require directive for this would be. Since there is already Require valid-user is it possible to just remove those last two directives?

Though it looks like you already found a workable solution - just ignore those errors.

Bingo Eric:

Removing the

order ….

satisfy …

pair from the configuration resolved the problem without needing to overload LUA logging.

Ric