Hello, we are seeing MaxRequestWorkers error on our OOD.
[mpm_prefork:error] [pid 2186] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
Could you help me with how to see the MaxRequestWorkers value for OOD and how to increase it?
I found <IfModule mpm_worker_module> line in /opt/rh/httpd24/root/etc/httpd/conf.modules.d/01-cgi.conf. Is it safe to modify this file?
Thank you,
Gisoo
I think here’s a good serverfault topic on the subject.
Yes it’s safe to modify these files, but they could be overwritten by a new install, so you’d be wise to write some automation around configuring your apache. Though the file /opt/rh/httpd24/root/etc/httpd/conf.modules.d/00-mpm.conf determines which type of mpm module you use (you’ve linked a CGI config file, that’s not the file/config you’re looking fo). Looks like default is prefork. If you want to switch types, then modify this file, if not, just leave it. If you’re not using any PHP modules, I’d suggest switching to mpm_event_module, otherwise stick with prefork. There are also links in this file to the actual apache documentation on all these threading modules.
Then I’d suggest dropping a brand new file into /opt/rh/httpd24/root/etc/httpd/conf.d/ like say worker.conf that won’t get overwritten and putting your worker configurations there.
Looks like prefork is the default and what you’ve got configured (from the logs given), so that element has the only values in it.
<IfModule mpm_prefork_module>
MaxRequestWorkers 512
</IfModule>
<IfModule mpm_event_module>
</IfModule>
<IfModule mpm_worker_module>
</IfModule>
If you do stick with prefork, just keep an eye on your memory when you start to top out in request workers.
Thank you. I have switched it to mpm_event_module.
That’s awesome, but you’ll still maybe want to do some quick math about how many concurrent connections you expect and whether or not your configuration can support that. Good luck!
Hello Jeff,
How do you check concurrent users and general ood usage. Do you use some special tool to get a data? I know you guys are using Google Analytic, but unsure it’s also show real data or only historic data.
So with apache you’ve got a bunch of options, but basically you’ll want to plugin to whatever Monitoring systems you’re already using for your HPC infrastructure.
We do use Google Analytics for pulling historic things like monthly reports. We use Splunk if need to do something a little more ad-hoc both in real-time or historic, so we’re indexing the apache logs. We’re currently using ganglia for a little bit of graphing, but are shortly going to switch to a prometheus stack for graphinga and alerting at some point. All of these require you to buy into that stack, a Splunk subscription or the Prometheus infrastructure, etc. If you already have these things, cool!
If not, you may try netdata with the apache plugin.This is a install just on the OOD webserver so you don’t need a larger by in to other stacks. Looks like you can even set it behind apache for authentication. (This is just my suggestion because I’ve found netdata super helpful in the past running Apache stacks. I haven’t heard of other folks running it - we at OSC don’t - but again, in the past, in other projects it was super helpful. It’ll give you concurrent connections, request per second, CPU, memory, all in very nice graphs without having to install and maintain other infrastructure).
Other than that, a quick google search of “apache log parser bash” came up with some results. Those could be simple enough to use to achieve what you need. Like a simple netstat -ant | grep -c ESTAB every 30 or so seconds may give you good baseline, or at least it’s some data easily obtained.
(As an aside, thanks for the question, because it looks like we have no documentation on how to monitor OOD so I’ve created this ticket for that!)