OOD Host configuration recommendations

Get Help
1

Posting general contents of an offline email thread here in case it would be of assistance to anyone else in the future:

From: Ashok Ragavendran , Brown University

The question relates to server loads and how you manage balancing and provisioning of the actual front end web servers. I am wondering if you can provide any information on the current configuration that is being implemented at the OSC and any metrics on the loads that your experience. Any recommendations you can provide within this context would be really valuable in setting up our infrastructure over here. We do have JMeter on premises for load testing and I am curious to know if you have employed that tool or any other platform for load testing for your deployment. Other than this, any other advice that you could provide would also be extremely valuable.

From Alan Chalker, OSC

At a high level, we have some specs on our setup available here: https://osc.github.io/ood-documentation/master/requirements.html

From Trey Dockendorf, OSC

The OSC instance of OnDemand is a single VM on a VMware cluster with 16 vCPUs and 64GB of RAM. These specs have always been far more than we actually need. In the last week our peek users at any one time has been 113 while the peak memory consumption has been around 10GB and peak CPU % usage has been around 50%. We recently switched Apache MPM to use the event module which is thread based for Apache connections to reduce load on the system. I think the default is prefork and that spawns processes to handle connections.

We also have a classroom instance of OnDemand that has had much higher load than our main instance as it has many classes and students accessing the web instance at the same time and it also had same specs as our main OnDemand instance and we found that switching to event MPM module was key to having that system better handle the load of hundreds of concurrent connections. Both our production instance and classroom instance are configured in Apache to handle 512 concurrent connections and that’s been sufficient thus far.

I can provide config examples for how we configure Apache as the MPM configuration is outside the scope of what OnDemand configures for you.

I do not believe we’ve used jmeter against any instances of OnDemand.

For Apache MPM we have the following files in /opt/rh/httpd24/root/etc/httpd/conf.modules.d:

# event.conf:
<IfModule mpm_event_module>
  ServerLimit            32
  StartServers           2
  MaxRequestWorkers      512
  MinSpareThreads        25
  MaxSpareThreads        75
  ThreadsPerChild        32
  MaxRequestsPerChild    0
  ThreadLimit            512
  ListenBacklog          511
</IfModule>
 
# event.load
LoadModule mpm_event_module modules/mod_mpm_event.so

You will want to make sure to remove mentions mod_mpm_prefix as that is replaced by the mpm_event_module. Also in our case the OnDemand VM is dedicated to OnDemand and one thing you can’t have with mpm_event_module is mod_php, which isn’t needed by OnDemand so if your VM is a dedicated OnDemand VM then it should be fine. Any Apache modules that aren’t thread-safe will not work with mpm_event_module, but none directly required by OnDemand have this issue. The file names we use and the fact they are split is just a byproduct of using Puppet to deploy Apache. I think a stock httpd24-httpd install would have these in 00-mpm.conf.

1 Like
2

Thanks for the info Allen!