Greetings. Please forgive any newbie-posting errors.
We recently discovered that our httpd log directory is filling up due to entries (in both error and access logs) for healthchecker entries: we are in AWS and an ELB health check runs periodically. We don’t believe we’ve made any changes on the frequency of checks (knowingly), and have increased the interval between checks.
Has anyone else seen this issue? Any thoughts on what might be happening here?
Aside, we’re using OnDemand v4.0.11 on Amazon Linux2 (for not much longer); and running ParallelCluster 3.7.2.
Thanks for reading and considering this post.
Best,
Bob
Bob Freeman, PhD
Sr. RCD Facilitator
Harvard University Research Computing
I guess you could double check who’s making the health checks by checking the ip. Maybe it’s you, maybe it’s something else you’re unaware of?
Beyond that, you can use the logrotate program to rotate logs based off of size, not only time. So if you set an aggressive crontab entry (like every 5 minutes or similar) then logrotate can check your logs can rotate them based on current size of the file. That may help alleviate the issue if you can’t pinpoint or stop whoever is making the requests.
Turns out that the ELB healthcheck was only one of the “issues”. For some reason, the lua info messages have increased significantly, though the usage of our cluster has not increased similarly. So we’re wondering what has changed, and how. Perhaps a flag had been changed somewhere when performing our 2x/month maintenance and patching of systems (AL2 OS, ParallelCluster service, etc). If not that, then is there a deeper problem somewhere that is generating more log spam?
For now, we’re going to change the LUA logging to WARN instead of INFO, in hopes this will lessen the chattiness in the logs. But we’d prefer to lessen that # of log entries without having to ‘silence’ the logging by moving to WARN.