PassengerAgent threads

General Discussion
1

Hi!

We noted that each PUN was spawning a lot of PassengerAgent threads. On OoD servers with hundreds of users, this obviously doesn’t look too good.

Here’s an example on a 64-core/128-thread server:

# /opt/ood/nginx_stage/sbin/nginx_stage nginx_list | wc -l
61
# pgrep -c PassengerAgent
120
# pgrep -wc PassengerAgent
16132

So, 61 active PUNs, 120 PassengerAgent processes, and… over 16 thousand PassengerAgent threads.

Looking for ways to keep this under control, we found the list of possible configuration options for Passenger at Configuration reference - Nginx - Passenger Library and tried different combinations in our /etc/ood/config/nginx_stage.yml file. Unfortunately, all the relevant options (passenger_thread_count, passenger_concurrency_model…) seem to be only available in Passenger Enterprise edition, and result in messages like this:

Error – nginx: [emerg] “passenger_thread_count” directive : this feature is only available in Phusion Passenger Enterprise. You are currently running the open source Phusion Passenger. Please learn more about and/or buy Phusion Passenger Enterprise at Passenger - Enterprise grade web app server for Ruby, Node.js, Python ; in /var/lib/ondemand-nginx/config/puns/kilian.conf:85

Is there any way to control the number of PassengerAgent threads? Or any plans to eventually move to a different, non-freemium web server?

Thanks!

2

Are you noticing any performance issues or hitting any kind of max_thread_count that lead to looking into the thread count?

The short answer is no, there’s no conversation around moving away from Phusion Passenger.

What does the memory usage look like?

Is your web-node where the PUNs spin up its own separate server or is it a login-node on the cluster?

Also, many of those threads could be idle and not in use based on how phusion passenger works. This is a complex question is what I’m getting at and this isn’t quite enough to indicate a problem without more data.

But, were there specific performance issues happening that lead you to look at the thread counts?

3

No specific performance issue related to this at this time. Memory usage is not a concern, and we’re nowhere close to /proc/sys/kernel/threads-max, but still, spawning #CPUs threads for each single PUN doesn’t seem like a very sane approach, and it makes things harder to monitor and debug:process trees and htop outputs are more difficult to make sense of, just because of the sheer number of irrelevant (ie. idle) threads.
Plus, not being able to control this just because of a basic configuration option is locked out in a enterprise license is quite frustrating, and doesn’t look too great.

But I guess it’s more of a principles question than just a potential performance issue. Thanks for clarifying that there’s no plan to explore Phusion Passenger alternatives, I was just curious if that excessive number of PassengerAgent threads had been brought up before.

4

Sure, glad to discuss and thanks for bringing it up! There’s been no discussion largely because we’ve not heard anything from the community about it (that I’m aware of). So, this is great to post and share for the project history.

Typically Rails comes with Puma instead of Passenger, but even though it does have a cost tier, the community edition is pretty great for running apps in various languages and ease of configuration.

Puma does offer (based on what I’ve read) better performance tuning and manages threads more efficiently, but it has a higher maintenance cost. I wasn’t around when the choice was made, but my guess is the easier configuration won the day back when they discussed this and inertia kept it going.

If others in the community have data and stories to share around this we’d love to hear it though.