Customize passenger environment for apps

Background: Setting paths for tmp/scratch files for apps by setting $TMPDIR in the app is working (for tmp files of app on compute node).
New (different) issue: /tmp on OOD webservers is filling up, and seems to be from passenger tmp files. (/tmp/passenger.NAME don’t seem very large, but /tmp/RackMultipartNAME.dta have filled up /tmp in some cases)
Question: Is there a way to set passenger_data_buffer_dir for apps?
I see that a number of values for the passenger environment are set in /var/lib/ondemand-nginx/config/puns/USER.conf, some of which come from /etc/ood/config/nginx_stage.yml, but passenger_data_buffer_dir isn’t among them.
Would this idea resolve the issue? Is there a better approach?

Hi and thanks for the question.

The simplest way to set something like this is to look at the nginx_stage docs, and notice the passenger_options configuration option:

You can add configurations from Configuration reference - Nginx - Passenger Library by simply throwing them into this hash and they will be picked up and applied.

Let me know if this doesn’t work as expected though of if you need more clarification.

Thanks for the information!

Setting passenger_data_buffer_dir in passenger_options in nginx_stage.yml did result in the value being set in /var/lib/ondemand-nginx/config/puns/USER.conf. But passenger tmp files are still being written in /tmp rather than the path set. The path isn’t available at boot time as it is NFS-mounted. After restarting the httpd service on the server, and ‘Restart Web Server’ in the website, the new path still isn’t used. (I have confirmed that the path exists, is writable by all (permission 777), and SELinux is disabled.)

Is setting passenger_data_buffer_dir an incorrect approach to the issue, or might something be preventing it from working?

It’s possible it’s due to the NSF mounting. Is there a way you can define the tmp but use something on the local fs rather than the NSF mounted fs? That would at least show you if it is to do with the boot order.

The restarting of the PUN should have been sufficient to get this to work, so knowing about whether that fs not being there at boot is the key here.

Thanks for the suggestion. I tried using a local path and the passenger tmp files were still written in /tmp rather than the value of passenger_data_buffer_dir.

Ok, sorry for the issue.

What version of OOD are you running and what OS?

I’ll have to look at this on my end to try and see what is happening as that should work.

It is OOD v2.0.20 on RHEL8.

Also tested with OOD 2.0.28 on RHEL8 (with same result).

Ok, so it looks like Passenger does not quite play as nice as we’d hoped. They use a variable within the application called Tempfile that it does look like we can work with, but only using the env var for the dashboard of TMPDIR.

The solution could be instead of telling passenger to handle this, instead set it for the dashboard itself. This can be done by having a /etc/ood/config/apps/dashboard/env file that then has an entry for:


This should work with the NFS mounted filesystem and have these files land where you are hoping.

Please let me know if this works as expected for you.

Thanks for investigating. I tried adding a line for TMPDIR to /etc/ood/config/apps/dashboard/env in a few different ways, and for all attempts it kept using /tmp rather than the path I set. I tried both TMPDIR=/your/tmp and TMPDIR="/your/tmp", for both local and NFS mounted filesystems.

I confirmed this worked in my dev environment. I’d check 2 things

  • bounce your web server when you make a config change for it to pickup any new configuration.
  • check the permissions on the directory you set TMPDIR to.

Here’s how Ruby tries to find a tmpdir and you can see it requires it to be a directory that’s not only writable but world writable (with no sticky bit set).

Here’s what I saw in my logs to confirm. You can see the @tempfile paramter is in my $HOME

  Parameters: {"relativePath"=>"null", "name"=>"core.21946", 
"type"=>"application/octet-stream", "parent"=>"/users/PZS0714/johrstrom", 
@original_filename="core.21946", @content_type="application/octet-stream", 
@headers="Content-Disposition: form-data; name=\"file\"; 
filename=\"core.21946\"\r\nContent-Type: application/octet-stream\r\n">, 

Thanks for the information. I have done some further testing based on it.

  • For all tests, I restarted the httpd service and did ‘Restart Web Server’ in the browser. A few times, I also rebooted the server.
  • The Ruby fragment shows that the tmpdir must be writable, and if it’s world-writable then the sticky bit must be set. For one test case, the sticky bit wasn’t set when it was needed, so I corrected that.

I copied the tmpdir method into a test script which I ran on the server with different values of TMPDIR set. It seems to give the expected result for all cases. Script test.rb is:


class D

  @@systmpdir ||= defined?(Etc.systmpdir) ? Etc.systmpdir : '/tmp'

  def gettmp()
    ['TMPDIR', 'TMP', 'TEMP', ['system temporary path', @@systmpdir], ['/tmp']*2, ['.']*2].find do |name, dir|
      unless dir
        next if !(dir = ENV[name]) or dir.empty?
      dir = File.expand_path(dir)
      stat = File.stat(dir) rescue next
        when !
          warn "#{name} is not a directory: #{dir}"
        when !stat.writable?
          warn "#{name} is not writable: #{dir}"
        when stat.world_writable? && !stat.sticky?
          warn "#{name} is world-writable: #{dir}"
          break dir
    end or raise ArgumentError, "could not find a temporary directory"

d =
mydir = d.gettmp()
puts "tmpdir is: #{mydir}";   

I get these results:

$ echo $TMPDIR

$ ruby test.rb 
tmpdir is: /tmp
$ export TMPDIR=/tmp/test
$ ruby test.rb 
tmpdir is: /tmp/test
$ export TMPDIR=/exists/tmp 
$ ruby test.rb 
tmpdir is: /exists/tmp
$ export TMPDIR=/doesnt/exist         
$ ruby test.rb 
tmpdir is: /tmp

Logs like those you show look like they would be helpful. Where can I find them? (I only know where the Apache and Nginx logs are.)

They’re from development logs. I’m not sure how to enable them in production, as they’d be very verbose as they log a lot of information.

In any case, you could add an initializer at this location that could just print what we’re looking for.

# /etc/ood/config/apps/dashboard/initializers/tmp_debug.rb

puts "TMPDIR is currently #{Dir.tmpdir}"

And you’ll see output in the Nginx logs like:

App 127355 output: TMPDIR is currently /tmp

# ... or if it works
App 129486 output: TMPDIR is currently /users/PZS0714/johrstrom/temp/tmp_test

There’s also 1 more thing that may be getting in our way. The environment variables in /etc/ood/config/apps/dashboard/env won’t override existing environment variables. That is, if your system has somehow set TMPDIR, it won’t be overwritten by just the env file.

To overcome this, you can set TMPDIR in /etc/ood/config/apps/dashboard/env.overload. overload files aren’t documented because they’re mostly a development feature. For example you could override $HOME or $USER to some disastrous effect, so use this with caution.

I tried that and get:

App 34551 output: TMPDIR is currently /my/expected/path

So that part seems to be working. But directories passenger.NAME are still being created in /tmp. (I’ve been checking these since they’re always created, although RackMultipartNAME.dta are more important to place in a different location.) Should directories passenger.NAME be created in TMPDIR, or am I checking the wrong thing?

Sorry - I should have been clearer. Yes what you are seeing is what I anticipated. Setting the TMPDIR in that location is only for the dashboard, which is lower in the stack than Passenger. In any case, I thought the RackMultipartNAME are the real culprit for blowing up any drive, and so was focusing on that.

For the other settings/other temporary passenger things, maybe you’re looking for this or at least I’d search that page for tmp to indicate how you can migrate everything off of that directory.

I also tried putting the TMPDIR line from /etc/ood/config/apps/dashboard/env in /etc/ood/config/apps/dashboard/env.overload. This doesn’t change the behavior. Still get correct value for TMPDIR in the logs.

Thanks for clarifying. Since it seems like config should work for RackMultipartNAME, I’ll get someone to test that.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.