Customize passenger environment for apps

milberg · February 7, 2023, 12:43am

Background: Setting paths for tmp/scratch files for apps by setting $TMPDIR in the app is working (for tmp files of app on compute node).
New (different) issue: /tmp on OOD webservers is filling up, and seems to be from passenger tmp files. (/tmp/passenger.NAME don’t seem very large, but /tmp/RackMultipartNAME.dta have filled up /tmp in some cases)
Question: Is there a way to set passenger_data_buffer_dir for apps?
I see that a number of values for the passenger environment are set in /var/lib/ondemand-nginx/config/puns/USER.conf, some of which come from /etc/ood/config/nginx_stage.yml, but passenger_data_buffer_dir isn’t among them.
Would this idea resolve the issue? Is there a better approach?

travert · February 7, 2023, 3:02pm

Hi and thanks for the question.

The simplest way to set something like this is to look at the nginx_stage docs, and notice the passenger_options configuration option:
https://osc.github.io/ood-documentation/latest/reference/files/nginx-stage-yml.html?highlight=passenger_options

You can add configurations from Configuration reference - Nginx - Passenger Library by simply throwing them into this hash and they will be picked up and applied.

Let me know if this doesn’t work as expected though of if you need more clarification.

milberg · February 7, 2023, 8:31pm

Thanks for the information!

Setting passenger_data_buffer_dir in passenger_options in nginx_stage.yml did result in the value being set in /var/lib/ondemand-nginx/config/puns/USER.conf. But passenger tmp files are still being written in /tmp rather than the path set. The path isn’t available at boot time as it is NFS-mounted. After restarting the httpd service on the server, and ‘Restart Web Server’ in the website, the new path still isn’t used. (I have confirmed that the path exists, is writable by all (permission 777), and SELinux is disabled.)

Is setting passenger_data_buffer_dir an incorrect approach to the issue, or might something be preventing it from working?

travert · February 8, 2023, 5:42pm

It’s possible it’s due to the NSF mounting. Is there a way you can define the tmp but use something on the local fs rather than the NSF mounted fs? That would at least show you if it is to do with the boot order.

The restarting of the PUN should have been sufficient to get this to work, so knowing about whether that fs not being there at boot is the key here.

milberg · February 9, 2023, 9:09pm

Thanks for the suggestion. I tried using a local path and the passenger tmp files were still written in /tmp rather than the value of passenger_data_buffer_dir.

travert · February 9, 2023, 9:19pm

Ok, sorry for the issue.

What version of OOD are you running and what OS?

I’ll have to look at this on my end to try and see what is happening as that should work.

milberg · February 9, 2023, 9:21pm

It is OOD v2.0.20 on RHEL8.

milberg · February 9, 2023, 9:38pm

Also tested with OOD 2.0.28 on RHEL8 (with same result).

travert · February 10, 2023, 3:29pm

Ok, so it looks like Passenger does not quite play as nice as we’d hoped. They use a variable within the application called Tempfile that it does look like we can work with, but only using the env var for the dashboard of TMPDIR.

The solution could be instead of telling passenger to handle this, instead set it for the dashboard itself. This can be done by having a /etc/ood/config/apps/dashboard/env file that then has an entry for:

TMPDIR=/your/tmp

This should work with the NFS mounted filesystem and have these files land where you are hoping.

Please let me know if this works as expected for you.

milberg · February 10, 2023, 10:20pm

Thanks for investigating. I tried adding a line for TMPDIR to /etc/ood/config/apps/dashboard/env in a few different ways, and for all attempts it kept using /tmp rather than the path I set. I tried both TMPDIR=/your/tmp and TMPDIR="/your/tmp", for both local and NFS mounted filesystems.

jeff.ohrstrom · February 13, 2023, 2:41pm

I confirmed this worked in my dev environment. I’d check 2 things

bounce your web server when you make a config change for it to pickup any new configuration.
check the permissions on the directory you set TMPDIR to.

Here’s how Ruby tries to find a tmpdir and you can see it requires it to be a directory that’s not only writable but world writable (with no sticky bit set).

github.com

ruby/ruby/blob/bc6e587e86a4a4a46af3b1c2fbe1ff3b399735a6/lib/tmpdir.rb#L2296+


      
          # frozen_string_literal: true
          #
          # tmpdir - retrieve temporary directory path
          #
          # $Id$
          #
          
          
require 'fileutils'
          begin
            require 'etc.so'
          rescue LoadError # rescue LoadError for miniruby
          end
          
          
class Dir
          
          
  @@systmpdir ||= defined?(Etc.systmpdir) ? Etc.systmpdir : '/tmp'
          
          
  ##
            # Returns the operating system's temporary file path.
          
          
  def self.tmpdir
              ['TMPDIR', 'TMP', 'TEMP', ['system temporary path', @@systmpdir], ['/tmp']*2, ['.']*2].find do |name, dir|
                unless dir
                  next if !(dir = ENV[name]) or dir.empty?
                end
                dir = File.expand_path(dir)
                stat = File.stat(dir) rescue next
                case
                when !stat.directory?
                  warn "#{name} is not a directory: #{dir}"
                when !stat.writable?
                  warn "#{name} is not writable: #{dir}"
                when stat.world_writable? && !stat.sticky?
                  warn "#{name} is world-writable: #{dir}"
                else
                  break dir
                end
              end or raise ArgumentError, "could not find a temporary directory"
            end
          
          
  # Dir.mktmpdir creates a temporary directory.
            #
            # The directory is created with 0700 permission.
            # Application should not change the permission to make the temporary directory accessible from other users.
            #
            # The prefix and suffix of the name of the directory is specified by
            # the optional first argument, <i>prefix_suffix</i>.
            # - If it is not specified or nil, "d" is used as the prefix and no suffix is used.
            # - If it is a string, it is used as the prefix and no suffix is used.
            # - If it is an array, first element is used as the prefix and second element is used as a suffix.
            #
            #  Dir.mktmpdir {|dir| dir is ".../d..." }
            #  Dir.mktmpdir("foo") {|dir| dir is ".../foo..." }
            #  Dir.mktmpdir(["foo", "bar"]) {|dir| dir is ".../foo...bar" }
            #
            # The directory is created under Dir.tmpdir or
            # the optional second argument <i>tmpdir</i> if non-nil value is given.
            #
            #  Dir.mktmpdir {|dir| dir is "#{Dir.tmpdir}/d..." }
            #  Dir.mktmpdir(nil, "/var/tmp") {|dir| dir is "/var/tmp/d..." }
            #
            # If a block is given,
            # it is yielded with the path of the directory.
            # The directory and its contents are removed
            # using FileUtils.remove_entry before Dir.mktmpdir returns.
            # The value of the block is returned.
            #
            #  Dir.mktmpdir {|dir|
            #    # use the directory...
            #    open("#{dir}/foo", "w") { something using the file }
            #  }
            #
            # If a block is not given,
            # The path of the directory is returned.
            # In this case, Dir.mktmpdir doesn't remove the directory.
            #
            #  dir = Dir.mktmpdir
            #  begin
            #    # use the directory...
            #    open("#{dir}/foo", "w") { something using the file }
            #  ensure
            #    # remove the directory.
            #    FileUtils.remove_entry dir
            #  end
            #
            def self.mktmpdir(prefix_suffix=nil, *rest, **options)
              base = nil
              path = Tmpname.create(prefix_suffix || "d", *rest, **options) {|path, _, _, d|
                base = d
                mkdir(path, 0700)
              }
              if block_given?
                begin
                  yield path.dup
                ensure
                  unless base
                    stat = File.stat(File.dirname(path))
                    if stat.world_writable? and !stat.sticky?
                      raise ArgumentError, "parent directory is world writable but not sticky"
                    end
                  end
                  FileUtils.remove_entry path
                end
              else
                path
              end
            end
          
          
  # Temporary name generator
            module Tmpname # :nodoc:
              module_function
          
          
    def tmpdir
                Dir.tmpdir
              end
          
          
    # Unusable characters as path name
              UNUSABLE_CHARS = "^,-.0-9A-Z_a-z~"
          
          
    # Dedicated random number generator
              RANDOM = Random.new
              class << RANDOM # :nodoc:
                # Maximum random number
                MAX = 36**6 # < 0x100000000
          
          
      # Returns new random string upto 6 bytes
                def next
                  rand(MAX).to_s(36)
                end
              end
              private_constant :RANDOM
          
          
    # Generates and yields random names to create a temporary name
              def create(basename, tmpdir=nil, max_try: nil, **opts)
                origdir = tmpdir
                tmpdir ||= tmpdir()
                n = nil
                prefix, suffix = basename
                prefix = (String.try_convert(prefix) or
                          raise ArgumentError, "unexpected prefix: #{prefix.inspect}")
                prefix = prefix.delete(UNUSABLE_CHARS)
                suffix &&= (String.try_convert(suffix) or
                            raise ArgumentError, "unexpected suffix: #{suffix.inspect}")
                suffix &&= suffix.delete(UNUSABLE_CHARS)
                begin
                  t = Time.now.strftime("%Y%m%d")
                  path = "#{prefix}#{t}-#{$$}-#{RANDOM.next}"\
                         "#{n ? %[-#{n}] : ''}#{suffix||''}"
                  path = File.join(tmpdir, path)
                  yield(path, n, opts, origdir)
                rescue Errno::EEXIST
                  n ||= 0
                  n += 1
                  retry if !max_try or n < max_try
                  raise "cannot generate temporary name using `#{basename}' under `#{tmpdir}'"
                end
                path
              end
            end
          end

This file has been truncated. show original

Here’s what I saw in my logs to confirm. You can see the @tempfile paramter is in my $HOME

  Parameters: {"relativePath"=>"null", "name"=>"core.21946", 
"type"=>"application/octet-stream", "parent"=>"/users/PZS0714/johrstrom", 
"file"=>#<ActionDispatch::Http::UploadedFile:0x00007f2df4d252e8 
@tempfile=#<Tempfile:/users/PZS0714/johrstrom/temp/tmp_test/RackMultipart20230210-51306-v9vxte.21946>, 
@original_filename="core.21946", @content_type="application/octet-stream", 
@headers="Content-Disposition: form-data; name=\"file\"; 
filename=\"core.21946\"\r\nContent-Type: application/octet-stream\r\n">, 
"fs"=>"fs"}

milberg · February 15, 2023, 5:30pm

Thanks for the information. I have done some further testing based on it.

For all tests, I restarted the httpd service and did ‘Restart Web Server’ in the browser. A few times, I also rebooted the server.
The Ruby fragment shows that the tmpdir must be writable, and if it’s world-writable then the sticky bit must be set. For one test case, the sticky bit wasn’t set when it was needed, so I corrected that.

I copied the tmpdir method into a test script which I ran on the server with different values of TMPDIR set. It seems to give the expected result for all cases. Script test.rb is:

#!/usr/bin/ruby

class D

  @@systmpdir ||= defined?(Etc.systmpdir) ? Etc.systmpdir : '/tmp'

  def gettmp()
    ['TMPDIR', 'TMP', 'TEMP', ['system temporary path', @@systmpdir], ['/tmp']*2, ['.']*2].find do |name, dir|
      unless dir
        next if !(dir = ENV[name]) or dir.empty?
      end
      dir = File.expand_path(dir)
      stat = File.stat(dir) rescue next
      case
        when !stat.directory?
          warn "#{name} is not a directory: #{dir}"
        when !stat.writable?
          warn "#{name} is not writable: #{dir}"
        when stat.world_writable? && !stat.sticky?
          warn "#{name} is world-writable: #{dir}"
        else
          break dir
      end
    end or raise ArgumentError, "could not find a temporary directory"
  end
end

d = D.new
mydir = d.gettmp()
puts "tmpdir is: #{mydir}";

I get these results:

$ echo $TMPDIR

$ ruby test.rb 
tmpdir is: /tmp
$ export TMPDIR=/tmp/test
$ ruby test.rb 
tmpdir is: /tmp/test
$ export TMPDIR=/exists/tmp 
$ ruby test.rb 
tmpdir is: /exists/tmp
$ export TMPDIR=/doesnt/exist         
$ ruby test.rb 
tmpdir is: /tmp

Logs like those you show look like they would be helpful. Where can I find them? (I only know where the Apache and Nginx logs are.)

jeff.ohrstrom · February 15, 2023, 6:27pm

They’re from development logs. I’m not sure how to enable them in production, as they’d be very verbose as they log a lot of information.

In any case, you could add an initializer at this location that could just print what we’re looking for.

# /etc/ood/config/apps/dashboard/initializers/tmp_debug.rb

puts "TMPDIR is currently #{Dir.tmpdir}"

And you’ll see output in the Nginx logs like:

App 127355 output: TMPDIR is currently /tmp

# ... or if it works
App 129486 output: TMPDIR is currently /users/PZS0714/johrstrom/temp/tmp_test

There’s also 1 more thing that may be getting in our way. The environment variables in /etc/ood/config/apps/dashboard/env won’t override existing environment variables. That is, if your system has somehow set TMPDIR, it won’t be overwritten by just the env file.

To overcome this, you can set TMPDIR in /etc/ood/config/apps/dashboard/env.overload. overload files aren’t documented because they’re mostly a development feature. For example you could override $HOME or $USER to some disastrous effect, so use this with caution.

milberg · February 15, 2023, 9:11pm

I tried that and get:

App 34551 output: TMPDIR is currently /my/expected/path

So that part seems to be working. But directories passenger.NAME are still being created in /tmp. (I’ve been checking these since they’re always created, although RackMultipartNAME.dta are more important to place in a different location.) Should directories passenger.NAME be created in TMPDIR, or am I checking the wrong thing?

jeff.ohrstrom · February 15, 2023, 9:44pm

Sorry - I should have been clearer. Yes what you are seeing is what I anticipated. Setting the TMPDIR in that location is only for the dashboard, which is lower in the stack than Passenger. In any case, I thought the RackMultipartNAME are the real culprit for blowing up any drive, and so was focusing on that.

For the other settings/other temporary passenger things, maybe you’re looking for this or at least I’d search that page for tmp to indicate how you can migrate everything off of that directory.

https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_instance_registry_dir

milberg · February 15, 2023, 9:45pm

I also tried putting the TMPDIR line from /etc/ood/config/apps/dashboard/env in /etc/ood/config/apps/dashboard/env.overload. This doesn’t change the behavior. Still get correct value for TMPDIR in the logs.

milberg · February 15, 2023, 9:47pm

Thanks for clarifying. Since it seems like config should work for RackMultipartNAME, I’ll get someone to test that.

system · August 14, 2023, 9:48pm

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Proper file location to set nginx parameters Get Help question	5	353	January 29, 2023
Customizing Files app favorites OOD 2.0.29 Get Help ondemand2 , question	9	247	August 8, 2023
Setting different environment for each user Get Help question	3	225	March 19, 2023
Separate folder plugins and passenger apps Get Help question	4	658	May 26, 2022
Creating a Node.js/React.js Passenger App Get Help question	6	339	February 12, 2023

Customize passenger environment for apps

Related Topics