Failed upgrade 1.6.19 ->1.7.6 cannot load such file -- bundler/setup (LoadError)

We had a working 1.6.19 version and installed the 1.7.6 version

error when starting

[ N 2020-03-19 11:50:25.6380 20921/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2020-03-19 11:50:25.7067 20925/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2020-03-19 11:50:25.7069 20925/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2020-03-19 11:50:25.8505 20925/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 20925
App 20985 output: Error: The application encountered the following error: cannot load such file -- bundler/setup (LoadError)
App 20985 output:     /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
App 20985 output:     /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
App 20985 output: /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:363: 

in activate_gem' App 20985 output: /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:221:in block in run_load_path_setup_codeā€™
App 20985 output: /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:527:in running_bundler' App 20985 output: /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:220:i n run_load_path_setup_codeā€™
App 20985 output: /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:86:in load_app' App 20985 output: /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:116:in block in module:Appā€™
App 20985 output: /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380:in run_block_and_record_step_progress' App 20985 output: /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:115:in module:Appā€™
App 20985 output: /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:28:in <module:PhusionPassenger>' App 20985 output: /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:27:in ā€™
[ E 2020-03-19 11:50:26.5219 20925/T17 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/ood/apps/sys/dashboard: The app
lication encountered the following error: cannot load such file ā€“ bundler/setup (LoadError)
Error ID: d4b54ac9
Error details saved to: /tmp/passenger-error-tejLCj.html

I thought the needed ruby is part of ood

[xx ~]# yum list installed |grep ondemand
ondemand.x86_64                  1.7.6-1.el7                     @openondemand_web
ondemand-apache.x86_64           1.7-8.el7                       @openondemand_web
ondemand-gems-1.7.6.x86_64       1.7.6-1.el7                     @openondemand_web
ondemand-nginx.x86_64            1.17.3-2.p6.0.4.el7             @openondemand_web
ondemand-nodejs.x86_64           1.7-8.el7                       @openondemand_web
ondemand-passenger.x86_64        6.0.4-2.el7                     @openondemand_web
ondemand-passenger-devel.x86_64  6.0.4-2.el7                     @openondemand_web
ondemand-python.x86_64           1.7-8.el7                       @openondemand_web
ondemand-ruby.x86_64             1.7-8.el7                       @openondemand_web
ondemand-rubygem-bundler.noarch  1.17.3-1.el7                    @openondemand_web
ondemand-runtime.x86_64          1.7-8.el7                       @openondemand_web

The ruby

[xx ~]# yum list installed |grep ruby
ondemand-ruby.x86_64             1.7-8.el7                       @openondemand_web
ondemand-rubygem-bundler.noarch  1.17.3-1.el7                    @openondemand_web
rh-ruby25.x86_64                 2.5-2.el7                       @centos_scl_rh 
rh-ruby25-ruby.x86_64            2.5.5-7.el7                     @centos_scl_rh 
rh-ruby25-ruby-devel.x86_64      2.5.5-7.el7                     @centos_scl_rh 
rh-ruby25-ruby-irb.noarch        2.5.5-7.el7                     @centos_scl_rh 
rh-ruby25-ruby-libs.x86_64       2.5.5-7.el7                     @centos_scl_rh 
rh-ruby25-rubygem-bigdecimal.x86_64
rh-ruby25-rubygem-did_you_mean.noarch
rh-ruby25-rubygem-io-console.x86_64
rh-ruby25-rubygem-json.x86_64    2.1.0-7.el7                     @centos_scl_rh 
rh-ruby25-rubygem-openssl.x86_64 2.1.2-7.el7                     @centos_scl_rh 
rh-ruby25-rubygem-psych.x86_64   3.0.2-7.el7                     @centos_scl_rh 
rh-ruby25-rubygem-rake.noarch    12.3.0-7.el7                    @centos_scl_rh 
rh-ruby25-rubygem-rdoc.noarch    6.0.1-7.el7                     @centos_scl_rh 
rh-ruby25-rubygems.noarch        2.7.6.2-7.el7                   @centos_scl_rh 
rh-ruby25-rubygems-devel.noarch  2.7.6.2-7.el7                   @centos_scl_rh 
rh-ruby25-runtime.x86_64         2.5-2.el7                       @centos_scl_rh

Hey, Iā€™m looking into this now. Iā€™m unable to replicate right off the bat, but I can see that somehow I kept my ruby rh-ruby24-* gems after upgrade. That may be important. Iā€™ll keep you posted.

@baverhey sorry, Iā€™ve spent the morning trying to replicate your issue and I canā€™t. I did so on a fresh VM so I was able to jump directly from 1.6.19 to 1.7.6. I tried with and without the rh-ruby24* libraries. I tried bouncing things and/or keeping them around after the upgrade.

Can you let me know what other configurations you run with? Do you have anything set in /etc/ood/config/nginx_stage.yml? We definitely should have all the ruby gems setup.

Can you attach your Error details saved to: /tmp/passenger-error-tejLCj.html file? (you can obfuscate what you need, Iā€™m really interested in the environment variables).

Thx for looking in to this.

in /etc/ood/config/nginx_stage.yml
nginx_bin: /usr/bin/ood_pun_wrapper

Still the same file as for 1.6
this files contains:

#!/bin/bash
# Start a per-user nginx in the right namespace/cgroup/...

# DEBUG=1 runs 'set -x'
# START_SHELL=1 starts a bash shell instead of nginx
# ood_pun_wrapper -c <file> use <file> as config for nginx
#                 username is taken from this (standard argument when starting ood)
# config=<username> overwrite the reading of the config file with <username>

allArgs="$@"

while [ "$1" != "" ]; do
    if [ "$1" == "-c" ]; then
        config="$2"
    fi
    shift
done

username=`echo $config | sed "s/^.*\/\([a-zA-Z0-9]*\)\.conf/\1/"`

eval $(sudo -u $username USER=$username /usr/bin/vsc_env bash)

# make sure internal keys are generated if they don't exist yet
sudo -u $username /etc/profile.d/vscautokey.sh

args=()

if [ ${DEBUG:-0} -gt 0 ]; then
    set -x
fi

# These could possibly be more generic...
# --bind: mount the folders in the same place
# Generic folders
mount=(/etc /opt /run /usr /var/run/munge /var/lib/sss /usr/share/lmod /var/www)
# VSC folders
mount+=(/apps /data /user /scratch /vscmnt /kyukon /phanpy)
# Ood nginx folders
mount+=(/var/log/ondemand-nginx /var/lib/ondemand-nginx /var/tmp/ondemand-nginx /var/run/ondemand-nginx)

# --symlink: recreate the symlink
link=(/bin /lib /lib64 /local /sbin)

# --dir: create a new empty folder
dir=(/var/tmp)

# --tmpfs: create a new tempfs
tmp=(/run)

# --setenv: copy env variables
env=(CPATH LD_LIBRARY_PATH PYTHONPATH PATH MANPATH XDG_DATA_DIRS PKG_CONFIG_PATH MODULEPATH HOSTNAME)

for d in "${dir[@]}"; do
    args+=("--dir" "$d")
done

for t in "${tmp[@]}"; do
    args+=("--tmpfs" "$t")
done

for m in "${mount[@]}"; do
    args+=("--bind" "$m" "$m")
done

for l in "${link[@]}"; do
    args+=("--symlink"  "`readlink -f $l`" "$l")
done

for e in "${env[@]}"; do
    if [ -n "${!e}" ]; then
        args+=("--setenv" "$e" "${!e}")
    fi
done

# Special folders
args+=("--dev" "/dev")
args+=("--proc" "/proc")
args+=("--dev-bind" "/dev" "/dev")

# Mark that we're in a wrapper
args+=("--setenv" "PUN_WRAPPER" "1")

# Make a temp dir
tempdir="/tmp/$username"
mkdir -p $tempdir
sudo chmod 700 $tempdir
sudo chown $username:$username $tempdir

# Set the /tmp to the temp dir
args+=("--bind" "$tempdir" "/tmp")

# Set the user
args+=("--setenv" "USER" "$username")

if [ ${START_SHELL:-0} -gt 0 ]; then
    sudo cgexec -g cpu,cpuacct,memory:hpcusers/$username bwrap ${args[@]} /usr/bin/bash
else
    sudo cgexec -g cpu,cpuacct,memory:hpcusers/$username bwrap ${args[@]} /opt/ood/ondemand/root/usr/sbin/nginx $allArgs
fi

the mentioned files in /tmp are not created. in/tmp but in

/tmp/vsc40682/passenger-xx (so the displayed path is not correct, due to our wrapper i presume)

How can i uploud a file here?
USed fiel bin for now: https://filebin.net/a2asmeaixkjy40ph

we do have snoopy

2020-03-19T20:25:43.236024+01:00 ood_login snoopy[32546]: [uid:0 username:root sid:7327 tty:(none) cwd:/ filename:/usr/bin/tr]: /usr/bin/tr \0      
2020-03-19T20:25:44.987848+01:00 ood_login snoopy[32549]: [uid:48 username:apache sid:11920 tty:(none) cwd:/ filename:/opt/ood/ood_auth_map/bin/ood_auth_map.regex]: /opt/ood/ood_auth_map/bin/ood_auth_map.regex vsc40682    
2020-03-19T20:25:45.196171+01:00 ood_login snoopy[32552]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/opt/ood/nginx_stage/bin/ruby]: /opt/ood/nginx_stage/bin/ruby /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb    
2020-03-19T20:25:45.203365+01:00 ood_login snoopy[32552]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/opt/rh/rh-ruby25/root/usr/bin/ruby]: ruby /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb    
2020-03-19T20:25:45.305678+01:00 ood_login snoopy[32581]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/usr/bin/uname]: /usr/bin/uname -p    
2020-03-19T20:25:45.668394+01:00 ood_login snoopy[32591]: [uid:48 username:apache sid:11920 tty:(none) cwd:/ filename:/opt/ood/ood_auth_map/bin/ood_auth_map.regex]: /opt/ood/ood_auth_map/bin/ood_auth_map.regex vsc40682    
2020-03-19T20:25:46.007849+01:00 ood_login snoopy[32594]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/opt/ood/nginx_stage/bin/ruby]: /opt/ood/nginx_stage/bin/ruby /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb    
2020-03-19T20:25:46.015053+01:00 ood_login snoopy[32594]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/opt/rh/rh-ruby25/root/usr/bin/ruby]: ruby /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb    
2020-03-19T20:25:46.120889+01:00 ood_login snoopy[32622]: [uid:2540682 username:vsc40682 sid:31377 tty:(none) cwd:/var/www/ood/apps/sys/dashboard filename:/usr/bin/uname]: /usr/bin/uname -p    
2020-03-19T20:25:54.939897+01:00 ood_login snoopy[32641]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/sbin/ip]: /sbin/ip addr    
2020-03-19T20:25:54.954035+01:00 ood_login snoopy[32642]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/sh]: /bin/sh -c /bin/ps -C mmfsd    
2020-03-19T20:25:54.961473+01:00 ood_login snoopy[32642]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/ps]: /bin/ps -C mmfsd    
2020-03-19T20:25:54.984252+01:00 ood_login snoopy[32645]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/sbin/ibstat]: /usr/sbin/ibstat    
2020-03-19T20:25:54.997483+01:00 ood_login snoopy[32647]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/sh]: /bin/sh -c /bin/df -P '/var/mmfs' '/tmp/mmfs'    
2020-03-19T20:25:55.004078+01:00 ood_login snoopy[32647]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/df]: /bin/df -P /var/mmfs /tmp/mmfs    
2020-03-19T20:25:55.019840+01:00 ood_login snoopy[32649]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmccr]: /usr/lpp/mmfs/bin/mmccr check -Y -e    
2020-03-19T20:25:55.031253+01:00 ood_login snoopy[32651]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/sh]: /bin/sh -c ibdev2netdev    
2020-03-19T20:25:55.038140+01:00 ood_login snoopy[32651]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ibdev2netdev]: ibdev2netdev    
2020-03-19T20:25:55.044743+01:00 ood_login snoopy[32653]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/class/infiniband/    
2020-03-19T20:25:55.050330+01:00 ood_login snoopy[32654]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/bond0/device/vendor    
2020-03-19T20:25:55.055021+01:00 ood_login snoopy[32655]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/bonding_masters/device/vendor    
2020-03-19T20:25:55.060133+01:00 ood_login snoopy[32656]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/eth0/device/vendor    
2020-03-19T20:25:55.064806+01:00 ood_login snoopy[32657]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/eth1/device/vendor    
2020-03-19T20:25:55.069338+01:00 ood_login snoopy[32658]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/ib0/device/vendor    
2020-03-19T20:25:55.074008+01:00 ood_login snoopy[32659]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/ib1/device/vendor    
2020-03-19T20:25:55.078648+01:00 ood_login snoopy[32660]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/grep]: grep 0x15b3 /sys/class/net/lo/device/vendor    
2020-03-19T20:25:55.084813+01:00 ood_login snoopy[32661]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/infiniband/mlx5_0/device/resource    
2020-03-19T20:25:55.089124+01:00 ood_login snoopy[32662]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/class/net/    
2020-03-19T20:25:55.094725+01:00 ood_login snoopy[32663]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib0/device/resource    
2020-03-19T20:25:55.099221+01:00 ood_login snoopy[32665]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib0/dev_port    
2020-03-19T20:25:55.103356+01:00 ood_login snoopy[32666]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/tsctl]: /usr/lpp/mmfs/bin/tsctl nqStatus -Y    
2020-03-19T20:25:55.105162+01:00 ood_login snoopy[32668]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib0/dev_id    
2020-03-19T20:25:55.111803+01:00 ood_login snoopy[32672]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib0/carrier    
2020-03-19T20:25:55.117980+01:00 ood_login snoopy[32676]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices    
2020-03-19T20:25:55.120455+01:00 ood_login snoopy[32674]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmdiag]: /usr/lpp/mmfs/bin/mmdiag --deadlock    
2020-03-19T20:25:55.124135+01:00 ood_login snoopy[32678]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices/0000:01:01.0/infiniband/    
2020-03-19T20:25:55.129652+01:00 ood_login snoopy[32682]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices/0000:01:02.0/infiniband/    
2020-03-19T20:25:55.131945+01:00 ood_login snoopy[32680]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcommon]: /usr/lpp/mmfs/bin/mmcommon run showCfgValue tscCmdPortRange    
2020-03-19T20:25:55.136203+01:00 ood_login snoopy[32683]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib1/device/resource    
2020-03-19T20:25:55.140860+01:00 ood_login snoopy[32684]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/infiniband/mlx5_1/device/resource    
2020-03-19T20:25:55.145151+01:00 ood_login snoopy[32685]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/class/net/    
2020-03-19T20:25:55.150700+01:00 ood_login snoopy[32686]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib0/device/resource    
2020-03-19T20:25:55.155265+01:00 ood_login snoopy[32687]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib1/device/resource    
2020-03-19T20:25:55.159936+01:00 ood_login snoopy[32688]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib1/dev_port    
2020-03-19T20:25:55.165341+01:00 ood_login snoopy[32690]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -s    
2020-03-19T20:25:55.166598+01:00 ood_login snoopy[32691]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib1/dev_id    
2020-03-19T20:25:55.168260+01:00 ood_login snoopy[32692]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/id]: /usr/bin/id -u    
2020-03-19T20:25:55.173800+01:00 ood_login snoopy[32695]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/cat]: cat /sys/class/net/ib1/carrier    
2020-03-19T20:25:55.178229+01:00 ood_login snoopy[32697]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -r    
2020-03-19T20:25:55.180437+01:00 ood_login snoopy[32699]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices    
2020-03-19T20:25:55.181202+01:00 ood_login snoopy[32700]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -m    
2020-03-19T20:25:55.184610+01:00 ood_login snoopy[32701]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^ID/ { gsub(/"/, "", $2) ; print toupper($2) ; exit } /etc/os-release    
2020-03-19T20:25:55.187811+01:00 ood_login snoopy[32703]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices/0000:01:01.0/infiniband/    
2020-03-19T20:25:55.188914+01:00 ood_login snoopy[32704]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^PRETTY_NAME/ { gsub(/"/, "", $2) ; print $2; exit } /etc/os-release    
2020-03-19T20:25:55.194246+01:00 ood_login snoopy[32707]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/ls]: ls /sys/bus/pci/devices/0000:01:02.0/infiniband/    
2020-03-19T20:25:55.195402+01:00 ood_login snoopy[32706]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^VERSION_ID/ && NF>=2 \#012     { gsub(/"/, "", $2); dot=index($2, "."); \#012       if (dot>0) { print substr($2, 0, dot-1),substr($2, dot+1) } \#012       else { print $2,"0" } \#012       exit \#012     } /etc/os-release    
2020-03-19T20:25:55.202574+01:00 ood_login snoopy[32709]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/systemctl]: /usr/bin/systemctl --version    
2020-03-19T20:25:55.203050+01:00 ood_login snoopy[32710]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk #012    BEGIN { v = 0 }#012    /^systemd/ { v = $2 ; exit }#012    END { print v }    
2020-03-19T20:25:55.209828+01:00 ood_login snoopy[32712]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk $1 == "logDir" {value = $2} END {print value} /var/mmfs/gen/mmfs.cfg    
2020-03-19T20:25:55.210561+01:00 ood_login snoopy[32711]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmfsadm]: /usr/lpp/mmfs/bin/mmfsadm dump verbs    
2020-03-19T20:25:55.214064+01:00 ood_login snoopy[32713]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/mkdir]: /bin/mkdir -p /var/mmfs/gen/ /var/mmfs/etc/ /var/mmfs/tmp/cmdTmpDir.mmcommon.32680/_empty /var/adm/ras/ /var/mmfs/gen/mmLockDir /var/mmfs/mmbackup /dev /var/mmfs/gen/nodeFiles/ /etc/gpfs/    
2020-03-19T20:25:55.218039+01:00 ood_login snoopy[32716]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/mkdir]: /bin/mkdir -m 0700 -p /var/mmfs/ssl/ /var/mmfs/ssl/stage/ /var/mmfs/mmpmon/ /var/mmfs/ssl/keyServ/ /var/mmfs/ssl/keyServ/tmp/mmcommon.32680/ /var/mmfs/tmp/cleanupAllow    
2020-03-19T20:25:55.222399+01:00 ood_login snoopy[32715]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcommon]: /usr/lpp/mmfs/bin/mmcommon run showCfgValue tscCmdPortRange    
2020-03-19T20:25:55.255273+01:00 ood_login snoopy[32717]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -s    
2020-03-19T20:25:55.258145+01:00 ood_login snoopy[32718]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/id]: /usr/bin/id -u    
2020-03-19T20:25:55.266230+01:00 ood_login snoopy[32719]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -r    
2020-03-19T20:25:55.268882+01:00 ood_login snoopy[32720]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/uname]: /bin/uname -m    
2020-03-19T20:25:55.271716+01:00 ood_login snoopy[32721]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^ID/ { gsub(/"/, "", $2) ; print toupper($2) ; exit } /etc/os-release    
2020-03-19T20:25:55.275589+01:00 ood_login snoopy[32722]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^PRETTY_NAME/ { gsub(/"/, "", $2) ; print $2; exit } /etc/os-release    
2020-03-19T20:25:55.279562+01:00 ood_login snoopy[32724]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/head]: /usr/bin/head -1 /var/mmfs/gen/mmsdrfs    
2020-03-19T20:25:55.281213+01:00 ood_login snoopy[32723]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F = /^VERSION_ID/ && NF>=2 \#012     { gsub(/"/, "", $2); dot=index($2, "."); \#012       if (dot>0) { print substr($2, 0, dot-1),substr($2, dot+1) } \#012       else { print $2,"0" } \#012       exit \#012     } /etc/os-release    
2020-03-19T20:25:55.282789+01:00 ood_login snoopy[32725]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode /usr/bin/ssh    
2020-03-19T20:25:55.286911+01:00 ood_login snoopy[32728]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk #012    BEGIN { v = 0 }#012    /^systemd/ { v = $2 ; exit }#012    END { print v }    
2020-03-19T20:25:55.287205+01:00 ood_login snoopy[32727]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/systemctl]: /usr/bin/systemctl --version    
2020-03-19T20:25:55.287831+01:00 ood_login snoopy[32729]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode /usr/bin/scp    
2020-03-19T20:25:55.292307+01:00 ood_login snoopy[32730]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk $1 == "logDir" {value = $2} END {print value} /var/mmfs/gen/mmfs.cfg    
2020-03-19T20:25:55.292431+01:00 ood_login snoopy[32731]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/head]: /usr/bin/head -1 /var/mmfs/gen/mmfsNodeData    
2020-03-19T20:25:55.295377+01:00 ood_login snoopy[32732]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/mkdir]: /bin/mkdir -p /var/mmfs/gen/ /var/mmfs/etc/ /var/mmfs/tmp/cmdTmpDir.mmcommon.32715/_empty /var/adm/ras/ /var/mmfs/gen/mmLockDir /var/mmfs/mmbackup /dev /var/mmfs/gen/nodeFiles/ /etc/gpfs/    
2020-03-19T20:25:55.295915+01:00 ood_login snoopy[32733]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode 10.143.10.66    
2020-03-19T20:25:55.298408+01:00 ood_login snoopy[32734]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/mkdir]: /bin/mkdir -m 0700 -p /var/mmfs/ssl/ /var/mmfs/ssl/stage/ /var/mmfs/mmpmon/ /var/mmfs/ssl/keyServ/ /var/mmfs/ssl/keyServ/tmp/mmcommon.32715/ /var/mmfs/tmp/cleanupAllow    
2020-03-19T20:25:55.302423+01:00 ood_login snoopy[32735]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F: -v Yflag= #012      tolower($1)  == "tsccmdportrange" && ( $3 == "" || ","$3"," ~ /,620,/ ) {#012        parmValue = $2 }#012      END {#012        if (Yflag != "-Y") {#012          gsub("%25", "%", parmValue) ;#012          gsub("%3A", ":", parmValue) ;#012        }#012        print parmValue#012      }#012     /var/mmfs/gen/mmfs.cfg.show    
2020-03-19T20:25:55.308007+01:00 ood_login snoopy[32736]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F: -v Yflag= #012      tolower($1)  == "commandaudit" && ( $3 == "" || ","$3"," ~ /,620,/ ) {#012        parmValue = $2 }#012      END {#012        if (Yflag != "-Y") {#012          gsub("%25", "%", parmValue) ;#012          gsub("%3A", ":", parmValue) ;#012        }#012        print parmValue#012      }#012     /var/mmfs/gen/mmfs.cfg.show    
2020-03-19T20:25:55.313130+01:00 ood_login snoopy[32737]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/rm]: /bin/rm -f /var/mmfs/gen/mmsdrfs.32680 /var/mmfs/tmp/allClusterNodes.mmcommon.32680 /var/mmfs/tmp/allQuorumNodes.mmcommon.32680 /var/mmfs/tmp/allNonQuorumNodes.mmcommon.32680 /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32680.pub /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32680.priv /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32680.cert /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32680.keystore /var/mmfs/tmp/nodefile.mmcommon.32680 /var/mmfs/tmp/diskfile.mmcommon.32680 /var/mmfs/tmp/diskNamesFile.mmcommon.32680    
2020-03-19T20:25:55.316752+01:00 ood_login snoopy[32738]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/rm]: /bin/rm -rf /var/mmfs/tmp/cmdTmpDir.mmcommon.32680/ /var/mmfs/ssl/keyServ/tmp/mmcommon.32680/    
2020-03-19T20:25:55.359414+01:00 ood_login snoopy[32739]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/head]: /usr/bin/head -1 /var/mmfs/gen/mmsdrfs    
2020-03-19T20:25:55.362342+01:00 ood_login snoopy[32740]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode /usr/bin/ssh    
2020-03-19T20:25:55.366519+01:00 ood_login snoopy[32741]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode /usr/bin/scp    
2020-03-19T20:25:55.370985+01:00 ood_login snoopy[32742]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/bin/head]: /usr/bin/head -1 /var/mmfs/gen/mmfsNodeData    
2020-03-19T20:25:55.373887+01:00 ood_login snoopy[32743]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/usr/lpp/mmfs/bin/mmcmi]: /usr/lpp/mmfs/bin/mmcmi percentdecode 10.143.10.66    
2020-03-19T20:25:55.380506+01:00 ood_login snoopy[32744]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F: -v Yflag= #012      tolower($1)  == "tsccmdportrange" && ( $3 == "" || ","$3"," ~ /,620,/ ) {#012        parmValue = $2 }#012      END {#012        if (Yflag != "-Y") {#012          gsub("%25", "%", parmValue) ;#012          gsub("%3A", ":", parmValue) ;#012        }#012        print parmValue#012      }#012     /var/mmfs/gen/mmfs.cfg.show    
2020-03-19T20:25:55.385050+01:00 ood_login snoopy[32745]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/awk]: /bin/awk -F: -v Yflag= #012      tolower($1)  == "commandaudit" && ( $3 == "" || ","$3"," ~ /,620,/ ) {#012        parmValue = $2 }#012      END {#012        if (Yflag != "-Y") {#012          gsub("%25", "%", parmValue) ;#012          gsub("%3A", ":", parmValue) ;#012        }#012        print parmValue#012      }#012     /var/mmfs/gen/mmfs.cfg.show    
2020-03-19T20:25:55.389354+01:00 ood_login snoopy[32746]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/rm]: /bin/rm -f /var/mmfs/gen/mmsdrfs.32715 /var/mmfs/tmp/allClusterNodes.mmcommon.32715 /var/mmfs/tmp/allQuorumNodes.mmcommon.32715 /var/mmfs/tmp/allNonQuorumNodes.mmcommon.32715 /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32715.pub /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32715.priv /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32715.cert /var/mmfs/ssl/stage/tmpKeyData.mmcommon.32715.keystore /var/mmfs/tmp/nodefile.mmcommon.32715 /var/mmfs/tmp/diskfile.mmcommon.32715 /var/mmfs/tmp/diskNamesFile.mmcommon.32715    
2020-03-19T20:25:55.392556+01:00 ood_login snoopy[32747]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/bin/rm]: /bin/rm -rf /var/mmfs/tmp/cmdTmpDir.mmcommon.32715/ /var/mmfs/ssl/keyServ/tmp/mmcommon.32715/    
2020-03-19T20:25:55.487088+01:00 ood_login snoopy[32748]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/sbin/ibportstate]: /sbin/ibportstate -G 0xff0af7d0dd500017 1    
2020-03-19T20:25:55.566037+01:00 ood_login kernel: infiniband mlx5_0: _mlx5_ib_post_send:5893:(pid 32748): Send SMP MADs is not allowed    
2020-03-19T20:25:55.569002+01:00 ood_login snoopy[32750]: [uid:0 username:root sid:14103 tty:(none) cwd:/ filename:/sbin/ibportstate]: /sbin/ibportstate -G 0xff0af7d0dd500001 1    
2020-03-19T20:25:55.636138+01:00 ood_login kernel: infiniband mlx5_0: _mlx5_ib_post_send:5893:(pid 32750): Send SMP MADs is not allowed    
2020-03-19T20:25:58.250847+01:00 ood_login snoopy[32754]: [uid:0 username:root sid:7327 tty:(none) cwd:/ filename:/usr/bin/tr]: /usr/bin/tr \0      

a strace of the nginix process of the user:

31424 epoll_wait(12, [{EPOLLIN, {u32=21125168, u64=21125168}}], 512, -1) = 1
31424 accept4(8, {sa_family=AF_LOCAL, NULL}, [2], SOCK_NONBLOCK) = 3
31424 epoll_ctl(12, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=21125648, u64=21125648}}) = 0
31424 epoll_wait(12, [{EPOLLIN, {u32=21125648, u64=21125648}}], 512, 60000) = 1
31424 recvfrom(3, ā€œGET /pun/sys/dashboard HTTP/1.1\r\nHost: localhost\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\r\nSec-Fetch-Dest: document\r\nAccept: text/html,appliā€ā€¦, 1024, 0, NULL, NULL) = 996
31424 stat("/var/www/ood/apps/sys/dashboard/public", {st_mode=S_IFDIR|0755, st_size=4096, ā€¦}) = 0
31424 stat("/var/www/ood/apps/sys/dashboard/public/index.html", 0x136f070) = -1 ENOENT (No such file or directory)
31424 open("/var/www/ood/apps/sys/dashboard/Passengerfile.json", O_RDONLY|O_NONBLOCK) = -1 ENOENT (No such file or directory)
31424 stat("/var/www/ood/apps/sys/dashboard/passenger_wsgi.py", 0x1411360) = -1 ENOENT (No such file or directory)
31424 stat("/var/www/ood/apps/sys/dashboard/config.ru", {st_mode=S_IFREG|0644, st_size=153, ā€¦}) = 0
31424 epoll_ctl(12, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=21125648, u64=21125648}}) = 0
31424 socket(AF_LOCAL, SOCK_STREAM, 0) = 10
31424 ioctl(10, FIONBIO, [1]) = 0
31424 epoll_ctl(12, EPOLL_CTL_ADD, 10, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=21125888, u64=21125888}}) = 0
31424 connect(10, {sa_family=AF_LOCAL, sun_path="/tmp/passenger.kvfSWWe/agents.s/core"}, 110) = 0
31424 getsockopt(10, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
31424 writev(10, [{ā€œGET /pun/sys/dashboard HTTP/1.1\r\nConnection: close\r\nHost: localhost\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\r\nSec-Fetch-Dest: document\r\nAcceā€ā€¦, 2251}], 1) = 2251
31424 epoll_wait(12, [{EPOLLOUT, {u32=21125648, u64=21125648}}, {EPOLLOUT, {u32=21125888, u64=21125888}}], 512, 12000000) = 2
31424 epoll_wait(12, [{EPOLLIN|EPOLLOUT, {u32=21125888, u64=21125888}}], 512, 11999980) = 1
31424 recvfrom(10, ā€œHTTP/1.1 500 Internal Server Error\r\nStatus: 500 Internal Server Error\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Thu, 19 Mar 2020 19:39:50 +0000\r\nConnection: close\r\nContent-Length: 4384\r\ncache-control: no-cache, no-store, must-revalidate\r\n\r\n<!doctype hā€ā€¦, 16384, 0, NULL, NULL) = 4629
31424 writev(3, [{ā€œHTTP/1.1 500 Internal Server Error\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 4384\r\nConnection: close\r\nStatus: 500 Internal Server Error\r\nDate: Thu, 19 Mar 2020 19:39:50 +0000\r\ncache-control: no-cache, no-store, must-revalidate\r\nServer: nginxā€ā€¦, 293}, {"<!doctype html>\n<html lang=ā€œenā€>\n \n <meta charset=ā€œutf-8ā€>\n <meta name=ā€œviewportā€ content=ā€œwidth=device-width, initial-scale=1ā€>\n Weā€™re sorry, but something went wrong: Web application could not be started\n <style type=ā€œteā€ā€¦, 4384}], 2) = 4677
31424 epoll_wait(12, [{EPOLLIN|EPOLLOUT|EPOLLHUP|EPOLLRDHUP, {u32=21125888, u64=21125888}}, {EPOLLOUT, {u32=21125648, u64=21125648}}], 512, 12000000) = 2
31424 recvfrom(10, ā€œā€, 16384, 0, NULL, NULL) = 0
31424 close(10) = 0
31424 write(7, ā€œunix: - - [19/Mar/2020:20:39:50 +0100] ā€œGET /pun/sys/dashboard HTTP/1.1ā€ 500 4384 ā€œ-ā€ ā€œMozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36ā€ ā€œ157.193.11.196, 10.141.10.60ā€\nā€, 233) = 233
31424 close(3) = 0
31424 epoll_wait(12, <detached ā€¦>

Thanks! Right off the top you can try adding GEM_HOME, GEM_PATH and BUNDLE_HOME to env. Iā€™m fairly sure thatā€™s it.

Also, thanks for the great example cgroup wrapper script. I think thatā€™s excellent!

Just for posterity, hereā€™s my environment.

MANPATH=/opt/rh/rh-nodejs10/root/usr/share/man:/opt/rh/rh-ruby25/root/usr/local/share/man:/opt/rh/rh-ruby25/root/usr/share/man:/opt/rh/httpd24/root/usr/share/man:/opt/ood/ondemand/root/usr/share/man:
HOSTNAME=ba4e976e368f
GEM_HOME=/opt/ood/ondemand/root/usr/share/gems/2.5
SHELL=/bin/bash
TERM=xterm
LIBRARY_PATH=/opt/rh/httpd24/root/usr/lib64
X_SCLS=rh-nodejs10 rh-ruby25 httpd24 ondemand 
USER=jeff
LD_LIBRARY_PATH=/opt/rh/rh-nodejs10/root/usr/lib64:/opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/httpd24/root/usr/lib64:/opt/ood/ondemand/root/usr/lib64
SUDO_USER=apache
SUDO_UID=48
USERNAME=root
PATH=/opt/rh/rh-nodejs10/root/usr/bin:/opt/rh/rh-ruby25/root/usr/local/bin:/opt/rh/rh-ruby25/root/usr/bin:/opt/rh/httpd24/root/usr/bin:/opt/rh/httpd24/root/usr/sbin:/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
MAIL=/var/mail/root
PWD=/SH
LVL=0
SUDO_COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u jeff -a http%3a%2f%2flocalhost%3a8080%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri
HOME=/home/jeff
PYTHONPATH=/opt/rh/rh-nodejs10/root/usr/lib/python2.7/site-packages
LOGNAME=root
XDG_DATA_DIRS=/opt/rh/rh-ruby25/root/usr/local/share:/opt/rh/rh-ruby25/root/usr/share:/usr/local/share:/usr/share
GEM_PATH=/opt/ood/ondemand/root/usr/share/gems/2.5:/opt/ood/ondemand/root/usr/share/gems/2.5/ondemand/1.7.6:
PKG_CONFIG_PATH=/opt/rh/rh-ruby25/root/usr/local/lib64/pkgconfig:/opt/rh/rh-ruby25/root/usr/lib64/pkgconfig:/opt/rh/httpd24/root/usr/lib64/pkgconfig:/opt/ood/ondemand/root/usr/lib64/pkgconfig
SUDO_GID=48
RUBYLIB=/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:/opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby
PYTHON=python
ONDEMAND_VERSION=v1.7.7_rc3-upgrade-rails-myjobs.f9769201
ONDEMAND_PORTAL=ondemand
ONDEMAND_TITLE=Open OnDemand
SECRET_KEY_BASE=caf394f2a05e61c3e07df12791e80e9efc2d54f0dba2f30f79a98d2a5f3d23a2adec7d73f2e2a47eac9fe9316e376b47b0db1863abfc4ea194f4c33e8fce8ecb
OOD_DEV_APPS_ROOT=/var/www/ood/apps/dev/jeff/gateway
PASSENGER_USE_FEEDBACK_FD=true
UID=1000

adding GEM_HOME , GEM_PATH worked BUNDLE_HOME is not listed in your env

our path is bit different. i added to the env in a wrapper. that woked.
When adding the variables to sys/dashboard/.env.production id did not work.

How was this done in 1.6?
As is we would need to update for every ood version.

In sys/dashboard/.env.production
Is it not an idea to add a variable GEM_ROOT or someting that the users can fill in and uses that as base

or is this the purpose of BUNDEL_HOME?
in our case OOD_GEM_ROOT= /opt/rh/ondemand/
and ood addes the rest

Iā€™ll give more context below, but the simple answer is is that you shouldnā€™t be hard coding them. They get set as a part of the ondemand SCL so you should be able to pass them through like you do with PATH and LD_LIBRARY_PATH and all the others.

In 1.6 every app had itā€™s own relative vendor/bundle directory where gems where installed in.

Now weā€™re trying to optimize so install gems in a central location so all the apps can use the same gems and we donā€™t need to duplicate them.

GEM_HOME looks to be ruby version specific. When we update ruby versions, youā€™ll likely need to update this. GEM_PATH has the OOD version in it (1.7.6) so this will have to be updated every time. If you hard coded them, which you shouldnā€™t need to.

By the time youā€™re reading sys/dashboard/.env.production your application has already booted and thatā€™s too late. GEM_PATH related variables have to be populated correctly used during boot, i.e., for the app to fully boot up it has to find all the gems it needs.

The BUNDLE_HOME bit was a mistake on my part, I was actually thinking about BUNDLE_USER_CONFIG. We now set this to /dev/null by default in the nginx stage, but I donā€™t think you need set it or to pass it through here because the bundler default is what weā€™re overriding. So thereā€™s nothing to be done there.

Thx for the super support,
I just noticed the desktop app does not show anymoreā€¦

Thatā€™s odd. You put all the configs (yml files and submit/scheduler.yml.erb) into /etc/ood/config/apps/bc_desktop? And theyā€™re readable by you (your uid).

Itā€™s all deployed using quattor, so no Manual changes between 1.6 and 1.7 on our end, only the files changed by the rpm.

submit.yml.erb the default i think
ā€”
batch_connect:
template: vnc

We dont have and had any other .yml.erb in bc_desktop
so no submit/scheduler.yml.erb in bc_desktop

We do have in the cluster files for vnc

   vnc:
     script_wrapper: '

      module purge

      module load TurboVNC

      export WEBSOCKIFY_CMD="/usr/bin/websockify"

      %s'

WEBSOCKIFY_CMD also changed in env ?

(:+1: for automation)

Surprisingly that is valid yaml. I would suggest this though. In either case, I donā€™t believe thatā€™s the issue.

   vnc:
     script_wrapper: |
      module purge
      module load TurboVNC
      export WEBSOCKIFY_CMD="/usr/bin/websockify"
      %s

Are you saying you don't have any files in `/etc/ood/config/apps/bc_desktop`?  If that's the case, how did you configure the desktop in 1.6?

No custom configuration,
worked as is in 1.6.

 [root@xx bc_desktop]# pwd
/var/www/ood/apps/sys/bc_desktop
[root@xx bc_desktop]# ls -l *
-rw-r--r-- 1 root root 2386 Dec 20 20:16 CHANGELOG.md
-rw-r--r-- 1 root root  266 Dec 20 20:16 form.yml
-rw-r--r-- 1 root root 1087 Dec 20 20:16 LICENSE.txt
-rw-r--r-- 1 root root  312 Dec 20 20:16 manifest.yml
-rw-r--r-- 1 root root 2209 Dec 20 20:16 README.md
-rw-r--r-- 1 root root   35 Dec 20 20:16 submit.yml.erb

template:
total 12
-rwxr-xr-x 1 root root  100 Dec 20 20:16 before.sh.erb
drwxr-xr-x 2 root root 4096 Mar 19 10:03 desktops
-rwxr-xr-x 1 root root  502 Dec 20 20:16 script.sh.erb

You add configurations in /etc/ood/config (/etc not /var/www) . Youā€™re looking in the source directory, which although it didnā€™t change, it also would have gotten overwritten by the rpm installation.

You can view our configs we drop the apps folder, as is, into /etc/ood/config.

The settings for bc_desktop where not in quattor yet.
where can you modify the form for the desktop app?
the docs explicitly state

Before we begin modifying form attributes. Let us first take a look at the default form definition located in the source file /var/www/ood/apps/sys/bc_desktop/form.yml (do not modify):

we want just one desktop app mentioning for all the regular clusters using jobs
and one desktop app mentioning the login node cluster for post processing en.

we use the following code to get a dropdown o our clusters

<%

This header is not tested

require ā€˜jsonā€™
clusters = OodAppkit.clusters
clusters_comma_separated = clusters.map{|c| c.id}.join(",")
wrapper_command_start = "ood_erb_wrapper --clusters %s " % [clusters_comma_separated]
all_versions = JSON.parse(#{wrapper_command_start} --versions IPython)
all_versions = all_versions.sort
active_reservations = JSON.parse(#{wrapper_command_start} --reservations)
available_clusters = JSON.parse(#{wrapper_command_start} --has_available IPython)
%>

num_cores:
    label: "Number of cores"
    widget: "select"
    options:
        <% clusters.each do |key,c| %>
            <% max_cpus = OodAppkit.clusters[c.id].metadata["cpus"] %>

            <% if max_cpus.is_a? Integer %>
              <% 1.upto(max_cpus) do |i| %>
                - ["<%= i.to_s %>;<%= c.id %>", "<%= i.to_s %>"]
              <% end %>
            <% end %>
        <% end %>
    value: ""
    required: true


cluster:
    label: "Cluster"
    widget: "select"
    options:
        <% available_clusters.each do |c| %>
        - ["<%= c['name'] %>", "<%= c['module'] %>"]
        <% end %>
    value: ""
    required: true

Right now, unfortunately, we have a 1:1 ratio of form:cluster. That is, 1 form can only submit to 1 cluster. (but we have this ticket coming thatā€™ll fix that) but in the interim, there is a lot of overlap and like ours where youā€™ll basically the same file but for a different cluster.

You can still put logic into your submit/cluster.yml.erb but thatā€™s after youā€™ve made you choices and submitted the form.

That said, Iā€™m not sure what happens if you put a form.js in /etc/ood/config/apps/bc_desktop. I assume it reads it, and if it does you can add logic there to if cluster == foo; then max cpu = 30 and so on. You can see our jupyter for how we have hard coded mins and maxes in the form.yml then in the javascript,form.js, we react dynamically based off of the node_type change.

@baverhey In the default OnDemand install we have this file https://github.com/OSC/ondemand/blob/4992a049700e289cbd3f655c523f9bd477782e99/nginx_stage/etc/profile that executes source scl_source enable ondemand prior to launching the PUNs. This source scl_source enable ondemand should have addressed all the problems you ran into, without requiring manually setting the environment variables in the env file.

I wonder if this is a problem that we fixed in a later 1.7 patch version (the latest is 1.7.10). Or are you using something different than Software Collections?

1 Like