Open OnDemand and AFS

General Discussion
41

pun_custom_env_declarations:

  • KRB5CCNAME (a hyphen, not a period, obviously)
42

Yes I used a hyphen just like the commented out examples in the fileā€¦

43

Jeff, is it possible to verify the environment variables are set and the values?

44

Where/when? I assume before the app starts up (given you canā€™t start the app up?)

45

Correct. I want to verify the value of KRB5CCNAME set in profile

46

Do you have this setting in this file? I think you need it for anything to pass to Nginx.

You can add env > /tmp/env.txt directly in the profile, but you need this config below for it to be injected into the Nginxā€™s environment.

# /etc/ood/config/nginx_stage.yml

pun_custom_env_declarations:
  - KRB5CCNAME

https://osc.github.io/ood-documentation/latest/reference/pun-environment.html

Assuming Nginx boots (even if the Ruby dashboard app canā€™t) you can see the Nginx workerā€™s process environment through /proc/<PID>/environ (you can try /proc/self/environ just to see).

47

Yes I have that setting. Iā€™ll check the file.

48

SUDO_USER is correct. KRB5CCNAME is not.

49

You mean UNPRIV_USER? Do you have a typo (could it be that simple)?

#!/bin/bash

# SUDO_COMMAND  is something like '... nginx_stage -u ktrout -a ...'

UNPRIV_USER=$(echo $SUDO_COMMAND | sed -r 's#.+\-u (.+) \-a.+#\1#g')
export KRB5CCNAME="FILE:/var/run/httpd/clientcaches/$UNPRIV_USER@ENEA.IT"
50

Sorry. I saw a SUDO_USER variable and copied/pasted. UNPRIV_USER is not in /proc/self/environ

51

Thatā€™s expected because UNPRIV_USER is just a temporary variable.

KRB5CCNAME is not correct, but itā€™s there?

52

In /proc/self/environ yes. KRB5CCNAME=FILE:/tmp/krb5cc_34180_G7hUSG

53

Sorry, not self - thatā€™s a magic keyword for the current process (Iā€™m guessing your terminal/bash session).

You need to try to start a PUN and find your Nginx worker processesā€™ PID through ps and see itā€™s environment through /proc/<PID of the Nginx worker>/environ.

54

So:

root 425524 1 0 16:29 ? 00:00:00 nginx: master process (dbars
dbarstis 425527 425524 0 16:29 ? 00:00:00 nginx: worker process

/proc/425527/environ is a binary file. strings, od and grep -a return nothing.

55

Put the env command in the profile. KRB5CCNAME is missing the user
KRB5CCNAME=FILE:/var/run/httpd/clientcaches/@CRC.ND.EDU

56

OK, so what else in the env output can you use to extract the username? SUDO_COMMAND should be thereā€¦ and I think that sed command should be working, though with that env output you can hack a bit to see whatā€™s available and why that sed may not be working.

Itā€™s not really a binary file, itā€™s just not really properly formatted text (I think the word seperators are weird keys). I think you can cat it and see search it manually.

57

So /etc/ood/profile looks like:

export UNPRIV_USER=$(echo $SUDO_COMMAND | sed -r ā€˜s#.+-u (.+) -a.+#\1#gā€™)
export KRB5CCNAME=ā€œFILE:/var/run/httpd/clientcaches/$UNPRIV_USER@CRC.ND.EDUā€
/bin/env > /tmp/env.txt

env.txt looks like:

SHELL=/bin/sh
PKG_CONFIG_PATH=/opt/ood/ondemand/root/usr/lib64/pkgconfig
KRB5CCNAME=FILE:/var/run/httpd/clientcaches/@CRC.ND.EDU
PWD=/root
LOGNAME=root
MANPATH=/opt/ood/ondemand/root/usr/share/man:
HOME=/root
LANG=en_US.UTF-8
X_SCLS=ondemand
GEM_PATH=/opt/ood/ondemand/root/usr/share/gems/3.0:/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1:
GEM_HOME=/opt/ood/ondemand/root/usr/share/gems/3.0
USER=root
SHLVL=1
LD_LIBRARY_PATH=/opt/ood/ondemand/root/usr/lib64
PATH=/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
UNPRIV_USER=
RUBYLIB=/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:/opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby
_=/bin/env

environ returns nothing (this one is a new process):

[root@ood-test 476003]# cat environ
[root@ood-test 476003]#

-r-------- 1 dbarstis campus 0 Dec 8 13:17 environ

So it doesnā€™t appear SUDO_COMMAND is set. It appears I have other things going on.

58

Thatā€™s odd. What is your pun_stage_cmd in this file? It should be sudo <some command> or just left as the default.

A quick read of the sudo man page says that that environment variable should be populated when the sudo command is issued.

https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html

59

So it was commented out. I uncommented it and now the environment variables appear to be set.

SHELL=/bin/bash
SUDO_GID=48
PKG_CONFIG_PATH=/opt/ood/ondemand/root/usr/lib64/pkgconfig
OOD_ALLOWED_HOSTS=ood-test.crc.nd.edu
SUDO_COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u dbarstis -a https%3a%2f%2food-test.crc.nd.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri
SUDO_USER=apache
KRB5CCNAME=FILE:/var/run/httpd/clientcaches/dbarstis@CRC.ND.EDU
PWD=/
LOGNAME=root
XDG_SESSION_TYPE=unspecified
MANPATH=/opt/ood/ondemand/root/usr/share/man:
HOME=/root
LANG=C
X_SCLS=ondemand
GEM_PATH=/opt/ood/ondemand/root/usr/share/gems/3.0:/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1:
GEM_HOME=/opt/ood/ondemand/root/usr/share/gems/3.0
XDG_SESSION_CLASS=background
TERM=unknown
USER=root
SHLVL=1
XDG_SESSION_ID=c15
LD_LIBRARY_PATH=/opt/ood/ondemand/root/usr/lib64
XDG_RUNTIME_DIR=/run/user/0
PATH=/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
SUDO_UID=48
UNPRIV_USER=dbarstis
MAIL=/var/mail/root
RUBYLIB=/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:/opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby
_=/bin/env

Still getting Permission denied @ dir_s_mkdir - /afs/crc.nd.edu/user/d/dbarstis/ondemand (Errno::EACCES) but appear to be getting closer.

60

So I put the tokens and klist commands in the profile. I do not have a token which could explain the permission denied error.

Luigi, do you get a token? Shouldnā€™t mod_waklog be executing the kinit/aklog?

Tokens held by the Cache Manager:

ā€“End of listā€“
Ticket cache: FILE:/var/run/httpd/clientcaches/dbarstis@CRC.ND.EDU
Default principal: dbarstis@CRC.ND.EDU

Valid starting Expires Service principal
12/11/23 14:04:02 01/10/24 14:04:02 krbtgt/CRC.ND.EDU@CRC.ND.EDU