Ok. I did that, and in the ood-portal.conf file in the apache config, this line is there:
SetEnv OOD_PUN_PRE_HOOK_EXPORTS “OIDC_ACCESS_TOKEN,FOO”
I just don’t see any evidence of those variables coming out of apache though. I put this line in the ood_prehook.sh script: export > /var/tmp/testing, and there’s nothing OOD related in it. There’s nothing referring to FOO in the apache debug logs either.
Just to be clear - you’re bouncing apache every time you reconfigure correct?
We’re missing something obvious here, because by all measures this should be working for you. I’m sure export is the same as env, but I know for sure this works and I’ve seen these things in the env output. Indeed there’s a test for the same, so I’m sort of at a loss as to why this isn’t working now for you.
Yes, I’m restarting http, and even bounced the box to make sure there was nothing hanging around.
Where are the apache variables being exported TO? There’s a few like OOD_USER_MAP_MATCH, OOD_PUN_STAGE_CMD, where are those being evaluated…what’s reading them? Maybe I can start to look there for extra debugging?
not sure if this helps at all, but in the pun .conf file that’s created when I log in, I only see these environments listed at the beginning:
env USER;
env LOGNAME;
env ONDEMAND_VERSION;
env ONDEMAND_PORTAL;
env ONDEMAND_TITLE;
env SECRET_KEY_BASE;
env NGINX_FILE_UPLOAD_MAX;
env OOD_DASHBOARD_TITLE;
env OOD_PORTAL;
env OOD_DEV_APPS_ROOT;
env OOD_FILES_URL;
env OOD_EDITOR_URL;
env RAILS_LOG_TO_STDOUT;
env OOD_BRAND_BG_COLOR;
env OOD_BRAND_LINK_ACTIVE_BG_COLOR;
env OOD_DASHBOARD_LOGO;
env PATH;
env LD_LIBRARY_PATH;
env X_SCLS;
env MANPATH;
env PCP_DIR;
env PERL5LIB;
env PKG_CONFIG_PATH;
env PYTHONPATH;
env XDG_DATA_DIRS;
env SCLS;
env RUBYLIB;
env KRB5CCNAME;
Here’s the block where we’re doing this work. It should be in /opt/ood/mode_ood_proxy/lib/ood/nginx_stage.lua.
We for a process - set the environment variables for that process - then invoke the command.
I’ll see if I can figure out the lua to dump the env_table just to see what it has at this stage. I know this is in the weeds, but I don’t know where else to find out that the deal is.
I did find one issue with set-k8s-creds.sh where it wasn’t able to find kubectl, so I manually set it to /usr/local/bin/kubectl (the other .sh file has a path setting in it). That makes my .kube/config file look a little better, but we’re still missing the tokens that we need.
Oh, this is getting annoying. I modified the nginx_stage.lua file and added a couple of lines:
local f = assert(io.open("/var/tmp/teststuff","w"))
for key,value in pairs(env_table) do
posix.setenv("OOD_" .. key, value) -- sudo rules allow for OOD_* env vars
f:write("OOD_" .. key, value)
end
unsurprising, in the /var/tmp/teststuff file after restarting the web server, I see:
OOD_OIDC_ACCESS_TOKEN<tokencontents>
So, the nginx_stage.lua sees the token, now the problem is after this step. I’ll assume posix.setenv did its job properly.
I’m just ‘guessing’ at this point that maybe sudo isn’t passing the OOD variables.
facepalm - what’s your /etc/sudoers.d/ood look like? Did you install from a package?
It’s the default one from the package, but I’m going to check the main /etc/sudoers file to see if maybe the organization put something in place that would stop that from working (not sure if they did, just a possibility).
Defaults:apache !requiretty, !authenticate
Defaults:apache env_keep += "NGINX_STAGE_* OOD_*"
apache ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage
Cmnd_Alias KUBECTL = /usr/local/bin/kubectl, /usr/bin/kubectl, /bin/kubectl
Defaults!KUBECTL !syslog
I enabled sudo debugging which made me think maybe this was happening because I saw this line:
exec /opt/ood/nginx_stage/sbin/nginx_stage [/opt/ood/nginx_stage/sbin/nginx_stage nginx -u myuser -s stop] [LANG=C MAIL=/var/mail/root PATH=/sbin:/bin:/usr/sbin:/usr/bin LOGNAME=root USER=root USERNAME=root HOME=/root SHELL=/bin/bash TERM=unknown SUDO_COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage nginx -u myuser -s stop SUDO_USER=apache SUDO_UID=48 SUDO_GID=48 XDG_SESSION_ID=c20 XDG_RUNTIME_DIR=/run/user/0]
If that doesn’t lead anywhere, I might put nginx_stage inside a wrapper to dump the env at that point.
I put an env dump in nginx_stage and it does NOT have any OOD variables in it. Is that running post sudo?
I think this is closer to whatever my problem is. I gave the apache user a shell, logged in and did:
export OOD_STUFF=“whatever”, then ran sudo /opt/ood/nginx_stage/sbin/nginx_stage, and my variable didn’t end up in the output.
I looked through the other environment variables that were supposed to be kept, so I set LC_MONETARY=“whatever”, and that made it through the sudo output.
No, the pre hook will run during the nginx_stage command before we actually boot nginx.
What OS are you on? @tdockendorf do you know what could be preventing these variables from passing into sudo? Some other sudo rule or configuration?
Ok, I got the variable set now, but here’s what I had to do, maybe there’s a config somewhere that’s blocking it from working the normal way.
I have a feeling there’s a config elsewhere on the system that would prevent me from needing all that.
I’m on RedHat 7. There very well could be something in the /etc/sudoers that says don’t preserve environments by default.
Of course after all of that, I don’t know if rancher will like the token, but that’s a whole other issue. I did notice in the rebuilt ~/.kube/config, I have the id-token now, but no refresh-token, should that be in there, or is that normal?
Yea - good to hear we’ve at least got it down to this 1 thing.
I think there’s an Apache configuration to request refresh tokens that you probably have to enable. Also there could be something on the Keycloak side to enable it too.
I have a strong feeling after all this Rancher won’t allow this to work because its authentication mechanism for a kube config does not seem to be happy with OIDC tokens and still requires ‘authentication’.
2 things - there is a note in the docs about Audiences. You’ve got to set that up if you want to use OOD tokens directly on k8s.
OR - instead of just grabbing the token from Apache - you could do a OIDC key exchange, then write the correct kube config.
I did have an audience created that should have been the Rancher client. I’ll keep poking around there.
When I had apache debugging turned on, I did see a refresh token returned, so it looks like keycloak is sending it. I’m not seeing the environment variable OIDC_REFRESH_TOKEN being set/passed anywhere though.
Duh, you have to explicitly export any environment variable you want to use. so add it to pun_pre_hook_exports
And that duh was to me not you, sorry if it read like that, I should have recognized that.