# OnDemand and Rancher integration

**URL:** <https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061>\
**Category:** General Discussion\
**Created:** [May 11, 2022, 1:46pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061 "2022-05-11T13:46:12Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 1:31pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/21 "2022-05-17T13:31:01Z")

</div>

Ok. I did that, and in the ood-portal.conf file in the apache config, this line is there:  
SetEnv OOD\_PUN\_PRE\_HOOK\_EXPORTS “OIDC\_ACCESS\_TOKEN,FOO”

I just don’t see any evidence of those variables coming out of apache though. I put this line in the ood\_prehook.sh script: export \> /var/tmp/testing, and there’s nothing OOD related in it. There’s nothing referring to FOO in the apache debug logs either.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 17, 2022, 1:51pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/22 "2022-05-17T13:51:41Z")

</div>

Just to be clear - you’re bouncing apache every time you reconfigure correct?

We’re missing something obvious here, because by all measures this should be working for you. I’m sure `export` is the same as `env`, but I know for sure this works and I’ve seen these things in the `env` output. Indeed there’s a test for the same, so I’m sort of at a loss as to why this isn’t working now for you.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 1:56pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/23 "2022-05-17T13:56:28Z")

</div>

Yes, I’m restarting http, and even bounced the box to make sure there was nothing hanging around.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 1:59pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/24 "2022-05-17T13:59:41Z")

</div>

Where are the apache variables being exported TO? There’s a few like OOD\_USER\_MAP\_MATCH, OOD\_PUN\_STAGE\_CMD, where are those being evaluated…what’s reading them? Maybe I can start to look there for extra debugging?

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 4:54pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/25 "2022-05-17T16:54:26Z")

</div>

not sure if this helps at all, but in the pun .conf file that’s created when I log in, I only see these environments listed at the beginning:

```auto
env USER;
env LOGNAME;
env ONDEMAND_VERSION;
env ONDEMAND_PORTAL;
env ONDEMAND_TITLE;
env SECRET_KEY_BASE;
env NGINX_FILE_UPLOAD_MAX;
env OOD_DASHBOARD_TITLE;
env OOD_PORTAL;
env OOD_DEV_APPS_ROOT;
env OOD_FILES_URL;
env OOD_EDITOR_URL;
env RAILS_LOG_TO_STDOUT;
env OOD_BRAND_BG_COLOR;
env OOD_BRAND_LINK_ACTIVE_BG_COLOR;
env OOD_DASHBOARD_LOGO;
env PATH;
env LD_LIBRARY_PATH;
env X_SCLS;
env MANPATH;
env PCP_DIR;
env PERL5LIB;
env PKG_CONFIG_PATH;
env PYTHONPATH;
env XDG_DATA_DIRS;
env SCLS;
env RUBYLIB;
env KRB5CCNAME;

```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 17, 2022, 5:14pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/26 "2022-05-17T17:14:01Z")

</div>

Here’s the block where we’re doing this work. It should be in `/opt/ood/mode_ood_proxy/lib/ood/nginx_stage.lua`.

We for a process - set the environment variables for _that_ process - then invoke the command.

> <https://github.com/OSC/ondemand/blob/3cae2bb0f1720e4d983296d48e9238c9ab8dee8e/mod_ood_proxy/lib/ood/nginx_stage.lua#L108-L121>

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 5:30pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/27 "2022-05-17T17:30:55Z")

</div>

I’ll see if I can figure out the lua to dump the env\_table just to see what it has at this stage. I know this is in the weeds, but I don’t know where else to find out that the deal is.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 17, 2022, 5:48pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/28 "2022-05-17T17:48:06Z")

</div>

I did find one issue with set-k8s-creds.sh where it wasn’t able to find kubectl, so I manually set it to /usr/local/bin/kubectl (the other .sh file has a path setting in it). That makes my .kube/config file look a little better, but we’re still missing the tokens that we need.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 1:36pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/29 "2022-05-18T13:36:31Z")

</div>

Oh, this is getting annoying. I modified the nginx\_stage.lua file and added a couple of lines:

```auto
local f = assert(io.open("/var/tmp/teststuff","w"))
    for key,value in pairs(env_table) do
      posix.setenv("OOD_" .. key, value) -- sudo rules allow for OOD_* env vars
      f:write("OOD_" .. key, value)
    end

```

unsurprising, in the /var/tmp/teststuff file after restarting the web server, I see:

```auto
OOD_OIDC_ACCESS_TOKEN<tokencontents>

```

So, the nginx\_stage.lua sees the token, now the problem is after this step. I’ll assume posix.setenv did its job properly.

I’m just ‘guessing’ at this point that maybe sudo isn’t passing the OOD variables.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 1:50pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/30 "2022-05-18T13:50:11Z")

</div>

facepalm - what’s your `/etc/sudoers.d/ood` look like? Did you install from a package?

> <https://github.com/OSC/ondemand/blob/e88b6f15acd67d92ed35d938699c0b388a63b5b2/packaging/files/sudo.erb>

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 2:00pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/31 "2022-05-18T14:00:34Z")

</div>

It’s the default one from the package, but I’m going to check the main /etc/sudoers file to see if maybe the organization put something in place that would stop that from working (not sure if they did, just a possibility).

```auto
Defaults:apache !requiretty, !authenticate
Defaults:apache env_keep += "NGINX_STAGE_* OOD_*"
apache ALL=(ALL) NOPASSWD: /opt/ood/nginx_stage/sbin/nginx_stage
Cmnd_Alias KUBECTL = /usr/local/bin/kubectl, /usr/bin/kubectl, /bin/kubectl
Defaults!KUBECTL !syslog

```

I enabled sudo debugging which made me think maybe this was happening because I saw this line:

```auto
 exec /opt/ood/nginx_stage/sbin/nginx_stage [/opt/ood/nginx_stage/sbin/nginx_stage nginx -u myuser -s stop] [LANG=C MAIL=/var/mail/root PATH=/sbin:/bin:/usr/sbin:/usr/bin LOGNAME=root USER=root USERNAME=root HOME=/root SHELL=/bin/bash TERM=unknown SUDO_COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage nginx -u myuser -s stop SUDO_USER=apache SUDO_UID=48 SUDO_GID=48 XDG_SESSION_ID=c20 XDG_RUNTIME_DIR=/run/user/0]

```

If that doesn’t lead anywhere, I might put nginx\_stage inside a wrapper to dump the env at that point.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 2:04pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/32 "2022-05-18T14:04:45Z")

</div>

I put an env dump in nginx\_stage and it does NOT have any OOD variables in it. Is that running post sudo?

I think this is closer to whatever my problem is. I gave the apache user a shell, logged in and did:  
export OOD\_STUFF=“whatever”, then ran sudo /opt/ood/nginx\_stage/sbin/nginx\_stage, and my variable didn’t end up in the output.

I looked through the other environment variables that were supposed to be kept, so I set LC\_MONETARY=“whatever”, and that made it through the sudo output.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 2:18pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/33 "2022-05-18T14:18:18Z")

</div>

> [@groucho64738](#):
>
> Is that running post sudo?

No, the pre hook will run during the `nginx_stage` command before we actually boot nginx.

What OS are you on? @tdockendorf do you know what could be preventing these variables from passing into sudo? Some other sudo rule or configuration?

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 2:19pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/34 "2022-05-18T14:19:06Z")

</div>

Ok, I got the variable set now, but here’s what I had to do, maybe there’s a config somewhere that’s blocking it from working the normal way.

- edit /etc/sudoers.d/ood and change the NOPASSWD line to NOPASSWD:SETENV:
- modified the ood\_portal.conf file in the http config to do sudo -E

I have a feeling there’s a config elsewhere on the system that would prevent me from needing all that.  
I’m on RedHat 7. There very well could be something in the /etc/sudoers that says don’t preserve environments by default.

Of course after all of that, I don’t know if rancher will like the token, but that’s a whole other issue. I did notice in the rebuilt ~/.kube/config, I have the id-token now, but no refresh-token, should that be in there, or is that normal?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 2:58pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/35 "2022-05-18T14:58:33Z")

</div>

Yea - good to hear we’ve at least got it down to this 1 thing.

> [@groucho64738](#):
>
> I have the id-token now, but no refresh-token, should that be in there, or is that normal?

I think there’s an Apache configuration to request refresh tokens that you probably have to enable. Also there could be something on the Keycloak side to enable it too.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 3:01pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/36 "2022-05-18T15:01:03Z")

</div>

I have a strong feeling after all this Rancher won’t allow this to work because its authentication mechanism for a kube config does not seem to be happy with OIDC tokens and still requires ‘authentication’.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 3:04pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/37 "2022-05-18T15:04:33Z")

</div>

2 things - there is a note in the docs about Audiences. You’ve got to set that up if you want to use OOD tokens directly on k8s.

OR - instead of just grabbing the token from Apache - you could do a OIDC key exchange, then write the correct kube config.

---

<div class="post-metadata">

**Author:** ![groucho64738](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/groucho64738/32/721_2.png) [@groucho64738](https://discourse.openondemand.org/u/groucho64738)\
**Post date:** [May 18, 2022, 3:13pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/38 "2022-05-18T15:13:39Z")

</div>

I did have an audience created that should have been the Rancher client. I’ll keep poking around there.

When I had apache debugging turned on, I did see a refresh token returned, so it looks like keycloak is sending it. I’m not seeing the environment variable OIDC\_REFRESH\_TOKEN being set/passed anywhere though.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 3:31pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/39 "2022-05-18T15:31:17Z")

</div>

Duh, you have to explicitly export any environment variable you want to use. so add it to `pun_pre_hook_exports`

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 18, 2022, 3:32pm UTC](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061/40 "2022-05-18T15:32:07Z")

</div>

And that `duh` was to me not you, sorry if it read like that, I should have recognized that.

[Previous page](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061.md?page=1)

[Next page](https://discourse.openondemand.org/t/ondemand-and-rancher-integration/2061.md?page=3)