# Open OnDemand and AFS

**URL:** <https://discourse.openondemand.org/t/open-ondemand-and-afs/3095>\
**Category:** General Discussion\
**Created:** [November 13, 2023, 2:39pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095 "2023-11-13T14:39:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [November 13, 2023, 2:39pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/1 "2023-11-13T14:39:17Z")

</div>

Wondering if anyone has integrated Open OnDemand with AFS. Trying to figure out authentication since I need a token to access the home directory.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 13, 2023, 2:44pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/2 "2023-11-13T14:44:29Z")

</div>

I think kerberos is a shortcoming on our side. I seem to recall a lot of discussion for the same if you search Kerberos on this site - but I don’t know if there was any resolution.

We can help as best we can, but since we don’t use it, we can only help guide. I don’t think we have a good test system for the same.

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [November 21, 2023, 2:31pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/3 "2023-11-21T14:31:00Z")

</div>

Kerberos related problem. KRB5CCNAME is not available in PUN env variables. KRB5CCNAME points to a cached ticket file. Which is the best way to pick REMOTE\_USER env variable to automatically pass to the path of this local cached file?

Thanks,  
Luiss

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [November 21, 2023, 4:06pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/4 "2023-11-21T16:06:21Z")

</div>

Te be more precise, if I edit /etc/ood/profile with the following:  
export KRB5CCNAME=FILE:/var/run/httpd/clientcaches/$REMOTE\_USER@ENEA.IT  
and then I export in /etc/ood/config/nginx\_stage.yml:  
pun\_custom\_env\_declarations:  
- KRB5CCNAME  
In /proc/PID\_lbucci\_passenger\_core/environ I obtain:  
KRB5CCNAME=FILE:/var/run/httpd/clientcaches/root@ENEA.IT  
instead of  
KRB5CCNAME=FILE:/var/run/httpd/clientcaches/lbucci@ENEA.IT  
How can I obtain the effective REMOTE\_USER ‘lbucci’ instead of ‘root’ in the path?

Thanks,  
Luiss

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 21, 2023, 6:39pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/5 "2023-11-21T18:39:47Z")

</div>

I’m having to dig around for you. I think this is all happening before we boot the PUN. At that point (at a later point in time) `USER` and `LOGNAME` are set correctly, but again, that may be later in the process.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 21, 2023, 8:17pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/6 "2023-11-21T20:17:58Z")

</div>

Looks like you can pull the value from the `SUDO_COMMAND`. the `SUDO_COMMAND` environment variable is available when `/etc/ood/profile` is sourced, so you’re half the way there. You can just do something similar to this (I left SUDO\_COMMAND populated here, but obviously it’ll be dynamic and set for you when you source `/etc/ood/profile`)

```auto
SUDO_COMMAND="/opt/ood/nginx_stage/sbin/nginx_stage pun -u jeff -a http%3a%2f%2flocalhost%3a8080%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri"

UNPRIV_USER=$(echo $SUDO_COMMAND | sed -r 's#.+\-u (.+) \-a.+#\1#g')

```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 21, 2023, 8:19pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/7 "2023-11-21T20:19:13Z")

</div>

Note that I may not be dealing with newlines correctly there in `sed` and `echo`, so that’s a pitfall you’ll have to look out for.

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [November 22, 2023, 9:38am UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/8 "2023-11-22T09:38:12Z")

</div>

Dear Jeff,  
the point is that I need to populate the path to the cached ticket /var/run/httpd/clientcaches/lbucci@ENEA.IT  
with the remote\_user (lbucci), without hardcoding it with the option ‘-u lbucci’ in the SUDO\_COMMAND but with something like ‘-u $REMOTE\_USER’.  
Is it possible? Should I follow a different strategy?

Thanks a lot for your support.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 22, 2023, 2:04pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/9 "2023-11-22T14:04:17Z")

</div>

> [@luissbucci](#):
>
> with the remote\_user (lbucci), without hardcoding it with the option ‘-u lbucci’ in the SUDO\_COMMAND but with something like ‘-u $REMOTE\_USER’.

That’s not hard coded, though it was in that example. the `SUDO_COMMAND` that boots the per user nginx will always have a variable `-u <user>`. I left if there in the example just because it was an example for demonstration. (I inspected the available environment by doing `env > /tmp/somefile` in `/etc/ood/profile`)

When the system sources `/etc/ood/profile` - the `SUDO_COMMAND` is already set by the system (so you don’t have to initialize it, set it or reset it), you just need to extract the user from that already set/dynamic environment variable.

As you can see while I was testing in a container, the command was `-u jeff` (my local container user).

If you tried it with a different user, the nginx\_stage command would be different, that’s why we need `sed` or similar to extract it.

So your /etc/ood/profile could looks something like

```auto
#!/bin/bash

# SUDO_COMMAND is something like '... nginx_stage -u ktrout -a ...'

UNPRIV_USER=$(echo $SUDO_COMMAND | sed -r 's#.+\-u (.+) \-a.+#\1#g')
export KRB5CCNAME="FILE:/var/run/httpd/clientcaches/$UNPRIV_USER@ENEA.IT"

```

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [November 22, 2023, 3:03pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/10 "2023-11-22T15:03:47Z")

</div>

Thanks a lot! Now I have the kerberos Ticket!!!  
My next step is to try to forward the ticket to logon on the nodes of our cluster.  
If I succeed in this operation I will inform you.

Best Regards,  
Luiss

---

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [November 22, 2023, 3:20pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/11 "2023-11-22T15:20:22Z")

</div>

Please share! I’m interested in seeing how you’ve configured everything!

---

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [November 30, 2023, 7:51pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/12 "2023-11-30T19:51:38Z")

</div>

Luigi, would you mind sharing what the auth section looks like in your ood\_portal.yml file?  
I appear to be authenticating but not getting a ticket. I’m getting permission denied (Errno::EACCES) when trying to create the ondemand directory. I’m missing something obvious.

---

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [November 30, 2023, 8:37pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/13 "2023-11-30T20:37:18Z")

</div>

Jeff, could you give a little background here? So you’ve created a file called profile in /etc/ood. Executable by root/apache? Does this have to be defined somewhere? Is this different than the Configuration Profiles in the documentation?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 30, 2023, 8:49pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/14 "2023-11-30T20:49:46Z")

</div>

> [@dbarstis](#):
>
> Executable by root/apache? Does this have to be defined somewhere?

Not executable, 644 or even maybe 600. No need to define it somewhere, the system will source it if it exists just before starting the PUN.

> <https://github.com/OSC/ondemand/blob/251f5b8f1bf0239cc2d5173619851e2426eb5a1f/nginx_stage/sbin/nginx_stage#L18-L23>

> [@dbarstis](#):
>
> Is this different than the Configuration Profiles in the documentation?

Yes very different - I think it was shadowing `/etc/profile.d` files for shells. `/etc/ood/profile` is all about setting environment variables for the entire PUN.

[https://osc.github.io/ood-documentation/latest/reference/pun-environment.html](https://osc.github.io/ood-documentation/latest/reference/pun-environment.html)

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [December 4, 2023, 4:19pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/15 "2023-12-04T16:19:28Z")

</div>

Dear David,  
to get the kerberos ticket and the AFS token I use the mod\_auth\_gssapi and the mod\_waklog modules. In this way I am able to access the AFS-filesystem and also propagate the ticket/token to our compute login nodes.  
The Apache directives I use are the following:  
AuthType GSSAPI  
AuthName “GSSAPI Login”  
GssapiBasicAuth On  
GssapiCredStore keytab:/etc/ood/http.keytab  
GssapiLocalName On  
GssapiCredStore ccache:FILE:/var/run/httpd/krb5ccache  
GssapiDelegCcacheDir /var/run/httpd/clientcaches  
GssapiDelegCcachePerms mode:4660  
Require valid-user  
WaklogEnabled On  
WaklogUseUserTokens On

I hope this help you.

Best Regards  
Luigi

---

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [December 4, 2023, 4:25pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/16 "2023-12-04T16:25:07Z")

</div>

Thank you Luigi. Am I correct in that you do not use mod\_session?

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [December 4, 2023, 4:32pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/17 "2023-12-04T16:32:03Z")

</div>

You’re right, I don’t use it.

---

<div class="post-metadata">

**Author:** ![dbarstis](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dbarstis](https://discourse.openondemand.org/u/dbarstis)\
**Post date:** [December 4, 2023, 10:05pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/18 "2023-12-04T22:05:53Z")

</div>

Sorry to keep asking questions. Did you have to set WaklogAFSCell? If so, where

---

<div class="post-metadata">

**Author:** ![luissbucci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/luissbucci/32/1986_2.png) [@luissbucci](https://discourse.openondemand.org/u/luissbucci)\
**Post date:** [December 5, 2023, 9:15am UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/19 "2023-12-05T09:15:58Z")

</div>

I find a working solution is to manually enter these settings at the top of the ood-portal.conf file.

LoadModule auth\_gssapi\_module /opt/rh/httpd/root/etc/httpd/modules/mod\_auth\_gssapi.so  
LoadModule waklog\_module /opt/rh/httpd/root/etc/httpd/modules/mod\_waklog.so

WaklogAfsCell enea.it

I am sure that Jeff has o more elegant way to include these settings via ood\_portal.yml.

Luigi

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [December 5, 2023, 2:02pm UTC](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095/20 "2023-12-05T14:02:02Z")

</div>

> [@luissbucci](#):
>
> I am sure that Jeff has o more elegant way to include these settings via ood\_portal.yml.

The workaround right now is to add the lines to the `ssl` config. Even though they’re not a part of the ssl config, they’re templated in the right place.

[Next page](https://discourse.openondemand.org/t/open-ondemand-and-afs/3095.md?page=2)