OnDemand and Rancher integration

General Discussion
1

If you don’t mind, I’m going to create a topic here to discuss the integration of Open Ondemand and Rancher. The theory is that Ondemand doesn’t care about the specific K8S installation, but I have a feeling there will be some things to work through to get it working with the authentication mechanisms that Rancher puts in place which may be different. The topic is meant to be a placeholder for comments/issues as I work through it, and maybe if others have already done this they can put in their experiences.

Thanks.

2

My Rancher installation is using our sites Active Directory for authentication, as is our OnDemand instance (through PAM). We’ve created our cluster and created a Project inside of that cluster called ‘ondemand’ so that any namespaces we create can hopefully be all put in there.

I’ve downloaded the kubeconfig for this cluster, and it has lots of certificate info and info about the systems in the cluster, but I think the user info that is in the yaml file is specific to me, but we’ll find out.

This is my starting point, and haven’t gotten further with it at the moment. I’m still working through the OnDemand documentation to see how I’ll have to adapt it.

Right off the bat, it looks like the Rancher concept of projects isn’t going to fly with this. According to the Rancher documentation Rancher Docs: Projects and Kubernetes Namespaces with Rancher, projects are a Rancher thing, and kubectl won’t recognize them as such. This may or may not be a problem for us in the long run, but maybe we do something like create a specific cluster just for ondemand, or accept that there will be namespaces in the ‘default’ project in Rancher. I might be ok with this.

3

Thanks for the post!

4

No problem. Hopefully it’ll be a successful experiment, but if not, at least we tried.

5

Rancher has its own available cli which can interface with kubectl and handle projects, but I have a feeling that’ll be a little hacky to get going, so to start, I’ll just set up a single VM as a K8s cluster for ondemand to eliminate this is a thing for right now.

1 Like
6

Ok. I’m taking a break for right now and think I’m almost at a wall on configuration. Here’s what I’ve done so far.

In Rancher I downloaded the kube config file for my account so that I can work on the CLI at this stage. I transferred it to the ondemand web server.

I grabbed the token as described in the ondemand documentation.
At this stage, rather than generate a new yaml file for authentication, I modified the one I downloaded from Rancher to change these settings:

  • removed the main rancher cluster information from the file
  • used the second cluster that was listed (which points directly to the K8s node)
  • removed the context that referred to the main rancher cluster
  • set the user in the context that refers to the kubernetes node to ondemand@ondemand
  • changed the user info to name: ondemand@ondemand, and pasted the token I gathered from before

I was able to run the kubectl cluster-info command.

At this point I’m stopped. One question in this kind of config though, if I have a multi-node cluster, this would only communicate to one of the nodes, how would we do this so we could communicate to all of the nodes? I guess I could put multiple contexts in for each node, but OOD wouldn’t use them.

Also, the next step of what I’m supposed to do for OpenOndemand is a little confusing. I guess I have to do something with the hooks, but if the users themselves are not authenticating to Rancher, how do the tokens for those users get created? We’re not using OIDC so I am not quite sure of what I need to do here to modify it for this environment.

I can provide redacted configs as needed if it helps.

7

I’m thinking this authentication piece is a big road block. We have no real way to get an api key out of Rancher without going into the GUI as far as I can tell (if someone know of a different way, that might be good to know), which would really diminish the ease of use of this or make it really impossible to manage. Is it true in the docs that OIDC contains the api key and is then places in the users .kube/config file? If that’s the case, I don’t think this setup will ever work with Rancher in the mix.

8

This command gets me a bearer token from Rancher:

curl -k -s -H ‘content-type: application/json’ ‘https://rancherserver.example.org/v3-public/localProviders/local?action=login’ --data-binary ‘{“username”:“myuser”,“password”:“mypassword”}’ | jq -r .token

I think I found the LDAP version of the same command, so I’m sure I can adapt it to AD. This token can be used with the Rancher CLI to log in, I’m now looking for a rancher cli command to generate a kube config file. The Rancher CLI saves the config info in $HOME/.rancher/cli2.json and the info in there has essentially the same content as the kube config file would have. You can run ‘rancher kubectl’ for any of the commands you would normally do, so maybe there’s a way to hook that directly into OOD instead of kubectl directly, but that might not be feasible.

The issue here is the obvious password on the command-line kind of thing, and can OOD provide user/pass to the curl statement from its own authentication mechanisms.

9

Yes, but it’s opt in and it’s because we setup our cluster to use OIDC as well. Looks like Rancher can use OIDC & Keycloak. What type of authentication are you using for Rancher?

You can setup the users ~/.kube/config in an initializer that runs in the user context (not privileged), though we never know you’re password - so I don’t know how you could pass that through from the user to Open OnDemand.

Apache has access to the OIDC token, so we directly use it to create a kube config. I’d be hesitant to store passwords.

Pull requests welcome! But as it stands we use kubectl to submit requests to create Pods, Services, and Secrets directly. So it’s really built to directly talk to kuberentes - unless Rancher can also accept essentially exact same request kubernetes can.

10

We’re using AD for our authentication. I’ve not set up keycloak before, but I’m looking at it now, maybe that would be a way to do all of this. It might take me awhile to get there since we have people using Rancher actively at the moment.

11

I have a new test environment with Keycloak running, and a copy of my ondemand environment on another system which authenticates against the Keycloak server and a test Rancher installation which authenticates to Keycloak as well, we’ll see how this part goes.

12

I’m trying to work through getting the jupyter app to run. I’m getting this error though: error: error reading input for Password and am a little stuck as to what to do about it. I’m following through Add a Jupyter App on a Kubernetes Cluster — Open OnDemand 2.0.20 documentation and am not changing much in the config to keep it simple, but this error keeps appearing when I try to launch.

13

kubectl is prompting you. Likely because the user being used in your ~/.kube/config doesn’t exist.

If you have OIDC then you should be able to use the pun_root_pre_hook and set the kube config.

I’d

  • check the contents of your ~/.kube/config. Back it up and remove it as needed
  • Check your /var/log/ondemand-nginx/$USER/error.log logs for the actual kubectl command you’re issuing. You can see the context and namespace it’s trying to use here.
  • Given the steps above you’ll know (a) what you’re doing to get setup and (b) how OOD is issuing commands. Something must be off with the username. Note that we can add prefixes to usernames (for validation purposes) so that may be a spot you’re running into on (some piece - like bootstrapping - is using a prefix and another - like kubectl - is not)
  • I don’t know how to setup Keycloak with domains, but I do know our documentation as information on the same. Which is to say Keycloak to work in this manner isn’t out of the box. Our documentation should say what you’ll need to do to Keycloak, so I’d be sure it’s setup right.

Lastly - I’d suggest get the pun_root_pre_hook working first without having to fool with an OOD app. That is, login to run the pre hook - then in a shell session on the web node, issue kubectl commands to debug what you have and what you’d expect.

14

Thanks. That’ll give me some things to look at. I looked at my .kube/config and it has some of the right info in it, but no token of any kind for my user. Looking at the error log I see the kubectl command running, but didn’t see what was after the ‘create’, regardless, I just ran a get pods command using the kubeconfig that was generated and it’s asking for entering the Username/password.

15

I’ve stepped through the prehook things, and I’m almost there, the namespace is configured in Rancher, so that’s good. The prehook is running and is going to the /opt/ood/hooks/k8s-bootstrap scripts. It seems like the $OOD_OIDC_REFRESH_TOKEN and $OOD_OIDC_ACCESS_TOKEN isn’t being set.

I have OIDC_ACCESS_TOKEN configured as one of the pun_pre_hook_exports configured in ood_portal.yaml, and it is in the generated apache config, but that doesn’t seem to be passing through.

16

Because of sudo rules, anything you export has OOD_ prefixed to it in the script.

The docs say the same.
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html

You may pass in environment variables from apache to this command, though they are prefixed with OOD_. For example if you configure this to pass OIDC_ACCESS_TOKEN to the pre hook command, you can read the variable as OOD_OIDC_ACCESS_TOKEN.
17

Right, I saw that, but those variables are not being set. I configured OIDC_ACCESS_TOKEN as a variable to be exported, but it is not available to the prehook command. The prehook command I have is the same as described in the documentation Kuberenetes — Open OnDemand 2.0.20 documentation. I modified this script to dump the contents of the entire environment and there are no OOD variables being set at all.

I turned on apache debugging, and I do see that an access and refresh token are being offered up by keycloak, so that is at least functional here.

Also in the generated ood_portal.conf file for apache, this is set: SetEnv OOD_PUN_PRE_HOOK_EXPORTS “OIDC_ACCESS_TOKEN”

or am I not reading the docs correctly?

18

Got it. Seems like apache isn’t setting the environment variables (even if it’s getting the response from Keycloak).

What are your OIDC related settings in ood_portal.yml? You may need to specify this as environment.

19

In looking at the debug logs I do see apache claiming that it’s setting environment variables (lots of OIDC_CLAIM ones and OIDC_access_token).

My OIDC settings are in auth_openidc.conf (also pretty much copied from the docs):

# Keep sessions alive for 8 hours
OIDCSessionInactivityTimeout 28800
OIDCSessionMaxDuration 28800

# Set REMOTE_USER
OIDCRemoteUserClaim preferred_username

# Don't pass claims to backend servers
OIDCPassClaimsAs environment

# Strip out session cookies before passing to backend
OIDCStripCookies mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1
20

We’re either overlooking something very simple or you’ve found a bug.

Can you try this one test to export OOD_OIDC_ACCESS_TOKEN and FOO.

pun_pre_hook_exports: 'OIDC_ACCESS_TOKEN,FOO'

I want to see if there’s a bug with giving only one value to export.