that’s fine. I’m having lots of duh moments with this.
Looking through the Apache Debug output, it doesn’t look like the auth_oidc configuration is setting an environment variable for refresh token. The oidc_util_http_call response has it in there along with the access token, so it’s definitely receiving it, but it doesn’t seem to generate any internal environment variables that store it.
Ok, I found it already. I need to set OIDCPassRefreshToken on in the auth_openidc.conf file (I don’t know if you want to mention that in the docs or the example file that you have).
yay, new error. Unable to connect to the server: oidc: failed to query metadata endpoint 404 Not Found: "{“error”:"RESTEASY003210: Could not find resource for full path: https://keycloak…
I’m taking a break from this.
Here are some useful mod_auth_openidc values we use at OSC:
OIDCPassClaimsAs environment
OIDCPassIDTokenAs serialized
OIDCPassRefreshToken On
I use Puppet to setup the kubeadm YAML , not sure what the analogue is for Rancher but these are the arguments passed directly to Kubernetes API server:
- "oidc-issuer-url: https://idp.OMIT/auth/realms/osc"
- 'oidc-client-id: kubernetes'
- 'oidc-username-claim: preferred_username'
- 'oidc-username-prefix: "-"'
- 'oidc-groups-claim: groups'
One thing we do that is unique to make this all work is we use Keycloak’s ability to set OIDC audience so the token for OnDemand is also valid for Kubernetes, since at our site those are two different clients in Keycloak. Configure Kuberenetes — Open OnDemand 2.0.20 documentation
So if you use AD you could setup Keycloak to pull everything from AD, we use OpenLDAP but Keycloak has good support for AD. Then users can still authentication with Rancher using AD credentials but the request will pass through Keycloak using OIDC.
I think I’m starting to finally see some of what needs to happen. Rancher is a little weird because it’s a frontend to things. Technically, I’m not authenticating against Rancher, I’m authenticating against the Kubernetes cluster specifically, so I need to configure that cluster to specifically use OIDC, regardless of whether or not Rancher is configured to do this…in fact, Rancher doesn’t have really anything to do with it at this stage (I don’t think).
So, what I ended up doing inside of Rancher is edit the cluster config (edit as YAML), and added the following under the services → kube-api section
kube-api:
extra_args:
oidc-client-id: myclient-in-keycloak
oidc-groups-claim: groups
oidc-issuer-url: 'https://keycloak.example.org/realms/OnDemand'
oidc-username-claim: preferred_username
Then rancher restarted the API server and I got some progress, until…I was forbidden to do anything in the cluster. I think that’s just normal permissions that I’ll have to work through with Rancher.
Error from server (Forbidden): error when creating "STDIN": pods is forbidden: User "https://keycloak.example.org/realms/OnDemand#myuser" cannot create resource "pods" in API group "" in the namespace "user-myuser"
Error from server (Forbidden): error when creating "STDIN": services is forbidden: User "https://keycloak.example.org/realms/OnDemand#myuser" cannot create resource "services" in API group "" in the namespace "user-myuser"
Error from server (Forbidden): error when creating "STDIN": configmaps is forbidden: User "https://keycloak.example.org/realms/OnDemand#myuser" cannot create resource "configmaps" in API group "" in the namespace "user-myuser"
There’s one piece here in the bootstrapping that’s relevant. We should have created all those resources and the rolebindings for them. We do this in k8s-bootstrap-ondemand.sh - templating some yaml files with envsubst and applying them.
That said - when I get errors like the one you’re seeing it’s a regular username like johrstrom not an URL to keycloak.
hm…good point. I wonder why that is coming up with the full url#user.
So along with bootstrapping RBAC for OnDemand, if you want to authorize a group or user to be cluster admins can do this (just replace sysstf with your group).
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-sysstf-admin
subjects:
- kind: Group
name: sysstf
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-sysstf-cluster-admin
subjects:
- kind: Group
name: sysstf
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
You also need this applied before the k8-boostrap stuff will work:
Per Configure Kuberenetes — Open OnDemand 2.0.20 documentation
Also need to allow root on OnDemand host to run certain kubectl commands:
Let me see if I can track down the username piece, maybe that’s it. I had followed the directions on the wiki, so I think the bootstrapping should be ok. It’s actually creating the namespace when I log in, so some piece of that is working.
Checking through the steps. The k8s-bootstrap-ondemand.sh identifies my regular username, the set-k8s-creds.sh identifies my regular username.
Looking through the staged root directory files, the job_script_options.json only references my regular username. Same with pod.yml and user_defined_context.json.
At this point I’m looking at the kubernetes server itself as being the problem.
I also decoded the id-token that’s in the ~/.kube/config and the preferred_username in there is my regular username, no references to the full URL#username in there either.
I see you deleted the question, but I’m guessing your issue remains. I did however find these issues by googling found 1 parts of token that could give you some insight.
and
yeah, I thought I was past it, but I’m not. I’m glad you saw it first though because I was just about to repost it.
Seems like they’re going on mostly about oauth2_proxy, which we’re not using (wonder if we should perhaps). Is that something you run on your K8s cluster?
Yea that may be a red herring as they’re using those as a side-car for some apps? Your issue seems to be with the api server directly (not any individual containers).
Looking at the kubernetes configurations for the api server, they seem to be minimal. That is, there’s not much to misconfigure on that side. Which leads me to think it’s a Keycloak misconfiguration with your realm. (OR I’m missing something that Rancher is doing which could absolutely be the case).
We are using the same realm for OnDemand and Kubernetes, though I don’t know much about it. Can you share the error log entries you’re seeing in the API server again? Maybe @tdockendorf can see something I don’t.
I’m trying to keep rancher out of it as much as possible. So, we’re authenticating directly to the kubernetes nodes, and have the settings that we listed way up top in the thread for the apiserver. I’m using the same client name in kubernetes as my ondemand host (to eliminate that as a thing), and they’re all going to the same realm I can log in to ondemand with no problem as my user, but that doesn’t really use the cluster.
The error in the api server is:
time=“2023-01-26T16:44:22Z” level=info msg=“Processing v1Authenticate request…”
time=“2023-01-26T16:44:22Z” level=error msg=“found 1 parts of token”
It’s not really descriptive at all.
In the other post I had put the decrypted contents of one of the tokens. I can do that again here.
{
"alg": "RS256",
"typ": "JWT",
"kid": "BAccuYs7L7SRWMgL_hJvTguKyIHCGQzzeew08lAV-2I"
}
{
"exp": 1674745783,
"iat": 1674745483,
"auth_time": 1674745321,
"jti": "b39d43e3-2037-47cc-bdce-ac1c10eb8179",
"iss": "https://keycloak.example.com/realms/OnDemand",
"aud": [
"myclient.example.com"
],
"sub": "f8f1e286-22eb-441f-893a-02afa5027479",
"typ": "ID",
"azp": "myclient.example.com"
"nonce": "7QKP2DsbVnWBmImTWL6F-1Mzpw1GmDEbPqoR-kpAH9w",
"session_state": "b351e95c-5053-4d38-a76a-2cce6def5a1a",
"at_hash": "B9uxD8B4Hs7iksF3-k899w",
"acr": "0",
"sid": "b351e95c-5053-4d38-a76a-2cce6def5a1a",
"email_verified": true,
"name": "My User",
"preferred_username": "myuser",
"given_name": "My",
"family_name": "User",
"email": "myuser@example.com"
}
OK - I found the error you’re seeing from the Rancher’s kube-api-auth.
For whatever reason you should see a spec JSON object which I don’t see in the response of keycloak.
You may have to open a ticket with the Rancher team and/or look on their repository for what’s going on. You have this wrapper/shim kube-api-auth that is expecting something different that what Keycloak is directly providing.
Some sort of translation must be happening inbetween Keycloak’s response and this error which I’m unable to track down.
What distribution of kubernetes are you running there?
Stock Kubernetes managed by puppet I’m fairly sure.
OSC uses vanilla Kubernetes. I don’t recall exact versions we’ve used with OnDemand and Keycloak but I want to say we started in like 1.19 and are now on 1.24. This is our Keycloak Client config for Kubernetes. The redirect URIs can be ignored, we use same Keycloak client for Kubernetes Dashboard that uses OAuth2 proxy side car for authentication.
{
"id" : "kubernetes",
"clientId" : "kubernetes",
"surrogateAuthRequired" : false,
"enabled" : true,
"alwaysDisplayInConsole" : false,
"clientAuthenticatorType" : "client-secret",
"secret" : "OMIT",
"redirectUris" : [ "https://OMIT/oauth2/callback", "https://OMIT/oauth2/callback" ],
"webOrigins" : [ ],
"notBefore" : 0,
"bearerOnly" : false,
"consentRequired" : false,
"standardFlowEnabled" : true,
"implicitFlowEnabled" : false,
"directAccessGrantsEnabled" : true,
"serviceAccountsEnabled" : false,
"publicClient" : false,
"frontchannelLogout" : false,
"protocol" : "openid-connect",
"attributes" : {
"post.logout.redirect.uris" : "+"
},
"authenticationFlowBindingOverrides" : { },
"fullScopeAllowed" : true,
"nodeReRegistrationTimeout" : -1,
"defaultClientScopes" : [ "web-origins", "offline_access", "roles", "profile", "groups", "email" ],
"optionalClientScopes" : [ ],
"access" : {
"view" : true,
"configure" : true,
"manage" : true
}
}