OIDC (Keycloak) and kubeconfig generation

Get Help
1

We’re trying to set up OOD with Keycloak for authentication and Openshift for the HPC cluster.

We’ve already set up Keycloak according to the official documentation, but adapted for Ubuntu 22.04. Openshift is also configured and we’ve set up an audience on Keycloak.

The login node is connected to the internal domain which sits behind Keycloak, thus allowing users to log in without manually creating a system user first. We use the provided hook scripts to configure K8s and we’ve added an additional script to create the user’s home directory.

Everything I’ve listed so far works, users can log in and see their home directory. However, the ~/.kube/config file isn’t generated properly. According to the documentation, we would expect to find the file populated with the relevant OIDC user configuration, with the context and cluster added when accessing the dashboard. However, we’ve noticed the access/id token and refresh token are never passed to the pre-hook.sh script or we’re not sure exactly which environment variables contain these tokens (we’ve tried all mentioned in the OOD documentation and Apache module’s example config).

Here is the portal config (important OIDC settings are configured in the /etc/apache2/mods-enabled/auth_openidc.conf file, as directed in the official tutorial):

  • /etc/ood/config/ood_portal.yml
---
#
# Portal configuration
#

# The address and port to listen for connections on
# Example:
#     listen_addr_port: 80
# Default: null (don't add any more listen directives)
#listen_addr_port: 80

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
#servername: null
servername: open-ondemand.domain.com

# The server aliases used for the name-based Virtual Host
# Example:
#     server_aliases:
#       - foo.example.com
server_aliases:
  #  - open-ondemand.int.domain.com 
  - open-ondemand.domain.com
# The server name used for rewrites
# Example:
#     proxy_server: 'proxy.example.com'
# Default: The value of servername
#proxy_server: null

# The port specification for the Virtual Host
# Example:
#     port: 8080
# Default: null (use default port 80 or 443 if SSL enabled)
port: 443

# List of SSL Apache directives
# Example:
ssl:
       - 'SSLCertificateFile "/etc/ssl/certs/chain.crt"'
       - 'SSLCertificateKeyFile "/etc/ssl/private/domain.key"'
# Default: null (no SSL support)
#ssl: true

# Disable logging entirely. With this you can have custom log directives
# in a seperate configuration file.
# Example:
#     disable_logs: true
# Default: false (logs are enabled)
#disable_logs: false

# Root directory of log files (can be relative ServerRoot)
# Example:
#     logroot: '/path/to/my/logs'
# Default: 'logs' (this is relative to ServerRoot)
#logroot: 'logs'

# Error log filename
# Example:
#     errorlog: 'error.log'
# Default: 'error.log' (If 'servername' and 'ssl' options are defined 
# the default value will be <servername>_error_ssl.log)
#errorlog: 'error.log'

# Access log filename
# Example:
#     accesslog: 'access.log'
# Default: 'access.log' (If 'servername' and 'ssl' options are defined 
# the default value will be <servername>_access_ssl.log)
#accesslog: 'access.log'

# Apache access log format (Don't specify log nickname see: http://httpd.apache.org/docs/current/mod/mod_log_config.html#transferlog)
# Example:
#     logformat: '"%v %h \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{SSL_PROTOCOL}x %T"'
# Default: Apache combined format
#logformat: Apache combine format

# Should RewriteEngine be used
# Example:
#     use_rewrites: false
# Default: true
#use_rewrites: true

# Specify the host to redirect to when redirecting from port 80
# Example:
#     http_redirect_host: my.proxy.host
# Default: '%{HTTP_HOST}'
#http_redirect_host: '%{HTTP_HOST}'

# Should Maintenance Rewrite rules be added
# Example:
#   use_maintenance: false
# Default: true
#use_maintenance: true

# List of IPs to allowed when maintenance is enabled
# Example:
#   maintenance_ip_allowlist:
#     - 192.168.0..*
#     - 192.168.1..*
# Default: [] (no IPs allowed)
#maintenance_ip_allowlist: []

# Set Header Content-Security-Policy frame-ancestors.
# Example:
#   security_csp_frame_ancestors: https://ondemand.osc.edu
# Example to disable setting:
#   security_csp_frame_ancestors: false
# Default: based on servername and ssl settings
#security_csp_frame_ancestors:

# Set Header Strict-Transport-Security to help enforce SSL
# Example:
#   security_strict_transport: false
# Default: true when ssl is enabled, false otherwise
#security_strict_transport: false

# Root directory of the Lua handler code
# Example:
#     lua_root: '/path/to/lua/handlers'
# Default : '/opt/ood/mod_ood_proxy/lib' (default install directory of mod_ood_proxy)
#lua_root: '/opt/ood/mod_ood_proxy/lib'

# Verbosity of the Lua module logging
# (see https://httpd.apache.org/docs/2.4/mod/core.html#loglevel)
# Example:
#     lua_log_level: 'warn'
# Default: 'info' (get verbose logs)
lua_log_level: 'info'

# Lua regular expression used to map authenticated-user to system-user
# This configuration is ignored if user_map_cmd is defined
# Example:
#     user_map_match: '^([^@]+)@.*$'
# Default: '.*'
user_map_match: '.*'

# System command used to map authenticated-user to system-user
# This option takes precedence over user_map_match
# Example:
#     user_map_cmd: '/usr/local/bin/ondemand-usermap'
# Default: null (use user_map_match)
#user_map_cmd: null

# Use an alternative CGI environment variable instead of REMOTE_USER for
# determining the authenticated-user fed to the mapping script
# Example:
#     user_env: 'OIDC_CLAIM_preferred_username'
# Default: null (use REMOTE_USER)
#user_env: null

# Redirect user to the following URI if fail to map there authenticated-user to
# a system-user
# Example:
#     map_fail_uri: '/register'
# Default: null (don't redirect, just display error message)
#map_fail_uri: null

# System command used to run the `nginx_stage` script with sudo privileges
# Example:
#     pun_stage_cmd: 'sudo /path/to/nginx_stage'
# Default: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage' (don't forget sudo)
#pun_stage_cmd: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage'

# List of Apache authentication directives
# NB: Be sure the appropriate Apache module is installed for this
# Default: (see below, uses OIDC auth with Dex)
auth:
  - 'AuthType openid-connect'
  - 'Require valid-user'

pam:
  service: login 
# List of custom apache directives to apply to the entire vhost.
# Note this is an array of strings.
# Example:
#     custom_vhost_directives:
#       - 'SetEnv SPECIAL_ENV_VAR custom'
# Default: [] (no custom directives)
#custom_vhost_directives: []

# List of custom apache directives to apply to the Locations.
# Note this is an array of strings.
# Example:
#     custom_location_directives:
#       - 'SetEnv SPECIAL_ENV_VAR custom'
# Default: [] (no custom directives)
#custom_location_directives: []

# Redirect user to the following URI when accessing root URI
# Example:
#     root_uri: '/my_uri'
#     # https://www.example.com/ => https://www.example.com/my_uri
# Default: '/pun/sys/dashboard' (default location of the OOD Dashboard app)
#root_uri: '/pun/sys/dashboard'

# Track server-side analytics with a Google Analytics account and property
# (see https://github.com/OSC/mod_ood_proxy/blob/master/lib/analytics.lua for
# information on how to setup the GA property)
# Example:
#     analytics:
#       url: 'http://www.google-analytics.com/collect'
#       id: 'UA-79331310-4'
# Default: null (do not track)
#analytics: null

#
# Publicly available assets
#

# Public sub-uri (available to public with no authentication)
# Example:
#     public_uri: '/assets'
# Default: '/public'
#public_uri: '/public'

# Root directory that serves the public sub-uri (be careful, everything under
# here is open to the public)
# Example:
#     public_root: '/path/to/public/assets'
# Default: '/var/www/ood/public'
#public_root: '/var/www/ood/public'

#
# Logout redirect helper
#

# Logout sub-uri
# Example
#     logout_uri: '/log_me_out'
# NB: If you change this, then modify the Dashboard app with the new sub-uri
# Default: '/logout' (the Dashboard app is by default going to expect this)
#logout_uri: '/logout'

# Redirect user to the following URI when accessing logout URI
# Example:
#     logout_redirect: '/oidc?logout=https%3A%2F%2Fwww.example.com'
# Default: '/pun/sys/dashboard/logout' (the Dashboard app provides a simple
# HTML page explaining logout to the user)
#logout_redirect: '/pun/sys/dashboard/logout'
logout_redirect: '/oidc?logout=https://open-ondemand.domain.com/pun/sys/dashboard/logout'

#
# Reverse proxy to backend nodes
#

# Regular expression used for whitelisting allowed hostnames of nodes
# Example:
#     host_regex: '[\w.-]+\.example\.com'
# Default: '[^/]+' (allow reverse proxying to all hosts, this allows external
# hosts as well)
#host_regex: '[^/]+'

# Sub-uri used to reverse proxy to backend web server running on node that
# knows the full URI path
# Example:
#     node_uri: '/node'
# Default: null (disable this feature)
#node_uri: null

# Sub-uri used to reverse proxy to backend web server running on node that
# ONLY uses *relative* URI paths
# Example:
#     rnode_uri: '/rnode'
# Default: null (disable this feature)
#rnode_uri: null

#
# Per-user NGINX Passenger apps
#

# Sub-uri used to control PUN processes
# Example:
#     nginx_uri: '/my_pun_controller'
# Default: '/nginx'
#nginx_uri: '/nginx'

# Sub-uri used to access the PUN processes
# Example:
#     pun_uri: '/my_pun_apps'
# Default: '/pun'
#pun_uri: '/pun'

# Root directory that contains the PUN Unix sockets that the proxy uses to
# connect to
# Example:
#     pun_socket_root: '/path/to/pun/sockets'
# Default: '/var/run/ondemand-nginx' (default location set in nginx_stage)
#pun_socket_root: '/var/run/ondemand-nginx'

# Number of times the proxy attempts to connect to the PUN Unix socket before
# giving up and displaying an error to the user
# Example:
#     pun_max_retries: 25
# Default: 5 (only try 5 times)
#pun_max_retries: 5

# The PUN pre hook command to execute as root
#
# Example:
#    pun_pre_hook_root_cmd: '/opt/hpc-site/ood_pun_prehook'
# Default: null (do not run any PUN pre hook as root)
pun_pre_hook_root_cmd: /opt/ood/hooks/pre-hook.sh

# Comma separated list of environment variables to pass from the apache context
# into the PUN pre hook. Defaults to null so nothing is exported.
#
# Example:
#    pun_pre_hook_exports: 'OIDC_ACCESS_TOKEN,OIDC_CLAIM_EMAIL'
# Default: null (pass nothing)
pun_pre_hook_exports: 'OIDC_id_token,OIDC_refresh_token'

#
# Support for OpenID Connect
#

# Sub-uri used by mod_auth_openidc for authentication
# Example:
#     oidc_uri: '/oidc'
# Default: null (disable OpenID Connect support)
#oidc_uri: null
oidc_uri: '/oidc'

# Sub-uri user is redirected to if they are not authenticated. This is used to
# *discover* what ID provider the user will login through.
# Example:
#     oidc_discover_uri: '/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_uri: null

# Root directory on the filesystem that serves the HTML code used to display
# the discovery page
# Example:
#     oidc_discover_root: '/var/www/ood/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_root: null

#
# Support for registering unmapped users
#
# (Not necessary if using regular expressions for mapping users)
#

# Sub-uri user is redirected to if unable to map authenticated-user to
# system-user
# Example:
#     register_uri: '/register'
# Default: null (display error to user if mapping fails)
#register_uri: null

# Root directory on the filesystem that serves the HTML code used to register
# an unmapped user
# Example:
#     register_root: '/var/www/ood/register'
# Default: null (display error to user if mapping fails)
#register_root: null

# OIDC metadata URL
# Example:
#     oidc_provider_metadata_url: https://example.com:5554/.well-known/openid-configuration
# Default: null (value auto-generated if using Dex)
#oidc_provider_metadata_url: null

# OIDC client ID
# Example:
#     oidc_client_id: ondemand.example.com
# Default: null (value auto-generated if using Dex)
#oidc_client_id: null

# OIDC client secret
# Example:
#     oidc_client_secret: 334389048b872a533002b34d73f8c29fd09efc50
# Default: null (value auto-generated if using Dex)
#oidc_client_secret: null

# OIDC remote user claim. This is the claim that populates REMOTE_USER
# Example:
#     oidc_remote_user_claim: preferred_username
# Default: preferred_username
#oidc_remote_user_claim: preferred_username

# OIDC scopes
# Example:
#     oidc_scope: "openid profile email groups"
# Default: "openid profile email"
#oidc_scope: "openid profile email"

# OIDC crypto passphrase
# Example:
#   oidc_crypto_passphrase: "f1d2d2f924e986ac86fdf7b36c94bcdf32beec15"
# Default: SHA1 sum of servername
#oidc_crypto_passphrase: ~

# OIDC session inactivity timeout
# Example:
#     oidc_session_inactivity_timeout: 28800
# Default: 28800
#oidc_session_inactivity_timeout: 28800

# OIDC session max duration
# Example:
#     oidc_session_max_duration: 28800
# Default: 28800
#oidc_session_max_duration: 28800

# OIDC max number of state cookies and if to automatically clean old cookies
# Example:
#     oidc_state_max_number_of_cookies: "10 true"
# Default: "10 true"
#oidc_state_max_number_of_cookies: "10 true"

# OIDC Enable SameSite cookie
# When ssl is defined this defaults to 'Off'
# When ssl is not defined this defaults to 'On'
# Example:
#     oidc_cookie_same_site: 'Off'
# Default: 'On'
#oidc_cookie_same_site: 'On'

# Additional OIDC settings as key-value pairs
# Example:
#     oidc_settings:
#       OIDCPassIDTokenAs: serialized
#       OIDCPassRefreshToken: On
# Default: {} (empty hash)
oidc_settings:
  OIDCPassIDTokenAs: "serialized"
  OIDCPassRefreshToken: "On"
  OIDCPassClaimsAs: "environment"
  OIDCStripCookies: "mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1"
  OIDCResponseType: "code"

# The Dex URI behind Apache reverse proxy
# Set to false or null to disable Dex behing the Apache reverse proxy
# Example:
#   dex_uri: /auth
# Default: /dex
#dex_uri: /dex

# Dex configurations, values inside the "dex" structure are directly used to configure Dex
# If the value for "dex" key is false or null, Dex support is disabled
# Dex support will auto-enable if ondemand-dex package is installed
#dex:
  # Default based on if ssl key for ood-portal-generator is defined
#  ssl: false
  # Only used if SSL is disabled
#  http_port: "5556"
  # Only used if SSL is enabled
#  https_port: "5554"
  # tls_cert and tls_key take OnDemand configured values for ssl and copy keys to /etc/ood/dex maintaining file names
#  tls_cert: null
#  tls_key: null
#  storage_file: /etc/ood/dex/dex.db
#  grpc: null
#  expiry: null
  # Client ID, defaults to servername or FQDN
#  client_id: null
#  client_name: OnDemand
  # Client secret, value auto generated
  # A value that is a filesystem path can be used to store secret in a file
#  client_secret: /etc/ood/dex/ondemand.secret
   # The OnDemand redirectURI is auto-generated, this option allows adding additional URIs
#  client_redirect_uris: []
  # Additional Dex OIDC clients to configure
#  static_clients: []
  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
#  connectors:
#    - type: ldap
#      id: ldap
#      name: LDAP
#      config:
#        host: openldap.my_center.edu:636
#        insecureSkipVerify: false
#        bindDN: cn=admin,dc=example,dc=org
#        bindPW: admin
#        userSearch:
#          baseDN: ou=People,dc=example,dc=org
#          filter: "(objectClass=posixAccount)"
#          username: uid
#          idAttr: uid
#          emailAttr: mail
#          nameAttr: gecos
#          preferredUsernameAttr: uid
#        groupSearch:
#          baseDN: ou=Groups,dc=example,dc=org
#          filter: "(objectClass=posixGroup)"
#          userMatchers:
#            - userAttr: DN
#              groupAttr: member
#          nameAttr: cn
#  frontend:
#    theme: ondemand
#    dir: /usr/share/ondemand-dex/web

We’d appreciate any help

2

Hi and welcome!

Hmmmm… With these 2 settings, you should be good. Are you sure these settings made it to the .conf file that’s being generated? Bouncing httpd should create a new .conf file. I’d wonder if there’s some issue in generating this file with all your settings.

3

There was an issue with our pre-hook script. It was setting K8s user credentials before the home directory was created. We also had to add the “OOD_“ prefix to all OIDC environment variables used in the script.

Now we get a complete kubeconfig, but Openshift doesn’t accept it. Does this look alright?

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/ssl/certs/api-ocp-int-sedamit-hr.pem
    server: https://api.ocp.int.sedamit.hr:6443
  name: ocp-lab
contexts:
- context:
    cluster: ocp-lab
    namespace: user-pgrguric
    user: pgrguric
  name: ocp-lab
current-context: ""
kind: Config
preferences: {}
users:
- name: pgrguric
  user:
    auth-provider:
      config:
        ...

Openshift doesn’t work for now, but we’re updating our cluster to see if it’s related to its version, though we’re unsure if there is any additional configuration necessary to make this work. We’ve configured an audience in Keycloak according to the docs. We’ve checked the ID token that ends up in the kubeconfig and it shows both the OOD and OCP clients under audiences.

Additionally, how does this work when there are multiple K8s clusters? We can configure the default context if there is one, but what if there are multiple and some jobs need to run on different clusters?