Internal Server Error After Fresh Install

I have just installed Open OnDemand on a RHEL 7.7 server following the instructions under “Install Software from RPM”. I am on section 4 “Start Services”. I’m on the last step where I’m using a browser to try and access the Open OnDemand dashboard. Whenever I try to navigate to the server in a web browser, I get an HTTP 500 error. I see the following error in the apache logs:

oidc_authenticate_user: the URL hostname (host1.internal.edu) of the configured OIDCRedirectURI does not match the URL hostname of the URL being accessed (apps.cluster.edu): the "state" and "session" cookies will not be shared between the two!

host1.internal.edu is the internal domain name of the machine (changed for security)
apps.cluster.edu is the external FQDN that users will use to access Open OnDemand (changed for security).
I’m not sure how to resolve this issue. Any assistance would be greatly appreciated.

Hi and welcome!

Looks like you need to specify the oidc_provider_metadata_url (I believe we used the FQDN by default, which is what you have).

Here’s documentation on other OIDC configs you can set in ood_portal.yml.
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html#configure-openid-connect

Jeff,

Thanks! I set that value in ood_portal.yml. I then ran the following:
/opt/ood/ood-portal-generator/sbin/update_ood_portal
systemctl restart httpd24-httpd.service httpd24-htcacheclean.service
systemctl restart ondemand-dex.service

However, the exact same error still persists. I’m trying to set up LDAP authentication.

Got it. Sorry I misread that too and jumped to something. You’re using Dex + LDAP? If you’re just using LDAP you don’t need any of that OIDC stuff (though i have to warn you now that apache’s LDAP is very insecure).

If you’re going Dex + LDAP route - I think that you need to set servername to apps.cluster.edu (again in ood_portal.yml.

I was planning on using Dex + LDAP. I have the Dex section configured for LDAP. I have servername set already. Is there something else I’m missing?

Do you have client_id set? Looks like there’s some conflict here. You could either just remove it (it’ll default to servername) or set it to apps.cluster.edu.

This is how we’re building that URL.

client_id was set. I’ve removed it and done the three steps above to regenerate files and restart apache. The same error still persists.

Are you restarting dex when you bounce things?

I believe so. This is one of the commands I’m running:
systemctl restart ondemand-dex.service

Cool - also let’s be sure it’s the same error you’re getting around OIDCRedirectURI. It could be a new error.

Yep. It’s the exact same.

Any updates on this?

Can you share your ood_portal.yml removing anything that’s sensitive.

Jeff,

See below. Sensitive information removed.

---
#
# Portal configuration
#

# The address and port to listen for connections on
# Example:
#     listen_addr_port: 443
# Default: null (don't add any more listen directives)
#listen_addr_port: null

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
#servername: apps.cluster.edu

# The server name used for rewrites
# Example:
#     proxy_server: 'proxy.example.com'
# Default: The value of servername
#proxy_server: null

# The port specification for the Virtual Host
# Example:
#     port: 8080
#Default: null (use default port 80 or 443 if SSL enabled)
#port: null

# List of SSL Apache directives
# Example:
#     ssl:
#       - 'SSLCertificateFile "/etc/pki/tls/certs/www.example.com.crt"'
#       - 'SSLCertificateKeyFile "/etc/pki/tls/private/www.example.com.key"'
# Default: null (no SSL support)
#ssl: null

# Root directory of log files (can be relative ServerRoot)
# Example:
#     logroot: '/path/to/my/logs'
# Default: 'logs' (this is relative to ServerRoot)
#logroot: 'logs'

# Error log filename
# Example:
#     errorlog: 'error.log'
# Default: 'error.log' (If 'servername' and 'ssl' options are defined 
# the default value will be <servername>_error_ssl.log)
#errorlog: 'error.log'

# Access log filename
# Example:
#     accesslog: 'access.log'
# Default: 'access.log' (If 'servername' and 'ssl' options are defined 
# the default value will be <servername>_access_ssl.log)
#accesslog: 'access.log'

# Apache access log format (Don't specify log nickname see: http://httpd.apache.org/docs/current/mod/mod_log_config.html#transferlog)
# Example:
#     logformat: '"%v %h \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{SSL_PROTOCOL}x %T"'
# Default: Apache combined format

# Should RewriteEngine be used
# Example:
#     use_rewrites: false
# Default: true
#use_rewrites: true

# Should Maintenance Rewrite rules be added
# Example:
#   use_maintenance: false
# Default: true
#use_maintenance: true

# List of IPs to whitelist when maintenance is enabled
# Example:
#   maintenance_ip_whitelist:
#     - 192.168.0..*
#     - 192.168.1..*
# Default: [] (no IPs whitelisted)
#maintenance_ip_whitelist: []

# Set Header Content-Security-Policy frame-ancestors.
# Example:
#   security_csp_frame_ancestors: https://ondemand.osc.edu
# Example to disable setting:
#   security_csp_frame_ancestors: false
# Default: based on servername and ssl settings
#security_csp_frame_ancestors:

# Set Header Strict-Transport-Security to help enforce SSL
# Example:
#   security_strict_transport: false
# Default: true when ssl is enabled, false otherwise
#security_strict_transport: false

# Root directory of the Lua handler code
# Example:
#     lua_root: '/path/to/lua/handlers'
# Default : '/opt/ood/mod_ood_proxy/lib' (default install directory of mod_ood_proxy)
#lua_root: '/opt/ood/mod_ood_proxy/lib'

# Verbosity of the Lua module logging
# (see https://httpd.apache.org/docs/2.4/mod/core.html#loglevel)
# Example:
#     lua_log_level: 'warn'
# Default: 'info' (get verbose logs)
#lua_log_level: 'info'

# System command used to map authenticated-user to system-user
# Example:
#     user_map_cmd: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex --regex=''^(\w+)@example.com$'''
# Default: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex' (this echo's back auth-user)
#user_map_cmd: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex'

# Use an alternative CGI environment variable instead of REMOTE_USER for
# determining the authenticated-user fed to the mapping script
# Example:
#     user_env: 'OIDC_CLAIM_preferred_username'
# Default: null (use REMOTE_USER)
#user_env: null

# Redirect user to the following URI if fail to map there authenticated-user to
# a system-user
# Example:
#     map_fail_uri: '/register'
# Default: null (don't redirect, just display error message)
#map_fail_uri: null

# System command used to run the `nginx_stage` script with sudo privileges
# Example:
#     pun_stage_cmd: 'sudo /path/to/nginx_stage'
# Default: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage' (don't forget sudo)
#pun_stage_cmd: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage'

# List of Apache authentication directives
# NB: Be sure the appropriate Apache module is installed for this
# Default: (see below, uses OIDC auth with Dex)
#auth:
#  - 'AuthType openid-connect'
#  - 'Require valid-user'

# Redirect user to the following URI when accessing root URI
# Example:
#     root_uri: '/my_uri'
#     # https://www.example.com/ => https://www.example.com/my_uri
# Default: '/pun/sys/dashboard' (default location of the OOD Dashboard app)
#root_uri: '/pun/sys/dashboard'

# Track server-side analytics with a Google Analytics account and property
# (see https://github.com/OSC/mod_ood_proxy/blob/master/lib/analytics.lua for
# information on how to setup the GA property)
# Example:
#     analytics:
#       url: 'http://www.google-analytics.com/collect'
#       id: 'UA-79331310-4'
# Default: null (do not track)
#analytics: null

#
# Publicly available assets
#

# Public sub-uri (available to public with no authentication)
# Example:
#     public_uri: '/assets'
# Default: '/public'
#public_uri: '/public'

# Root directory that serves the public sub-uri (be careful, everything under
# here is open to the public)
# Example:
#     public_root: '/path/to/public/assets'
# Default: '/var/www/ood/public'
#public_root: '/var/www/ood/public'

#
# Logout redirect helper
#

# Logout sub-uri
# Example
#     logout_uri: '/log_me_out'
# NB: If you change this, then modify the Dashboard app with the new sub-uri
# Default: '/logout' (the Dashboard app is by default going to expect this)
#logout_uri: '/logout'

# Redirect user to the following URI when accessing logout URI
# Example:
#     logout_redirect: '/oidc?logout=https%3A%2F%2Fwww.example.com'
# Default: '/pun/sys/dashboard/logout' (the Dashboard app provides a simple
# HTML page explaining logout to the user)
#logout_redirect: '/pun/sys/dashboard/logout'

#
# Reverse proxy to backend nodes
#

# Regular expression used for whitelisting allowed hostnames of nodes
# Example:
#     host_regex: '[\w.-]+\.example\.com'
# Default: '[^/]+' (allow reverse proxying to all hosts, this allows external
# hosts as well)
#host_regex: '[^/]+'

# Sub-uri used to reverse proxy to backend web server running on node that
# knows the full URI path
# Example:
#     node_uri: '/node'
# Default: null (disable this feature)
#node_uri: null

# Sub-uri used to reverse proxy to backend web server running on node that
# ONLY uses *relative* URI paths
# Example:
#     rnode_uri: '/rnode'
# Default: null (disable this feature)
#rnode_uri: null

#
# Per-user NGINX Passenger apps
#

# Sub-uri used to control PUN processes
# Example:
#     nginx_uri: '/my_pun_controller'
# Default: '/nginx'
#nginx_uri: '/nginx'

# Sub-uri used to access the PUN processes
# Example:
#     pun_uri: '/my_pun_apps'
# Default: '/pun'
#pun_uri: '/pun'

# Root directory that contains the PUN Unix sockets that the proxy uses to
# connect to
# Example:
#     pun_socket_root: '/path/to/pun/sockets'
# Default: '/var/run/ondemand-nginx' (default location set in nginx_stage)
#pun_socket_root: '/var/run/ondemand-nginx'

# Number of times the proxy attempts to connect to the PUN Unix socket before
# giving up and displaying an error to the user
# Example:
#     pun_max_retries: 25
# Default: 5 (only try 5 times)
#pun_max_retries: 5

#
# Support for OpenID Connect
#

# Sub-uri used by mod_auth_openidc for authentication
# Example:
#     oidc_uri: '/oidc'
# Default: null (disable OpenID Connect support)
#oidc_uri: /oidc

# Sub-uri user is redirected to if they are not authenticated. This is used to
# *discover* what ID provider the user will login through.
# Example:
#     oidc_discover_uri: '/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_uri: null

# Root directory on the filesystem that serves the HTML code used to display
# the discovery page
# Example:
#     oidc_discover_root: '/var/www/ood/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_root: null

#
# Support for registering unmapped users
#
# (Not necessary if using regular expressions for mapping users)
#

# Sub-uri user is redirected to if unable to map authenticated-user to
# system-user
# Example:
#     register_uri: '/register'
# Default: null (display error to user if mapping fails)
#register_uri: null

# Root directory on the filesystem that serves the HTML code used to register
# an unmapped user
# Example:
#     register_root: '/var/www/ood/register'
# Default: null (display error to user if mapping fails)
#register_root: null

# OIDC metadata URL
# Example:
#     oidc_provider_metadata_url: https://example.com:5554/.well-known/openid-configuration
# Default: null (value auto-generated if using Dex)
#oidc_provider_metadata_url: https://apps.cluster.edu

# OIDC client ID
# Example:
#     oidc_client_id: ondemand.example.com
# Default: null (value auto-generated if using Dex)
#oidc_client_id: null

# OIDC client secret
# Example:
#     oidc_client_secret: 334389048b872a533002b34d73f8c29fd09efc50
# Default: null (value auto-generated if using Dex)
#oidc_client_secret: null

# OIDC remote user claim. This is the claim that populates REMOTE_USER
# Example:
#     oidc_remote_user_claim: preferred_username
# Default: preferred_username
#oidc_remote_user_claim: preferred_username

# OIDC scopes
# Example:
#     oidc_scope: "openid profile email groups"
# Default: "openid profile email"
#oidc_scope: "openid profile email"

# OIDC session inactivity timeout
# Example:
#     oidc_session_inactivity_timeout: 28800
# Default: 28800
#oidc_session_inactivity_timeout: 28800

# OIDC session max duration
# Example:
#     oidc_session_max_duration: 28800
# Default: 28800
#oidc_session_max_duration: 28800

# OIDC max number of state cookies and if to automatically clean old cookies
# Example:
#     oidc_state_max_number_of_cookies: "10 true"
# Default: "10 true"
#oidc_state_max_number_of_cookies: "10 true"

# OIDC Enable SameSite cookie
# When ssl is defined this defaults to 'Off'
# When ssl is not defined this defaults to 'On'
# Example:
#     oidc_cookie_same_site: 'Off'
# Default: 'On'
#oidc_cookie_same_site: 'On'

# Additional OIDC settings as key-value pairs
# Example:
#     oidc_settings:
#       OIDCPassIDTokenAs: serialized
#       OIDCPassRefreshToken: On
# Default: {} (empty hash)

# Dex configurations, values inside the "dex" structure are directly used to configure Dex
# If the value for "dex" key is false or null, Dex support is disabled
# Dex support will auto-enable if ondemand-dex package is installed
#dex:
  # Default based on if ssl key for ood-portal-generator is defined
#  ssl: false
  # Only used if SSL is disabled
#  http_port: "5556"
  # Only used if SSL is enabled
#  https_port: "5554"
  # tls_cert and tls_key take OnDemand configured values for ssl and copy keys to /etc/ood/dex maintaining file names
#  tls_cert: null
#  tls_key: null
#  storage_file: /etc/ood/dex/dex.db
#  grpc: null
#  expiry: null
  # Client ID, defaults to servername or FQDN
#  client_id:
#  client_name: OnDemand
  # Client secret, value auto generated
  # A value that is a filesystem path can be used to store secret in a file
#  client_secret: /etc/ood/dex/ondemand.secret
   # The OnDemand redirectURI is auto-generated, this option allows adding additional URIs
#  client_redirect_uris: []
  # Additional Dex OIDC clients to configure
#  static_clients: []
  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
#  connectors:
#    - type: ldap
#      id: ldap
#      name: LDAP
#      config:
#        host: ldap.cluster.edu:636
#        insecureSkipVerify: false
#        bindDN:
#        bindPW:
#        userSearch:
#          baseDN: cn=users,cn=accounts,dc=ldap,dc=cluster,dc=edu
#          filter: "(objectClass=posixAccount)"
#          username: uid
#          idAttr: uid
#          emailAttr: mail
#          nameAttr: gecos
#          preferredUsernameAttr: uid
#        groupSearch:
#          baseDN: cn=cluster_users,cn=groups,cn=accounts,dc=ldap,dc=cluster,dc=edu
#          filter: "(objectClass=posixGroup)"
#          userMatchers:
#            - userAttr: DN
#              groupAttr: member
#          nameAttr: cn
#  frontend:
#    theme: ondemand
#    dir: /usr/share/ondemand-dex/web

You know hashtags # are comments in YAML right?

Meaning none of those configurations are being picked up.

I did not. That’s embarrassing. I thought because they had default values that it had to be formatted that way. I will go fix that right now.

All good! I knew at this point it has to be something we’re overlooking.

Let me know if you have any more issues!

Thanks! I’ve uncommented some of the values and I’m now getting: Bad request. Invalid client_id (“apps.cluster.edu”)

It must not like this empty client_id? You can keep both of those commented. The name I think is just a UI thing, but I’d guess it’s getting tripped up on client_id.

Jeff,

I’ve tried uncommenting and setting client_id but it doesn’t seem to change anything. Below is the dex section with sensitive info removed. Can you check and see if you see anything wrong?

dex:
  # Default based on if ssl key for ood-portal-generator is defined
#  ssl: false
  # Only used if SSL is disabled
#  http_port: "5556"
  # Only used if SSL is enabled
#  https_port: "5554"
  # tls_cert and tls_key take OnDemand configured values for ssl and copy keys to /etc/ood/dex maintaining file names
#  tls_cert: null
#  tls_key: null
#  storage_file: /etc/ood/dex/dex.db
#  grpc: null
#  expiry: null
  # Client ID, defaults to servername or FQDN
  #client_id:
  client_name: OnDemand
  # Client secret, value auto generated
  # A value that is a filesystem path can be used to store secret in a file
#  client_secret: /etc/ood/dex/ondemand.secret
   # The OnDemand redirectURI is auto-generated, this option allows adding additional URIs
#  client_redirect_uris: []
  # Additional Dex OIDC clients to configure
#  static_clients: []
  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: ldap.cluster.edu:636
        insecureSkipVerify: false
        bindDN:
        bindPW:
        userSearch:
          baseDN: cn=users,cn=accounts,dc=ldap,dc=cluster,dc=edu
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: cn=cluster_users,cn=groups,cn=accounts,dc=ldap,dc=cluster,dc=edu
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
  frontend:
    theme: ondemand
    dir: /usr/share/ondemand-dex/web