Error 500 Internal Server Error

Hello there everyone,

before trying to roll out Open OnDemand on our cluster I wanted to test it in a VM for which I followed the official 2.0 documentation.
Now after installing I have trouble accessing the Dashboard as I always get:

Error 500 The server encountered an internal error or misconfiguration and was unable to complete your request.

Now this is what the errorlog in /var/log/httpd24/error.log gave me but I am unsure how to change the RedirectURl.

`[Tue Oct 25 11:44:08.074666 2022] [auth_openidc:error] [pid 1470] [client 172.24.0.1:62884] oidc_authenticate_user: the URL hostname (localhost) of the configured OIDCRedirectURI does not match the URL hostname of the URL being accessed (172.24.3.56): the "state" and "session" cookies will not be shared between the two!`

I am terribly sorry if this may be a dumb question but I am fairly new to HPC so I thought it better to ask.

Hello and welcome! First, no dumb questions here, it can get confusing quickly in this space :smiley:

Could you post the ood_portal.yml file you are using currently to configure your web-node?

Also, is your auth page coming up as expected when you hit the initial url or do you just see that 500 error immediately when going to the url to login?

Lastly, what OS and version of OOD are you running?

Hi thank you for the warm welcome :smiley: .

First of all here is my ood_portal.yml please note that it should be on a default config as I wanted to just test it first :

#
# Portal configuration
#

# The address and port to listen for connections on
# Example:
#     listen_addr_port: 443
# Default: null (don't add any more listen directives)
#listen_addr_port: null

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
#servername: null

# The server name used for rewrites
# Example:
#     proxy_server: 'proxy.example.com'
# Default: The value of servername
#proxy_server: null

# The port specification for the Virtual Host
# Example:
#     port: 8080
#Default: null (use default port 80 or 443 if SSL enabled)
#port: null

# List of SSL Apache directives
# Example:
#     ssl:
#       - 'SSLCertificateFile "/etc/pki/tls/certs/www.example.com.crt"'
#       - 'SSLCertificateKeyFile "/etc/pki/tls/private/www.example.com.key"'
# Default: null (no SSL support)
#ssl: null

# Root directory of log files (can be relative ServerRoot)
# Example:
#     logroot: '/path/to/my/logs'
# Default: 'logs' (this is relative to ServerRoot)
#logroot: 'logs'

# Error log filename
# Example:
#     errorlog: 'error.log'
# Default: 'error.log' (If 'servername' and 'ssl' options are defined
# the default value will be <servername>_error_ssl.log)
#errorlog: 'error.log'

# Access log filename
# Example:
#     accesslog: 'access.log'
# Default: 'access.log' (If 'servername' and 'ssl' options are defined
# the default value will be <servername>_access_ssl.log)
#accesslog: 'access.log'

# Apache access log format (Don't specify log nickname see: http://httpd.apache.org/docs/current/mod/mod_log_config.html#transferlog)
# Example:
#     logformat: '"%v %h \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{SSL_PROTOCOL}x %T"'
# Default: Apache combined format

# Should RewriteEngine be used
# Example:
#     use_rewrites: false
# Default: true
#use_rewrites: true

# Should Maintenance Rewrite rules be added
# Example:
#   use_maintenance: false
# Default: true
#use_maintenance: true

# List of IPs to whitelist when maintenance is enabled
# Example:
#   maintenance_ip_whitelist:
#     - 192.168.0..*
#     - 192.168.1..*
# Default: [] (no IPs whitelisted)
#maintenance_ip_whitelist: []

# Set Header Content-Security-Policy frame-ancestors.
# Example:
#   security_csp_frame_ancestors: https://ondemand.osc.edu
# Example to disable setting:
#   security_csp_frame_ancestors: false
# Default: based on servername and ssl settings
#security_csp_frame_ancestors:

# Set Header Strict-Transport-Security to help enforce SSL
# Example:
#   security_strict_transport: false
# Default: true when ssl is enabled, false otherwise
#security_strict_transport: false

# Root directory of the Lua handler code
# Example:
#     lua_root: '/path/to/lua/handlers'
# Default : '/opt/ood/mod_ood_proxy/lib' (default install directory of mod_ood_proxy)
#lua_root: '/opt/ood/mod_ood_proxy/lib'

# Verbosity of the Lua module logging
# (see https://httpd.apache.org/docs/2.4/mod/core.html#loglevel)
# Example:
#     lua_log_level: 'warn'
# Default: 'info' (get verbose logs)
#lua_log_level: 'info'

# Lua regular expression used to map authenticated-user to system-user
# This configuration is ignored if user_map_cmd is defined
# Example:
#     user_map_match: '^([^@]+)@.*$'
# Default: '.*'
# user_map_match: '.*'

# System command used to map authenticated-user to system-user
# This option takes precedence over user_map_match
# Example:
#     user_map_cmd: '/usr/local/bin/ondemand-usermap'
# Default: null (use user_map_match)
#user_map_cmd: null

# Use an alternative CGI environment variable instead of REMOTE_USER for
# determining the authenticated-user fed to the mapping script
# Example:
#     user_env: 'OIDC_CLAIM_preferred_username'
# Default: null (use REMOTE_USER)
#user_env: null

# Redirect user to the following URI if fail to map there authenticated-user to
# a system-user
# Example:
#     map_fail_uri: '/register'
# Default: null (don't redirect, just display error message)
#map_fail_uri: null

# System command used to run the `nginx_stage` script with sudo privileges
# Example:
#     pun_stage_cmd: 'sudo /path/to/nginx_stage'
# Default: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage' (don't forget sudo)
#pun_stage_cmd: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage'

# List of Apache authentication directives
# NB: Be sure the appropriate Apache module is installed for this
# Default: (see below, uses OIDC auth with Dex)
#auth:
#  - 'AuthType openid-connect'
#  - 'Require valid-user'

# Redirect user to the following URI when accessing root URI
# Example:
#     root_uri: '/my_uri'
#     # https://www.example.com/ => https://www.example.com/my_uri
# Default: '/pun/sys/dashboard' (default location of the OOD Dashboard app)
#root_uri: '/pun/sys/dashboard'

# Track server-side analytics with a Google Analytics account and property
# (see https://github.com/OSC/mod_ood_proxy/blob/master/lib/analytics.lua for
# information on how to setup the GA property)
# Example:
#     analytics:
#       url: 'http://www.google-analytics.com/collect'
#       id: 'UA-79331310-4'
# Default: null (do not track)
#analytics: null

#
# Publicly available assets
#

# Public sub-uri (available to public with no authentication)
# Example:
#     public_uri: '/assets'
# Default: '/public'
#public_uri: '/public'

# Root directory that serves the public sub-uri (be careful, everything under
# here is open to the public)
# Example:
#     public_root: '/path/to/public/assets'
# Default: '/var/www/ood/public'
#public_root: '/var/www/ood/public'

#
# Logout redirect helper
#

# Logout sub-uri
# Example
#     logout_uri: '/log_me_out'
# NB: If you change this, then modify the Dashboard app with the new sub-uri
# Default: '/logout' (the Dashboard app is by default going to expect this)
#logout_uri: '/logout'

# Redirect user to the following URI when accessing logout URI
# Example:
#     logout_redirect: '/oidc?logout=https%3A%2F%2Fwww.example.com'
# Default: '/pun/sys/dashboard/logout' (the Dashboard app provides a simple
# HTML page explaining logout to the user)
#logout_redirect: '/pun/sys/dashboard/logout'

#
# Reverse proxy to backend nodes
#

# Regular expression used for whitelisting allowed hostnames of nodes
# Example:
#     host_regex: '[\w.-]+\.example\.com'
# Default: '[^/]+' (allow reverse proxying to all hosts, this allows external
# hosts as well)
#host_regex: '[^/]+'

# Sub-uri used to reverse proxy to backend web server running on node that
# knows the full URI path
# Example:
#     node_uri: '/node'
# Default: null (disable this feature)
#node_uri: null

# Sub-uri used to reverse proxy to backend web server running on node that
# ONLY uses *relative* URI paths
# Example:
#     rnode_uri: '/rnode'
# Default: null (disable this feature)
#rnode_uri: null

#
# Per-user NGINX Passenger apps
#

# Sub-uri used to control PUN processes
# Example:
#     nginx_uri: '/my_pun_controller'
# Default: '/nginx'
#nginx_uri: '/nginx'

# Sub-uri used to access the PUN processes
# Example:
#     pun_uri: '/my_pun_apps'
# Default: '/pun'
#pun_uri: '/pun'

# Root directory that contains the PUN Unix sockets that the proxy uses to
# connect to
# Example:
#     pun_socket_root: '/path/to/pun/sockets'
# Default: '/var/run/ondemand-nginx' (default location set in nginx_stage)
#pun_socket_root: '/var/run/ondemand-nginx'

# Number of times the proxy attempts to connect to the PUN Unix socket before
# giving up and displaying an error to the user
# Example:
#     pun_max_retries: 25
# Default: 5 (only try 5 times)
#pun_max_retries: 5

# The PUN pre hook command to execute as root
#
# Example:
#    pun_pre_hook_root_cmd: '/opt/hpc-site/ood_pun_prehook'
# Default: null (do not run any PUN pre hook as root)
#pun_pre_hook_root_cmd: null

# Comma separated list of environment variables to pass from the apache context
# into the PUN pre hook. Defaults to null so nothing is exported.
#
# Example:
#    pun_pre_hook_exports: 'OIDC_ACCESS_TOKEN,OIDC_CLAIM_EMAIL'
# Default: null (pass nothing)
#pun_pre_hook_exports: null

#
# Support for OpenID Connect
#

# Sub-uri used by mod_auth_openidc for authentication
# Example:
#     oidc_uri: '/oidc'
# Default: null (disable OpenID Connect support)
#oidc_uri: null

# Sub-uri user is redirected to if they are not authenticated. This is used to
# *discover* what ID provider the user will login through.
# Example:
#     oidc_discover_uri: '/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_uri: null

# Root directory on the filesystem that serves the HTML code used to display
# the discovery page
# Example:
#     oidc_discover_root: '/var/www/ood/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_root: null

#
# Support for registering unmapped users
#
# (Not necessary if using regular expressions for mapping users)
#

# Sub-uri user is redirected to if unable to map authenticated-user to
# system-user
# Example:
#     register_uri: '/register'
# Default: null (display error to user if mapping fails)
#register_uri: null

# Root directory on the filesystem that serves the HTML code used to register
# an unmapped user
# Example:
#     register_root: '/var/www/ood/register'
# Default: null (display error to user if mapping fails)
#register_root: null

# OIDC metadata URL
# Example:
#     oidc_provider_metadata_url: https://example.com:5554/.well-known/openid-configuration
# Default: null (value auto-generated if using Dex)
#oidc_provider_metadata_url: null

# OIDC client ID
# Example:
#     oidc_client_id: ondemand.example.com
# Default: null (value auto-generated if using Dex)
#oidc_client_id: null

# OIDC client secret
# Example:
#     oidc_client_secret: 334389048b872a533002b34d73f8c29fd09efc50
# Default: null (value auto-generated if using Dex)
#oidc_client_secret: null

# OIDC remote user claim. This is the claim that populates REMOTE_USER
# Example:
#     oidc_remote_user_claim: preferred_username
# Default: preferred_username
#oidc_remote_user_claim: preferred_username

# OIDC scopes
# Example:
#     oidc_scope: "openid profile email groups"
# Default: "openid profile email"
#oidc_scope: "openid profile email"

# OIDC session inactivity timeout
# Example:
#     oidc_session_inactivity_timeout: 28800
# Default: 28800
#oidc_session_inactivity_timeout: 28800

# OIDC session max duration
# Example:
#     oidc_session_max_duration: 28800
# Default: 28800
#oidc_session_max_duration: 28800

# OIDC max number of state cookies and if to automatically clean old cookies
# Example:
#     oidc_state_max_number_of_cookies: "10 true"
# Default: "10 true"
#oidc_state_max_number_of_cookies: "10 true"

# OIDC Enable SameSite cookie
# When ssl is defined this defaults to 'Off'
# When ssl is not defined this defaults to 'On'
# Example:
#     oidc_cookie_same_site: 'Off'
# Default: 'On'
#oidc_cookie_same_site: 'On'

# Additional OIDC settings as key-value pairs
# Example:
#     oidc_settings:
#       OIDCPassIDTokenAs: serialized
#       OIDCPassRefreshToken: On
# Default: {} (empty hash)

# The Dex URI behind Apache reverse proxy
# Setting this value to some path will result in Dex listening on localhost
# as well as only using HTTP for proxied communication
# Example:
#   dex_uri: /dex
# Default: null
#dex_uri: null

# Dex configurations, values inside the "dex" structure are directly used to configure Dex
# If the value for "dex" key is false or null, Dex support is disabled
# Dex support will auto-enable if ondemand-dex package is installed
#dex:
  # Default based on if ssl key for ood-portal-generator is defined
#  ssl: false
  # Only used if SSL is disabled
#  http_port: "5556"
  # Only used if SSL is enabled
#  https_port: "5554"
  # tls_cert and tls_key take OnDemand configured values for ssl and copy keys to /etc/ood/dex maintaining file names
#  tls_cert: null
#  tls_key: null
#  storage_file: /etc/ood/dex/dex.db
#  grpc: null
#  expiry: null
  # Client ID, defaults to servername or FQDN
#  client_id: null
#  client_name: OnDemand
  # Client secret, value auto generated
  # A value that is a filesystem path can be used to store secret in a file
#  client_secret: /etc/ood/dex/ondemand.secret
   # The OnDemand redirectURI is auto-generated, this option allows adding additional URIs
#  client_redirect_uris: []
  # Additional Dex OIDC clients to configure
#  static_clients: []
  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
#  connectors:
#    - type: ldap
#      id: ldap
#      name: LDAP
#      config:
#        host: openldap.my_center.edu:636
#        insecureSkipVerify: false
#        bindDN: cn=admin,dc=example,dc=org
#        bindPW: admin
#        userSearch:
#          baseDN: ou=People,dc=example,dc=org
#          filter: "(objectClass=posixAccount)"
#          username: uid
#          idAttr: uid
#          emailAttr: mail
#          nameAttr: gecos
#          preferredUsernameAttr: uid
#        groupSearch:
#          baseDN: ou=Groups,dc=example,dc=org
#          filter: "(objectClass=posixGroup)"
#          userMatchers:
#            - userAttr: DN
#              groupAttr: member
#          nameAttr: cn
#  frontend:
#    theme: ondemand
#    dir: /usr/share/ondemand-dex/web

To answer your other questions:

  • No there is no dashboard coming up. When I try to connect it just straight up shows me the error message.
  • Although I can access the apache start page by typing https://ip:443 but as I understood from the documentation that isnt the one I’m trying to get, right?
  • Lastly my OS version is CentOS Linux release 7.9.2009 (Core) and my OOD version would be 2.0.28

Thanks for the information!

When you went through the install, did you ensure to include the ondemand-dex package?

If so, did you also follow the instructions to enable and and start the service?
https://osc.github.io/ood-documentation/latest/installation/start-apache.html

Do you have anything like SELinux or iptables running that may need configuration?

Correct, that is just apache serving its own page and not part of ondemand.

Cool, and to be sure, you followed the package install instructions here for CentOS 7:
https://osc.github.io/ood-documentation/latest/installation/install-software.html

Hi,

yes I did ensure to include ondemand-dex and have enabled it.

ondemand-dex.service - OnDemand Dex - A federated OpenID Connect provider packaged for OnDemand
   Loaded: loaded (/usr/lib/systemd/system/ondemand-dex.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-10-27 21:49:33 CEST; 2min 11s ago
 Main PID: 1378 (ondemand-dex)
   CGroup: /system.slice/ondemand-dex.service
           └─1378 /usr/sbin/ondemand-dex serve /etc/ood/dex/config.yaml

The same goes for the httpd24-httpd service

httpd24-httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd24-httpd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/httpd24-httpd.service.d
           └─ood-portal.conf, ood.conf
   Active: active (running) since Thu 2022-10-27 21:49:33 CEST; 2min 35s ago
  Process: 873 ExecStartPre=/opt/ood/ood-portal-generator/sbin/update_ood_portal --rpm (code=exited, status=0/SUCCESS)
 Main PID: 1376 (httpd)
   Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec:   0 B/sec"
   CGroup: /system.slice/httpd24-httpd.service
           ├─1376 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
           ├─1384 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
           ├─1385 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
           ├─1386 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
           ├─1387 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND
           └─1388 /opt/rh/httpd24/root/usr/sbin/httpd -DFOREGROUND

Also no I’ve disabled both SELinux and the firewall because I wanted to avoid unnecessary problems.

firewalld.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)

getenforce
Disabled

And I’ve checked and should’ve installed all the packages from step 1-3 (excluding step 4 as I’ve disabled SELinux.

Thanks again for your help.

There’s a bug on our side with defaults in 2.0. You’ve defaulted to OIDC & dex even though you didn’t mean to.

What type of authentication are you trying to setup? We’ve put you in a weird initial state that doesn’t work (obviously) so you need to enable the actual authentication scheme (whether that’s actually OIDC or Shibboleth or another).

Hi, thank you for your reply.

So for my testing purposes I tried to install it with Dex and configuring the standard ood user to see if everything works.
This would also be the go to way for installing it on a live system with the exception that I’d not use the ood user but the LDAP Authentication instead.
So in both cases I’d use dex to authentificate (if I understood the installation guide correctly)

I’d go right to setting up your LDAP. That local user ood really should be avoided (I know it’s documented, we’re trying to fix that).

Hey there, I got it running so far but I think I messed up something while trying to configure LDAP for it.

My current config for it looks like this:

# If the value for "dex" key is false or null, Dex support is disabled
# Dex support will auto-enable if ondemand-dex package is installed
#dex:
  # Default based on if ssl key for ood-portal-generator is defined
#  ssl: false
  # Only used if SSL is disabled
#  http_port: "5556"
  # Only used if SSL is enabled
#  https_port: "5554"
  # tls_cert and tls_key take OnDemand configured values for ssl and copy keys to /etc/ood/dex maintaining file names
#  tls_cert: null
#  tls_key: null
#  storage_file: /etc/ood/dex/dex.db
#  grpc: null
#  expiry: null
  # Client ID, defaults to servername or FQDN
#  client_id: null
#  client_name: OnDemand
  # Client secret, value auto generated
  # A value that is a filesystem path can be used to store secret in a file
#  client_secret: /etc/ood/dex/ondemand.secret
   # The OnDemand redirectURI is auto-generated, this option allows adding additional URIs
#  client_redirect_uris: []
  # Additional Dex OIDC clients to configure
#  static_clients: []
  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
#  connectors:
#    - type: ldap
#      id: ldap
#      name: LDAP
#      config:
#        host: ###-###.net:389
#        insecureSkipVerify: true
#        insecureNoSSL: true
#        bindDN: cn=###,ou=functional_accounts,dc=###-###,dc=net
#        bindPW: ###
#        userSearch:
#          baseDN: dc=###-###,dc=net
#          filter: "(&(objectCategory=Person)(|(employeeID=*))(!(UserAccountControl:1.2.840.113556.1.4.803:=2)))"
#          username: sAMAccountName
#          idAttr: sAMAccountName
#          emailAttr: mail
#          nameAttr: sAMAccountName
#          preferredUsernameAttr: sAMAccountName
#        groupSearch:
#          baseDN: dc=###-###,dc=net
#          filter: "(&(objectCategory=Person)(|(employeeID=*))(!(UserAccountControl:1.2.840.113556.1.4.803:=2)))"
#          userMatchers:
#            - userAttr: DN
#              groupAttr: member
#          nameAttr: cn

The # are for infos I’m not specifying for safety reasons.

I am quite confused though as when I:

sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal

I only get the output that nothing changed with dex which would tell me that the config didn’t even go through.
Any idea what I did wrong?

Best
Nico

Besides the # for obfuscating sensitive information - did you mean to have all the lines begin with #? That means they’re commented lines in YAML, so the config is essentially nil. I’m not sure if that’s intended or not.

After running update_ood_portal you should see /etc/ood/dex/config.yaml show up. That’s where ondemand-dex (the systemd unit) will read configs from and you’ll see the configs you’ve supplied end up.

Ah yes, thank you again for your help I was being a massive idiot here.

I am also again apologizing in advance for another possible dumb question (thus taking up your time):

I got LDAP working now (hooray!) I can also login but when I do that I am welcomed by this message:

Error -- 
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

sudo: no tty present and no askpass program specified

And

/var/spool/mail/root

throws this message:

From root@xxx.xxx.net  Wed Nov 16 10:45:14 2022
Return-Path: <root@xxx.xxx.net>
X-Original-To: root
Delivered-To: root@xxx.xxx.net
Received: by xxx.xxx.net (Postfix, from userid 0)
        id E640E2072FC4; Wed, 16 Nov 2022 10:45:14 +0100 (CET)
To: root@xxx.xxx.net
From: apache@xxx.xxx.net
Auto-Submitted: auto-generated
Subject: *** SECURITY information for xxx.xxx.net ***
Message-Id: <20221116094514.E640E2072FC4@xxx.xxx.net>
Date: Wed, 16 Nov 2022 10:45:14 +0100 (CET)

xxx.xxx.net : Nov 16 10:45:14 : apache : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u (username) -a http%3a%2f%2fxxx.xxx.net%3a80%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri

Now, I am sure that I missed something somewhere because I don’t think that I should actually give every user sudo permissons just to access the dashboard right? (Because this would seem kinda unsafe for me to do)

Nvm I found the error. Thank you all for your help.

Good to hear it all worked out. Just open another topic if you run into more issues.