Internal server error after installing on RHEL8

Get Help
1

Hello,

I have a brand new RHEL 8 server that I am trying to get open ondemand working on.

Currently, this is what I have done.

dnf module enable ruby:2.7
dnf module enable nodejs:12
sudo subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms
sudo yum install https://yum.osc.edu/ondemand/2.0/ondemand-release-web-2.0-1.noarch.rpm
sudo yum install ondemand
sudo systemctl start httpd
sudo systemctl enable httpd
sudo groupadd ood
sudo useradd -d /home/ood -g ood -k /etc/skel -m ood

I have verified that selinux is off

cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

iptables is also currently disabled.

When I navigate to the site I get this error.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

In the error logs for httpd this is what I see.

[Fri Jun 17 16:42:03.001137 2022] [auth_openidc:error] [pid 49614:tid 139877365024512] [client 172.21.12.122:58648] oidc_provider_static_config: could not retrieve metadata from url: http://hostname:5556/.well-known/openid-configuration

I did try to install and use opendex too, but that was not working either.

2

Hi and welcome!

If ondemand-dex is installed we’ll automatically generate some OIDC (Open ID Connect) configurations for you. If it’s not, sadly, the 2.0 default is OIDC which you may or may not have installed.

Essentially - you’re (we’re) defaulting to OIDC configuration which you don’t have setup. What type of authentication are you looking to setup? OIDC with ondemand-dex or …

3

Hello,

I am looking to setup LDAP Authentication in the long run. I am just trying to get the platform up and running for evaluation purposes. I can implement whatever is easiest at this point.

4

I suppose it’s relative. If you’ve never setup an Apache before - there’s a little hurdle either any way you go, but ondemand-dex with LDAP is a very good option for you because even if you did do basic auth with a local user - that user may or may not be able to submit jobs to your scheduler, and that’s that whole reason you want to use Open OnDemand right?

Without a scheduler Open OnDemand is maybe a shell & file browsing over the web. But when you can submit jobs to a scheduler & interact with them - that’s the real use case isn’t it? And that is likely going to need tie into your LDAP. (Plus you actually need NFS HOME mounts for this use case to work too).

5

OK I have followed the directions to get ondemand-dex setup. I have added the LDAP info. When I go to the DNS name that I have configured. I still get the server error.

What config files or logs would help troubleshoot this issue?

6

Your httpd logs you’ve linked - from /var/log/httpd or /var/log/httpd-httpd24 would help if you’re getting 500s from apache.

You’ll likely need to provide your DNS name to ood_portal.yml through servername

servername: 'my DNS name'
7

Access.log

172.21.12.122 - - [22/Jun/2022:09:19:15 -0400] "GET / HTTP/1.1" 302 223 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
172.21.12.122 - - [22/Jun/2022:09:19:15 -0400] "GET /pun/sys/dashboard HTTP/1.1" 500 527 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"

error.log

[Sun Jun 19 03:23:02.230243 2022] [auth_openidc:warn] [pid 49608:tid 139878120876352] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCProviderMetadataURL SHOULD be "https" for security reasons!
[Sun Jun 19 03:23:02.230273 2022] [auth_openidc:warn] [pid 49608:tid 139878120876352] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCRedirectURI SHOULD be "https" for security reasons (moreover: some Providers may reject non-HTTPS URLs)
[Wed Jun 22 09:19:07.432592 2022] [auth_openidc:warn] [pid 134838:tid 139942840338752] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCProviderMetadataURL SHOULD be "https" for security reasons!
[Wed Jun 22 09:19:07.432604 2022] [auth_openidc:warn] [pid 134838:tid 139942840338752] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCRedirectURI SHOULD be "https" for security reasons (moreover: some Providers may reject non-HTTPS URLs)
[Wed Jun 22 09:19:15.483672 2022] [auth_openidc:error] [pid 134842:tid 139942126450432] [client 172.21.12.122:59682] oidc_util_http_call: curl_easy_perform() failed on: http://ood.hostname:5556/.well-known/openid-configuration (Failed to connect to ood.hostname port 5556: No route to host)
[Wed Jun 22 09:19:15.483720 2022] [auth_openidc:error] [pid 134842:tid 139942126450432] [client 172.21.12.122:59682] oidc_provider_static_config: could not retrieve metadata from url: http://ood.hostname:5556/.well-known/openid-configuration

ood_portal.yml

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
#servername: null
  servername: ood.hostname.edu

This could be part of the issue, if not the issue.

I ran the following, but when I ran curl ood.hostname.edu:5556 /.well-known/openid-configuration I got an error.

curl http://ri-prd-ood01.domain.local:5556/.well-known/openid-configuration
{
  "issuer": "http://ood.hostname.edu:5556",
  "authorization_endpoint": "http://ood.hostname.edu:5556/auth",
  "token_endpoint": "http://ood.hostname.edu:5556/token",
  "jwks_uri": "http://ood.hostname.edu:5556/keys",
  "userinfo_endpoint": "http://ood.hostname.edu:5556/userinfo",
  "device_authorization_endpoint": "http://ood.hostname.edu:5556/device/code",
  "grant_types_supported": [
    "authorization_code",
    "refresh_token",
    "urn:ietf:params:oauth:grant-type:device_code"
  ],
  "response_types_supported": [
    "code"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "code_challenge_methods_supported": [
    "S256",
    "plain"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "groups",
    "profile",
    "offline_access"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic"
  ],
  "claims_supported": [
    "aud",
    "email",
    "email_verified",
    "exp",
    "iat",
    "iss",
    "locale",
    "name",
    "sub"
  ]
}
8

OK so ondemand-dex is running, you’re able to see it. I can’t tell what is real and what you’ve obsfucated, but it seems to be a hostname/routing issue. Your curl command has a .local in it, so I can’t tell what the domain is.

Seems like everything should be ood.hostname.edu and never ri-prd-ood01.domain.local. I know you’ve got LDAP configs in your ood_portal.yml, you can remove any passwords and so on, but I am interested in whether you’ve added issuer or you let us default it.

9

Here is the LDAP Config that I used.

  # The following example is to configure OpenLDAP
  # Docs: https://github.com/dexidp/dex/blob/master/Documentation/connectors/ldap.md
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: domaincontroller:389
        insecureSkipVerify: true
        bindDN: CN=Open On Demand Auth,OU=Service Accounts,DC=domain,DC=local
        bindPW: pw
        userSearch:
          baseDN: OU=Users,DC=domain,DC=local
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: OU=Groups,DC=domain,DC=local
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
#  frontend:
#    theme: ondemand
#    dir: /usr/share/ondemand-dex/web
10

This is the error message we need to focus on. Let’s use the name ood.hostname.edu while discussing this. This is your servername configuration and a valid DNS entry.

Now can you reach http://ood.hostname.edu:5556? I see in your curl you’re using a different hostname. This is important. We need to be able to reach ood.hostname.edu on port 5556 from the machine your hosting OOD on and from your own laptop/desktop.

Looking at the docs, we use this config for defaults which is great, that’s all you should need to set (beyond the LDAP configs you have).

servername: 'ood.hostname.edu'

It appears from your curl command that Dex is indeed configured correctly. So… the question becomes why can you not reach http://ood.hostname.edu:5556? Do you need to open that port? The log entry doesn’t have the .edu domain - again I can’t tell if that’s been obsfucated and it was just a mistake or if it’s an actual typo somewhere.

11

I cannot access that URL. I can access it at the .local address when I do the CURL command, but not the ood.hostname.edu. I am working on getting that resolved now.

1 Like
12

Sorry for the delay. I have removed the site from our reverse proxy to try to get this working first.

Now I am getting a 404 error. I get this when I go to page. When I do the curl command, I get a successful response.

Here is a copy of the last few lines of the error log.

[Wed Jun 22 12:09:52.016079 2022] [auth_openidc:error] [pid 134840:tid 139941820483328] [client 172.21.12.122:56558] oidc_metadata_provider_retrieve: JSON parsing of retrieved Discovery document failed
[Wed Jun 22 12:09:52.016085 2022] [auth_openidc:error] [pid 134840:tid 139941820483328] [client 172.21.12.122:56558] oidc_provider_static_config: could not retrieve metadata from url: http://ood.hostname.edu:5556/.well-known/openid-configuration
[Wed Jun 22 12:52:17.589509 2022] [auth_openidc:error] [pid 134842:tid 139941434615552] [client 172.21.12.122:57478] oidc_util_decode_json_object: JSON parsing returned an error: '[' or '{' expected near '404' (404 page not found\n)
[Wed Jun 22 12:52:17.589551 2022] [auth_openidc:error] [pid 134842:tid 139941434615552] [client 172.21.12.122:57478] oidc_metadata_provider_retrieve: JSON parsing of retrieved Discovery document failed
[Wed Jun 22 12:52:17.589557 2022] [auth_openidc:error] [pid 134842:tid 139941434615552] [client 172.21.12.122:57478] oidc_provider_static_config: could not retrieve metadata from url: http://ood.hostname.edu:5556/.well-known/openid-configuration
[Thu Jun 23 06:26:35.242591 2022] [auth_openidc:warn] [pid 151180:tid 139909956196672] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCProviderMetadataURL SHOULD be "https" for security reasons!
[Thu Jun 23 06:26:35.242618 2022] [auth_openidc:warn] [pid 151180:tid 139909956196672] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCRedirectURI SHOULD be "https" for security reasons (moreover: some Providers may reject non-HTTPS URLs)
[Fri Jun 24 10:36:22.556870 2022] [auth_openidc:error] [pid 151282:tid 139909242308352] [client 172.21.12.122:38736] oidc_util_decode_json_object: JSON parsing returned an error: '[' or '{' expected near '404' (404 page not found\n)
[Fri Jun 24 10:36:22.556902 2022] [auth_openidc:error] [pid 151282:tid 139909242308352] [client 172.21.12.122:38736] oidc_metadata_provider_retrieve: JSON parsing of retrieved Discovery document failed
[Fri Jun 24 10:36:22.556906 2022] [auth_openidc:error] [pid 151282:tid 139909242308352] [client 172.21.12.122:38736] oidc_provider_static_config: could not retrieve metadata from url: http://ood.hostname.edu:5556/.well-known/openid-configuration
13

No issue in the delay - I’m happy to respond whenever.

OK if you’re using a proxy then you may need to use proxy_server in combination with servername in ood_portal.yml.

When you curl where did you curl from? A curl command from the ood.hostname.edu host itself (or wherever apache and dex are installed) is how you want to replicate. curl from your own machine and/or any other machine isn’t quite replicating this. Also be careful with flags you’re passing to curl. Specifically -L if you have to follow redirects. I don’t know if mod_auth_openidc can handle redirection here.

Clearly there’s something in the middle here, but without knowing more about your topology I’m not quite sure what.

14

I dropped the proxy for now. I will get the system working and then try to get that added back in.

As for the curl command I was running the command straight from the computer.

curl http://ood.hostname.edu:5556/.well-known/openid-confguration

When I run the curl command from the server I get a 404 page not found error. Its odd that I can curl from my workstation but not the server.

15

Someone’s intercepting this. There may/should be a Server header about who that was (maybe apache? or indeed your proxy server). In any case, the headers of the response will tell you something. Maybe even the response page itself will indicate who it’s from.

16

OK, Not sure what changed, but it’s working now. The system is asking for my email address (I also tried my AD username) and I entered that, but I am getting an username and password do not match. Are there any logs that I can look at help run down this new issue?

17

Very good! Progress!

You could look at systemctl status ondemand-dex or journalctl -u ondemand-dex but I don’t believe it’ll say anything meaningful. That is, unless you’re unable to bind/connect to your LDAP at all. That could be one issue. That you’re unable to query LDAP at all. I don’t know how dex would respond to the user with this info.

If you can correctly bind - then I’m guessing this is the documentation you’re looking for. My guess is your search/filter/query isn’t quite right.

https://dexidp.io/docs/connectors/ldap/

There’s also this note that I left for myself around ldapsearch and different ways to debug you’re query. I get the feeling that you’ll need to experiment with ldapsearch to refine your settings.

18

I had the original issue on RHEL 8. Once I installed OOD and Dex (started and enabled), config’d ports, certs… Then ran the portal generator (/opt/ood/ood-portal-generator/sbin/update_ood_portal) and restarted apache (systemctl try-restart httpd.service htcacheclean.service) it worked.

19

Hi Peter.

Welcome to the Open OnDemand Discourse. Thank you for your help here.

-gerald

20

I got the similar issue on Rocky 8.5.

It is a small demo cluster, one login(master) node, 3 computing nodes,
all are hosted in the Virtual box.
All virtual servers are works within the network 192.168.57.0/24.
The login/master node has the IP address 192.168.57.13.
I also add the line “192.168.57.13 hpc4you.login” to /etc/hosts file cuz LDAP is confiugred to use “hpc4you.login”.
I have tested the LDAP works for the cluster.

Then I installed OOD + Dex + phpLDAPadmin.
Yes, all these are running on the login(master) node.

For test purpose, I deployed the self-signed keys.
Now phpLDAPadmin works in https.
I can access phpLDAPadmin via https://192.168.57.13/users

If SSL is enabled, I can find the following msg:

[Tue Nov 08 19:52:19.753978 2022] [auth_openidc:error] [pid 13233:tid 139887009326848] [client 192.168.57.1:50670] oidc_util_http_call: curl_easy_perform() failed on: https://hpc4you.login:5554/.well-known/openid-configuration (SSL certificate problem: self signed certificate)
[Tue Nov 08 19:52:19.754098 2022] [auth_openidc:error] [pid 13233:tid 139887009326848] [client 192.168.57.1:50670] oidc_provider_static_config: could not retrieve metadata from url: https://hpc4you.login:5554/.well-known/openid-configuration.

curl -k https://hpc4you.login:5554/.well-known/openid-configuration

gives out the following msg,

[root@master ~]# curl -k https://hpc4you.login:5554/.well-known/openid-configuration
{
“issuer”: “https://hpc4you.login:5554”,
“authorization_endpoint”: “https://hpc4you.login:5554/auth”,
“token_endpoint”: “https://hpc4you.login:5554/token”,
“jwks_uri”: “https://hpc4you.login:5554/keys”,
“userinfo_endpoint”: “https://hpc4you.login:5554/userinfo”,
“device_authorization_endpoint”: “https://hpc4you.login:5554/device/code”,
“grant_types_supported”: [
“authorization_code”,
“refresh_token”,
“urn:ietf:params:oauth:grant-type:device_code”
],
“response_types_supported”: [
“code”
],
“subject_types_supported”: [
“public”
],
“id_token_signing_alg_values_supported”: [
“RS256”
],
“code_challenge_methods_supported”: [
“S256”,
“plain”
],
“scopes_supported”: [
“openid”,
“email”,
“groups”,
“profile”,
“offline_access”
],
“token_endpoint_auth_methods_supported”: [
“client_secret_basic”,
“client_secret_post”
],
“claims_supported”: [
“iss”,
“sub”,
“aud”,
“iat”,
“exp”,
“email”,
“email_verified”,
“locale”,
“name”,
“preferred_username”,
“at_hash”
]
}[root@master ~]#

Of course, curl https://... fails.

The problem is, is it possible to user self-signed keys?
Or, can I totally disable SSL?

It seems, without SSL, Job composer does not work.