I am installing a new production Open OnDemand on a RHEL 8.3 VM. Using Dex with ldap for authentication. I had everything working like I wanted until I tried to add the ssl certs and switch to https. As expected when you go to http://ondemand.rc.colorado it redirects to https but it fails with
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
The logs (/var/log/httpd/ondemand.rc.colorado.edu_error_ssl.log) reports the following errors:
[Tue Aug 03 11:09:06.185239 2021] [auth_openidc:error] [pid 24200:tid 140328719869696] [client 126.96.36.199:35442] oidc_util_http_call: curl_easy_perform() failed on: https://ondemand.rc.colorado.edu:5554/.well-known/openid-configuration (SSL certificate problem: unable to get local issuer certificate)
[Tue Aug 03 11:09:06.185355 2021] [auth_openidc:error] [pid 24200:tid 140328719869696] [client 188.8.131.52:35442] oidc_provider_static_config: could not retrieve metadata from url: https://ondemand.rc.colorado.edu:5554/.well-known/openid-configuration
Ports on firewall are open. I feel like I am missing something obvious.
Config file is all defaults except for the following:
- 'SSLCertificateFile "/etc/pki/tls/certs/ondemand_rc_colorado_edu_cert.crt"'
- 'SSLCertificateKeyFile "/etc/pki/tls/private/ondemand.rc.colorado.edu.key"'
- 'SSLCertificateChainFile "/etc/pki/tls/certs/ondemand_rc_colorado_edu_interm.crt"'
- type: ldap
- userAttr: DN
Looking at the error log it does seem like maybe a step was missed. You probably already did this but the first thing I wanted to check was after the changes, did you update restart the services:
sudo systemctl restart ondemand-dex
sudo systemctl restart httpd
Assuming you did that, I would check the permissions on the directories and the files, along with correct ownership of the directories and files. Knowing those are all configured correctly would help with further troubleshooting.
I did rebuild the config and then restarted the services. I have even rebooted the VM. Which files and directories should I be concerned with?
The actual SSL directories and files, so:
root owns all the directories and files. Ensure the permissions on the directories are all 700 and permissions on the files themselves should be 600.
Checked file directory and permissions and corrected where necessary. Restarted httpd and ondemand-dex; no change; same error in logs.
Rebooted VM. No change.
Weird issue though. The instance is supposed to be operating as ondemand.rc.colorado.edu. When I try and go the actual VM name https://ondemand2.rc.int.colorado.edu, I get the apache test page and an insecure webpage warning,
Hm, it may be that some default apache configs are getting in the way.
I’m unfortunately not too familiar with RH 8’s
httpd.conf file locations, but if you look somewhere like:
See if any of the listen directives are still uncommented, and then change them like so:
This may solve the problem you just mentioned, though not necessarily the larger issue, but let’s start there for now because it may be a port conflict that is throwing things off.
Also, I noticed there is a
dc=int in the
dex portion of the config, which is used in some of the URLs, but was not in the initial error message from the first message you had sent in.
Looking at the
servername I also don’t see it set there either, which makes me wonder about whether the correct servername is even being used by OOD itself, and that might be why you are also seeing apache pages.
I’d see if adjusting the
servername with the usual
update_ood_portal and restarts to see if that has any effect.
Commented out the one line of Listen 80 in /etc/httpd/conf/httpd.conf. Restarted httpd and ondemand-dex; no change.
Found other other Listen directive in /etc/httpd/conf.d/ssl.conf. Commenting that one out prevents httpd from starting.
Changed servername to ondeman2.rc.int.colorado.edu (this is the internal hostname and what you get with hostname). Rebuilt config and restarted server; No change.
Hey sorry, I was speaking only about the
httpd.conf files for those
I’m assuming you uncommented the
ssl.conf entry after the failed
It seems like this has to be something simple, the permissions are fine, the ownership is fine, the locations look good, you’ve rebuilt and restarted the services, and when you are saying “no change” does that mean you are still seeing the initial “Internal Server Error” message? Trying to make sure I know what you are seeing correctly since the start.
Also, I’m not sure the regex in
host_regex is capturing correctly. I think you need something like
[^\/]+ for that to capture correctly. https://regex101.com is great to test on.
So I added the following two lines to /etc/httpd.conf.d/ood-portal.conf.
This took me one step closer to success. I am now seeing the DEX login screen. Login how ever fails with the following message on the screen.
There is no ca-certificates file anywhere on the system. Not really sure why it is looking in the /etc/ssl/certs directory in the first place. Here is a listing of that directory.
[root@ondemand2 conf.d]# ll /etc/ssl/certs/
lrwxrwxrwx. 1 root root 49 Jun 22 2020 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root 55 Jun 22 2020 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-------. 1 root root 4098 Jul 26 12:41 localhost.crt
-rw-------. 1 root root 2874 Aug 2 16:56 ondemand_rc_colorado_edu_cert.crt
-rw-------. 1 root root 5660 Aug 3 09:39 ondemand_rc_colorado_edu_interm.crt
-rw-------. 1 root root 2281 Jul 26 12:26 postfix.pem
Ok, this is looking more familiar. I think at this point the issue has to do with the Certificate Chain. The fix I’ve seen work previously is to
ondemand_rc_colorado_edu_interm.crt into the site cert:
cat ondemand_rc_colorado_edu_interm.crt >> ondemand_rc_colorado_edu_cert.crt
Then, remove the
SSLCertificateChainFile entry in your
ood_portal.yml and see if this now works.
So to be clear, your
ood_portal.yml will only have two entries now under
ssl, the SSLCertificateFile and the SSLCertificateKeyFile, with the SSLCertificateChainFile having been removed and the contents of the CAChain having been added to the CertificateFile itself.
Hopefully that will clear this latest error.
Yes that did it. Can now login and see that everything is working as expected. Thanks for the help.