Internal Server Error After Updating SSL Certificates

Get Help
1

I just updated the certificates on our OnDemand server and now Dex is producing an “internal server error” message. Usually when we get the “internal server error” page, I just restart the ondemand-dex service and the error goes away. However, no matter what I do, the “internal server error” page will not go away. I’ve checked the permissions and ownership on the certificate and key files and they are correct. The only error I can find is in the httpd logs. The error is below:

[auth_openidc:error] [pid 967753:tid 967917] [client ] oidc_util_http_call: curl_easy_perform() failed on: https://mydomain.edu/dex/.well-known/openid-configuration (error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt)
[auth_openidc:error] [pid 967753:tid 967917] [client ] oidc_provider_static_config: could not retrieve metadata from url: https://mydomain.edu/dex/.well-known/openid-configuration

We are running OnDemand version 3.1.16 on RHEL 9.4. No configs have been changed. The only change that was made was replacing the old expired certificate and key file with a new certificate and key file. Any assistance would be greatly appreciated.

2

Does this file /etc/pki/tls/certs/ca-bundle.crt exist? Maybe @tdockendorf knows more.

3

My only guess is the CA that signed your new certificate is not trusted by the system’s CA bundle. We run into this sometimes with LetsEncrypt when a new Intermediate or root CA is used. Also verify the ca-bundle exists at the path in the error per Jeff’s question.

4

The ca-bundle.crt file does exist. How would I update my systems ca-bundle.crt file?

5

I just checked the apache logs after reinstalling the ca-certificates package. The error has now changed. It is below:

[auth_openidc:error] [pid 1323926:tid 1323962] [client ] oidc_util_http_call: curl_easy_perform() failed on: https://mydomain.edu/dex/.well-known/openid-configuration (SSL certificate problem: unable to get local issuer certificate)
[auth_openidc:error] [pid 1323926:tid 1323962] [client ] oidc_provider_static_config: could not retrieve metadata from url: https://mydomain.edu/dex/.well-known/openid-configuration

Any assistance would be greatly appreciated as our production instance of OnDemand is down due to this. Thanks!

6

Looks like I didn’t reply properly. @jeff.ohrstrom and @tdockendorf, I have provided the information requested along with an update.

7

Unknown certificate authority when using openondemand-dex - #2 by tdockendorf may have some pointers on how to update the CA trust.

8

If updating the CA trust doesn’t fix then please provide /etc/ood/dex/config.yaml

9

@tdockendorf,

I checked for updates for the ca-certificates package but none were available. Our /etc/ood/dex/config.yaml file is below:


issuer: https://apps.talon.und.edu/dex
storage:
type: sqlite3
config:
file: “/etc/ood/dex/dex.db”
web:
http: localhost:5556
telemetry:
http: 0.0.0.0:5558
staticClients:

id: mydomain.edu
redirectURIs:

https://mydomain.edu/oidc
name: OnDemand
secret: <secret>
connectors:

type: ldap
id: ldap
name: LDAP
config:
host: ldap.mydomain.edu:636
insecureSkipVerify: false
bindDN: uid=ondemand,cn=users,cn=accounts,dc=mydomain,dc=edu
bindPW: <password>
userSearch:
baseDN: cn=users,cn=accounts,dc=mydomain,dc=edu
filter: “(objectClass=posixAccount)”
username: uid
idAttr: uid
emailAttr: mail
nameAttr: cn
preferredUsernameAttr: uid
groupSearch:
baseDN: cn=talon_users,cn=groups,cn=accounts,dc=mydomain,dc=edu
filter: “(objectClass=posixaccount)”
userMatchers:
- userAttr: DN
groupAttr: memberof
nameAttr: cn
oauth2:
skipApprovalScreen: true
enablePasswordDB: false
frontend:
dir: “/usr/share/ondemand-dex/web”
theme: talon
issuer: OnDemand
extra:
loginTitle: Log in with your credentials. If you do not have a CRC account,
you will need to submit a CRC account request at crc.mydomain.edu to access this
application.
loginButtonText: Log In
usernamePlaceholder: username (usually first.last)
passwordPlaceholder: password
10

Did you add the CA that signed the certificate to central trust store?

sudo cp <CA cert> /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract
11

Yes, that did not resolve the issue.

12

You may need to set SSLCertificateChainFile. Example:

$ sudo grep "ssl:" -A 3  /etc/ood/config/ood_portal.yml
ssl:
- SSLCertificateFile /etc/letsencrypt/live/ondemand.osc.edu/cert.pem
- SSLCertificateKeyFile /etc/letsencrypt/live/ondemand.osc.edu/privkey.pem
- SSLCertificateChainFile /etc/letsencrypt/live/ondemand.osc.edu/chain.pem
13

@tdockendorf,

SSLCertificateChain file was set previously. I took it out during troubleshooting. I have put it back and it’s set as follows:

‘SSLCertificateChainFile “/etc/pki/tls/certs/Intermediate_ca_root_256_R3.crt”’

Is the intermediate cert the correct cert to point at for the chain file?

14

If that file has intermediate cert(s) and root, then that’s what you want to use.

15

Ok, that’s what it’s set to but it’s still not working. I tried reinstalling the ondemand-dex package but that didn’t help either unfortunately.

16

Are you sure that’s the correct intermediate and root CA?

openssl verify -CAfile /etc/pki/tls/certs/Intermediate_ca_root_256_R3.crt <file used for SSLCertificateFile>

Not certain this will work depending how the intermediate and CA are chained together. If you google ways to validate the certificate against the CA + intermediates, lots of ways to verify you have the correct files.

17

@tdockendorf,

I get “unable to get local issuer certificate” for both the intermediate and root certificate when I run the command you provided. I will search around to try and verify that I have the correct files.

18

This issue has been resolved. As it turns out, we switched to a new certificate provider (Sectigo), and that provider is not trusted by Red Hat by default. Once we added their intermediate and root certificates to the CA trust and restarted the services, everything started working. Thanks @tdockendorf and @jeff.ohrstrom for your help on this!