Error internal 500 after change servername and adding certificates

Hi,

We have an OOD implementation with the following features:

  • OOD version 3.0
  • Rocky Linux 8.5

OOD was working perfectly, but we want to announce to the users its availability. We followed the following steps:

  1. We have changed the servername to its final value.
  2. We added the certificates.
  3. We have launched an update_ood_portal
  4. We restarted ondemand-dex and httpd.

And we got an “Internal Server Error”, and in the log output:

 - [03/May/2023:09:48:25 +0200] "GET / HTTP/1.1" 302 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
 - [03/May/2023:09:48:25 +0200] "GET /pun/sys/dashboard HTTP/1.1" 500 527 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
 - [03/May/2023:09:48:30 +0200] "GET /favicon.ico HTTP/1.1" 404 196 "https://myhpc.domain.es/pun/sys/dashboard" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36

And in the error_log we have:

[Wed May 03 09:48:30.645786 2023] [auth_openidc:error] [pid 50875:tid 140181717690112] [client :46676] oidc_util_http_call: curl_easy_perform() failed on: https://myhpc.domain.es/dex/.well-known/openid-configuration (Connection timed out after 5000 milliseconds)
[Wed May 03 09:48:30.645859 2023] [auth_openidc:error] [pid 50875:tid 140181717690112] [client :46676] oidc_provider_static_config: could not retrieve metadata from url: https://myhpc.domain.es/dex/.well-known/openid-configuration

Regards

Timeouts are generally a connectivity issue. That is, you don’t seem to have connectivity to https://myhpc.domain.es/dex/.well-known/openid-configuration from the OnDemand instance. Seems like you need to open a network path from the OnDemand instance to that OIDC instance.

That looks like dex so maybe they’re on the same instance - myhpc.domain.es, but it can’t find itself? Perform a curl to that URL on that machine. Does it succeed?

Hi Jeff,

I have tried connecting both from outside the ondemand server and from the ondemand server itself to the public web access both port 80 and 443, and no problem. I have tried with telnet and seen traffic captured with tcpdump, there is no restriction on network access.

But then I had a doubt with the CA issue and looking at the documentation I have tried to put the intermediate certificates in /etc/pki/ca-trust/source/anchors/ directory and then updating but when I do the curl I still get this:

# cp full_chain.crt /etc/pki/ca-trust/source/anchors/
# update-ca-trust

# curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
* Trying IP...
* TCP_NODELAY set
* Connected to myhpc.domain.es (IP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443

Regards

I’m not entirely sure what’s going on, but it’s clearly some SSL issue. Maybe an update to update-ca-certificates may help resolve?

Where’d you get the certificates from? I know for sure there are issues if they’re self signed. You need to make sure the certificate authority is in the right place if the certificates are self signed.

Hi Jeff,
sorry for the delay in replying, I had to do other things.
Sorry also for obfuscating the domain. It really is a valid GEANT domain.

I have been testing, I have launched curl and openssl s_client from outside and from inside the internal network where the ondemand server is. From both I get good result similar to the following. Sorry for obfuscating it again.

pc@hostname:~$ curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
*   Trying PUBLICIP:443...
* Connected to myhpc.domain.es (PUBLICIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=ES; ST=Madrid; O=ORG CN=myhpc.domain.es
*  start date: Feb 17 00:00:00 2023 GMT
*  expire date: Feb 17 23:59:59 2024 GMT
*  subjectAltName: host "myhpc.domain.es" matched cert's "myhpc.domain.es"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /dex/.well-known/openid-configuration HTTP/1.1
> Host: myhpc.domain.es
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 09 Jun 2023 06:48:40 GMT
< Server: Apache/2.4.37 (rocky) OpenSSL/1.1.1k
< Content-Security-Policy: frame-ancestors https://myhpc.domain.es;
< Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
< Content-Length: 1158
< Content-Type: application/json
< 
{
  "issuer": "https://myhpc.domain.es/dex",
  "authorization_endpoint": "https://myhpc.domain.es/dex/auth",
  "token_endpoint": "https://myhpc.domain.es/dex/token",
  "jwks_uri": "https://myhpc.domain.es/dex/keys",
  "userinfo_endpoint": "https://myhpc.domain.es/dex/userinfo",
  "device_authorization_endpoint": "https://myhpc.domain.es/dex/device/code",
  "grant_types_supported": [
    "authorization_code",
    "refresh_token",
    "urn:ietf:params:oauth:grant-type:device_code"
  ],
  "response_types_supported": [
    "code"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "code_challenge_methods_supported": [
    "S256",
    "plain"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "groups",
    "profile",
    "offline_access"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic",
    "client_secret_post"
  ],
  "claims_supported": [
    "iss",
    "sub",
    "aud",
    "iat",
    "exp",
    "email",
    "email_verified",
    "locale",
    "name",
    "preferred_username",
    "at_hash"
  ]
* Connection #0 to host myhpc.domain.es left intact

I can’t quite see the cause, whether I try from a machine outside the network through the firewall I get good results from curl. If I do it from a machine on the internal(private) network that is on the same network as the ondemand server then no problem. In the firewall NAT reflection is enabled to allow this so this should not be the cause. But when I do it from the ondemand server itself I get something like this.

[root@ondemandserver ~]# curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
*   Trying PUBLICIP...
* TCP_NODELAY set
* Connected to myhpc.domain.es (PUBLICIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 

It must be something silly and obvious but I can’t find the cause.

Best regards

See the difference in these 2 outputs. Specifically these 2 items - you CAfile and your CApath.

When it works, you have the correct certificates installed (ca-certificates.crt).

When it fails, you do not have the appropriate certificates. No CApath and ca-bundle.crt isn’t what you need.