# Error internal 500 after change servername and adding certificates

**URL:** <https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677>\
**Category:** Get Help\
**Created:** [May 3, 2023, 8:03am UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677 "2023-05-03T08:03:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![vandsten](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/vandsten/32/1543_2.png) [@vandsten](https://discourse.openondemand.org/u/vandsten)\
**Post date:** [May 3, 2023, 8:03am UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/1 "2023-05-03T08:03:33Z")

</div>

Hi,

We have an OOD implementation with the following features:

- OOD version 3.0
- Rocky Linux 8.5

OOD was working perfectly, but we want to announce to the users its availability. We followed the following steps:

1. We have changed the servername to its final value.
2. We added the certificates.
3. We have launched an update\_ood\_portal
4. We restarted ondemand-dex and httpd.

And we got an “Internal Server Error”, and in the log output:

```auto
 - [03/May/2023:09:48:25 +0200] "GET / HTTP/1.1" 302 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
 - [03/May/2023:09:48:25 +0200] "GET /pun/sys/dashboard HTTP/1.1" 500 527 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
 - [03/May/2023:09:48:30 +0200] "GET /favicon.ico HTTP/1.1" 404 196 "https://myhpc.domain.es/pun/sys/dashboard" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36

```

And in the error\_log we have:

```auto
[Wed May 03 09:48:30.645786 2023] [auth_openidc:error] [pid 50875:tid 140181717690112] [client :46676] oidc_util_http_call: curl_easy_perform() failed on: https://myhpc.domain.es/dex/.well-known/openid-configuration (Connection timed out after 5000 milliseconds)
[Wed May 03 09:48:30.645859 2023] [auth_openidc:error] [pid 50875:tid 140181717690112] [client :46676] oidc_provider_static_config: could not retrieve metadata from url: https://myhpc.domain.es/dex/.well-known/openid-configuration

```

Regards

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 3, 2023, 1:44pm UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/2 "2023-05-03T13:44:32Z")

</div>

Timeouts are generally a connectivity issue. That is, you don’t seem to have connectivity to `https://myhpc.domain.es/dex/.well-known/openid-configuration` from the OnDemand instance. Seems like you need to open a network path from the OnDemand instance to that OIDC instance.

That looks like `dex` so maybe they’re on the same instance - `myhpc.domain.es`, but it can’t find itself? Perform a curl to that URL on that machine. Does it succeed?

---

<div class="post-metadata">

**Author:** ![vandsten](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/vandsten/32/1543_2.png) [@vandsten](https://discourse.openondemand.org/u/vandsten)\
**Post date:** [May 4, 2023, 12:14pm UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/3 "2023-05-04T12:14:18Z")

</div>

Hi Jeff,

I have tried connecting both from outside the ondemand server and from the ondemand server itself to the public web access both port 80 and 443, and no problem. I have tried with telnet and seen traffic captured with tcpdump, there is no restriction on network access.

But then I had a doubt with the CA issue and looking at the documentation I have tried to put the intermediate certificates in /etc/pki/ca-trust/source/anchors/ directory and then updating but when I do the curl I still get this:

```auto
# cp full_chain.crt /etc/pki/ca-trust/source/anchors/
# update-ca-trust

# curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
* Trying IP...
* TCP_NODELAY set
* Connected to myhpc.domain.es (IP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443

```

Regards

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [May 4, 2023, 4:25pm UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/4 "2023-05-04T16:25:54Z")

</div>

I’m not entirely sure what’s going on, but it’s clearly some SSL issue. Maybe an update to `update-ca-certificates` may help resolve?

Where’d you get the certificates from? I know for sure there are issues if they’re self signed. You need to make sure the certificate authority is in the right place if the certificates are self signed.

---

<div class="post-metadata">

**Author:** ![vandsten](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/vandsten/32/1543_2.png) [@vandsten](https://discourse.openondemand.org/u/vandsten)\
**Post date:** [June 9, 2023, 7:09am UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/5 "2023-06-09T07:09:34Z")

</div>

Hi Jeff,  
sorry for the delay in replying, I had to do other things.  
Sorry also for obfuscating the domain. It really is a valid GEANT domain.

I have been testing, I have launched curl and openssl s\_client from outside and from inside the internal network where the ondemand server is. From both I get good result similar to the following. Sorry for obfuscating it again.

```auto
pc@hostname:~$ curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
* Trying PUBLICIP:443...
* Connected to myhpc.domain.es (PUBLICIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=ES; ST=Madrid; O=ORG CN=myhpc.domain.es
* start date: Feb 17 00:00:00 2023 GMT
* expire date: Feb 17 23:59:59 2024 GMT
* subjectAltName: host "myhpc.domain.es" matched cert's "myhpc.domain.es"
* issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /dex/.well-known/openid-configuration HTTP/1.1
> Host: myhpc.domain.es
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 09 Jun 2023 06:48:40 GMT
< Server: Apache/2.4.37 (rocky) OpenSSL/1.1.1k
< Content-Security-Policy: frame-ancestors https://myhpc.domain.es;
< Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
< Content-Length: 1158
< Content-Type: application/json
< 
{
  "issuer": "https://myhpc.domain.es/dex",
  "authorization_endpoint": "https://myhpc.domain.es/dex/auth",
  "token_endpoint": "https://myhpc.domain.es/dex/token",
  "jwks_uri": "https://myhpc.domain.es/dex/keys",
  "userinfo_endpoint": "https://myhpc.domain.es/dex/userinfo",
  "device_authorization_endpoint": "https://myhpc.domain.es/dex/device/code",
  "grant_types_supported": [
    "authorization_code",
    "refresh_token",
    "urn:ietf:params:oauth:grant-type:device_code"
  ],
  "response_types_supported": [
    "code"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "code_challenge_methods_supported": [
    "S256",
    "plain"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "groups",
    "profile",
    "offline_access"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic",
    "client_secret_post"
  ],
  "claims_supported": [
    "iss",
    "sub",
    "aud",
    "iat",
    "exp",
    "email",
    "email_verified",
    "locale",
    "name",
    "preferred_username",
    "at_hash"
  ]
* Connection #0 to host myhpc.domain.es left intact

```

I can’t quite see the cause, whether I try from a machine outside the network through the firewall I get good results from curl. If I do it from a machine on the internal(private) network that is on the same network as the ondemand server then no problem. In the firewall NAT reflection is enabled to allow this so this should not be the cause. But when I do it from the ondemand server itself I get something like this.

```auto
[root@ondemandserver ~]# curl -vvvv https://myhpc.domain.es/dex/.well-known/openid-configuration
* Trying PUBLICIP...
* TCP_NODELAY set
* Connected to myhpc.domain.es (PUBLICIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to myhpc.domain.es:443 

```

It must be something silly and obvious but I can’t find the cause.

Best regards

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [June 9, 2023, 1:58pm UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/6 "2023-06-09T13:58:42Z")

</div>

See the difference in these 2 outputs. Specifically these 2 items - you CAfile and your CApath.

When it works, you have the correct certificates installed (ca-certificates.crt).

> [@vandsten](#):
>
> ```auto
> * CAfile: /etc/ssl/certs/ca-certificates.crt
> * CApath: /etc/ssl/certs
> 
> ```

When it fails, you do not have the appropriate certificates. No CApath and `ca-bundle.crt` isn’t what you need.

> [@vandsten](#):
>
> ```auto
> * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> CApath: none
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex015/uploads/osc/original/2X/b/bae70bd0ed39a3ae769c2108155f4cb3e9da8385.png) [@system](https://discourse.openondemand.org/u/system)\
**Post date:** [December 6, 2023, 1:58pm UTC](https://discourse.openondemand.org/t/error-internal-500-after-change-servername-and-adding-certificates/2677/7 "2023-12-06T13:58:42Z")

</div>

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.