OOD basic installation

Get Help
1

Hello OSC.
This is a fresh install of ondemand-release-web-1.8-1.noarch.rpm on CentOS7.7 VM.
Followed 1. Install Software From RPM, 3. Modify System Security/Firewall, 4. Start Services guided in Installation — Open OnDemand 1.8.12 documentation.
Installation seems to have gone well except when I try to http access using the VM ip, it redirects to the dashboard (/pun/sys/dashboard), then I get 500 error.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.

If I try to https access, it returns Apache test page.
/var/log/ondemand-nginx is empty.
Did I miss any important steps? How to make it work?

2

What are you using for authentication? Are you just trying to login with a default user?

/var/log/ondemand-nginx is where you will find the logs for per user sessions but OOD hasn’t gotten that far so there is nothing to log.

If this is a centos7 install, you should find the http server logs in /var/log/httpd24/....

3

I didn’t get to the auth setting step yet and trying to login with a default user.

/var/log/httpd24/error_log
[Wed Apr 14 03:07:11.588393 2021] [core:notice] [pid 13830] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Apr 14 03:07:11.589340 2021] [suexec:notice] [pid 13830] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Wed Apr 14 03:07:11.673714 2021] [http2:warn] [pid 13830] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Wed Apr 14 03:07:11.673726 2021] [http2:warn] [pid 13830] AH02951: mod_ssl does not seem to be enabled
[Wed Apr 14 03:07:11.674242 2021] [lbmethod_heartbeat:notice] [pid 13830] AH02282: No slotmem from mod_heartmonitor
[Wed Apr 14 03:07:11.677559 2021] [mpm_prefork:notice] [pid 13830] AH00163: Apache/2.4.34 (Red Hat) configured – resuming normal operations
[Wed Apr 14 03:07:11.677591 2021] [core:notice] [pid 13830] AH00094: Command line: ‘/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND’
[Wed Apr 14 03:13:30.381077 2021] [mpm_prefork:notice] [pid 13830] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Apr 14 03:13:30.814227 2021] [core:notice] [pid 15383] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Apr 14 03:13:30.815450 2021] [suexec:notice] [pid 15383] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Wed Apr 14 03:13:30.873251 2021] [http2:warn] [pid 15383] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Wed Apr 14 03:13:30.873807 2021] [lbmethod_heartbeat:notice] [pid 15383] AH02282: No slotmem from mod_heartmonitor
[Wed Apr 14 03:13:30.877349 2021] [mpm_prefork:notice] [pid 15383] AH00163: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips configured – resuming normal operations
[Wed Apr 14 03:13:30.877381 2021] [core:notice] [pid 15383] AH00094: Command line: ‘/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND’
[Wed Apr 14 03:18:46.597799 2021] [mpm_prefork:notice] [pid 15383] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Apr 14 03:19:40.669870 2021] [core:notice] [pid 2050] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Apr 14 03:19:40.671935 2021] [suexec:notice] [pid 2050] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Wed Apr 14 03:19:40.700471 2021] [http2:warn] [pid 2050] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Wed Apr 14 03:19:40.701427 2021] [lbmethod_heartbeat:notice] [pid 2050] AH02282: No slotmem from mod_heartmonitor
[Wed Apr 14 03:19:40.708276 2021] [mpm_prefork:notice] [pid 2050] AH00163: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips configured – resuming normal operations
[Wed Apr 14 03:19:40.708307 2021] [core:notice] [pid 2050] AH00094: Command line: ‘/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND’

/var/log/httpd24/error.log
[Wed Apr 14 03:17:14.668733 2021] [authn_core:error] [pid 15386] [client :51356] AH01796: AuthType openid-connect configured without corresponding module
[Wed Apr 14 03:17:24.231303 2021] [authn_core:error] [pid 15387] [client :51360] AH01796: AuthType openid-connect configured without corresponding module
[Wed Apr 14 03:19:40.705005 2021] [auth_openidc:warn] [pid 2050] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCProviderMetadataURL SHOULD be “https” for security reasons!
[Wed Apr 14 03:19:40.705118 2021] [auth_openidc:warn] [pid 2050] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCRedirectURI SHOULD be “https” for security reasons (moreover: some Providers may reject non-HTTPS URLs)
[Wed Apr 14 04:29:05.229830 2021] [auth_openidc:error] [pid 2055] [client :45012] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http:///
[Wed Apr 14 04:29:05.229886 2021] [auth_openidc:error] [pid 2055] [client :45012] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http:///
[Wed Apr 14 05:13:22.874752 2021] [auth_openidc:error] [pid 2054] [client :51672] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 05:13:22.874804 2021] [auth_openidc:error] [pid 2054] [client :51672] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 05:13:24.117908 2021] [auth_openidc:error] [pid 2054] [client :51712] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/
[Wed Apr 14 05:13:24.117962 2021] [auth_openidc:error] [pid 2054] [client :51712] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/
[Wed Apr 14 08:21:10.060078 2021] [auth_openidc:error] [pid 32365] [client :39726] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 08:21:10.060148 2021] [auth_openidc:error] [pid 32365] [client :39726] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 13:11:27.592683 2021] [auth_openidc:error] [pid 19187] [client :44212] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Wed Apr 14 13:11:27.592761 2021] [auth_openidc:error] [pid 19187] [client :44212] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Wed Apr 14 16:27:58.971446 2021] [auth_openidc:error] [pid 2054] [client :41322] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/
[Wed Apr 14 16:27:58.971506 2021] [auth_openidc:error] [pid 2054] [client :41322] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/
[Wed Apr 14 20:33:16.262544 2021] [auth_openidc:error] [pid 2053] [client :53252] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 20:33:16.262631 2021] [auth_openidc:error] [pid 2053] [client :53252] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 20:33:16.777720 2021] [auth_openidc:error] [pid 32364] [client :53312] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/
[Wed Apr 14 20:33:16.777776 2021] [auth_openidc:error] [pid 32364] [client :53312] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/
[Wed Apr 14 21:26:48.398188 2021] [auth_openidc:error] [pid 32364] [client :49158] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Wed Apr 14 21:26:48.398241 2021] [auth_openidc:error] [pid 32364] [client :49158] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Wed Apr 14 21:33:00.127699 2021] [auth_openidc:error] [pid 19188] [client :38810] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Wed Apr 14 21:33:00.127763 2021] [auth_openidc:error] [pid 19188] [client :38810] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Wed Apr 14 22:09:02.331455 2021] [auth_openidc:error] [pid 4533] [client :55044] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 22:09:02.331538 2021] [auth_openidc:error] [pid 4533] [client :55044] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Wed Apr 14 23:16:09.806390 2021] [auth_openidc:error] [pid 2052] [client :36900] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http:///
[Wed Apr 14 23:16:09.806451 2021] [auth_openidc:error] [pid 2052] [client :36900] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http:///
[Thu Apr 15 01:10:56.407188 2021] [auth_openidc:error] [pid 32760] [client :59566] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 01:10:56.407243 2021] [auth_openidc:error] [pid 32760] [client :59566] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 02:43:44.754234 2021] [auth_openidc:error] [pid 2052] [client :55434] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 02:43:44.754286 2021] [auth_openidc:error] [pid 2052] [client :55434] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 03:11:12.962100 2021] [auth_openidc:error] [pid 32365] [client :50064] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 03:11:12.962164 2021] [auth_openidc:error] [pid 32365] [client :50064] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 10:14:26.589595 2021] [core:error] [pid 27275] [client :1798] AH00126: Invalid URI in request GET /sdk/…/…/…/…/…/…/…/etc/vmware/hostd/vmInventory.xml HTTP/1.1
[Thu Apr 15 10:14:27.213197 2021] [core:error] [pid 2052] [client :1799] AH00126: Invalid URI in request GET /sdk/%2E%2E/%2E%2E/%2E%2E/%2E%2E/%2E%2E/%2E%2E/%2E%2E/etc/vmware/hostd/vmInventory.xml HTTP/1.1
[Thu Apr 15 10:14:27.830530 2021] [core:error] [pid 32365] [client :1800] AH00126: Invalid URI in request GET /…/…/…/…/…/…/…/…/…/…/etc/passwd HTTP/1.1
[Thu Apr 15 10:14:28.366616 2021] [core:error] [pid 21129] [client :1801] AH00126: Invalid URI in request GET /…/…/…/…/…/…/…/…/…/…/boot.ini http/1.1
[Thu Apr 15 10:14:29.296192 2021] [cgi:error] [pid 32760] [client :1802] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/mj_wwwusr
[Thu Apr 15 10:14:29.308543 2021] [cgi:error] [pid 32760] [client :1802] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/vcs
[Thu Apr 15 10:14:34.636305 2021] [cgi:error] [pid 4533] [client :1807] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/ffileman.cgi
[Thu Apr 15 10:14:35.362373 2021] [cgi:error] [pid 27275] [client :1808] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/ck
[Thu Apr 15 10:14:35.362417 2021] [cgi:error] [pid 27275] [client :1808] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/masterCGI
[Thu Apr 15 10:14:35.368453 2021] [cgi:error] [pid 27275] [client :1808] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/awstats.pl
[Thu Apr 15 10:14:36.351537 2021] [cgi:error] [pid 2052] [client :1809] AH02811: script not found or unable to stat: /opt/rh/httpd24/root/var/www/cgi-bin/image
[Thu Apr 15 12:50:50.101975 2021] [auth_openidc:error] [pid 15313] [client :49812] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 12:50:50.102069 2021] [auth_openidc:error] [pid 15313] [client :49812] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 14:27:02.991902 2021] [auth_openidc:error] [pid 32760] [client :32928] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied), referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Thu Apr 15 14:27:02.991962 2021] [auth_openidc:error] [pid 32760] [client :32928] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration, referer: http://:80/?s=/Index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1]=__HelloThinkPHP
[Thu Apr 15 14:27:04.018805 2021] [auth_openidc:error] [pid 32364] [client :55856] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 14:27:04.018918 2021] [auth_openidc:error] [pid 32364] [client :55856] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 16:26:42.478865 2021] [auth_openidc:error] [pid 4533] [client :54448] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 16:26:42.478923 2021] [auth_openidc:error] [pid 4533] [client :54448] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 17:59:30.258443 2021] [auth_openidc:error] [pid 4533] [client :36444] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 17:59:30.258507 2021] [auth_openidc:error] [pid 4533] [client :36444] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 18:22:23.625758 2021] [auth_openidc:error] [pid 2052] [client :54288] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 18:22:23.625817 2021] [auth_openidc:error] [pid 2052] [client :54288] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 19:55:23.512439 2021] [auth_openidc:error] [pid 22379] [client :45342] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 19:55:23.512498 2021] [auth_openidc:error] [pid 22379] [client :45342] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 20:21:17.362247 2021] [auth_openidc:error] [pid 21129] [client :51340] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 20:21:17.362325 2021] [auth_openidc:error] [pid 21129] [client :51340] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 21:38:15.907163 2021] [auth_openidc:error] [pid 27275] [client :42516] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 21:38:15.907261 2021] [auth_openidc:error] [pid 27275] [client :42516] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Thu Apr 15 23:58:05.866559 2021] [auth_openidc:error] [pid 22379] [client :37984] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Thu Apr 15 23:58:05.866621 2021] [auth_openidc:error] [pid 22379] [client :37984] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Fri Apr 16 02:08:27.795538 2021] [auth_openidc:error] [pid 32364] [client :55254] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Fri Apr 16 02:08:27.795594 2021] [auth_openidc:error] [pid 32364] [client :55254] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Fri Apr 16 02:08:33.780933 2021] [auth_openidc:error] [pid 21129] [client :55253] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Fri Apr 16 02:08:33.780991 2021] [auth_openidc:error] [pid 21129] [client :55253] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration
[Fri Apr 16 02:09:20.155303 2021] [auth_openidc:error] [pid 22379] [client :55291] oidc_util_http_call: curl_easy_perform() failed on: http://:5556/.well-known/openid-configuration (Failed to connect to : Permission denied)
[Fri Apr 16 02:09:20.155421 2021] [auth_openidc:error] [pid 22379] [client :55291] oidc_provider_static_config: could not retrieve metadata from url: http://:5556/.well-known/openid-configuration

4

What OS are you on? Are you using the selinux bits?

Based on the output it looks like you might be trying to configure some openid connect right now.

Can you send your ood-portal.yml output?

5

@mjbludwig thanks for the support! @BlueFooted Hi and welcome!

This entry is very telling. Somehow there’s no host in that URL. I’m guessing OIDCProviderMetadataURL in the apache config file isn’t right.

You likely need to specify this config in ood_portal.yml. I have it set to localhost, you should set it to the actual servername.

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
servername: localhost

But I believe you may also have found a bug where that seems to be required for dex, so there may be some extra finagling we can do here to alert folks that is is required to be set or default to localhost (instead of nil apparently).

6

Hi Morgan.
CentOS 7.7.1908. I didn’t aware but found selinux is enabled by default today. So I installed ondemand-selinux package today.
All the settings in ood_portal.yml, were default (commented). Today, I set servername to public DNS name, ssl to self-signed certificate, oidc_provider_metadata_url to https://<public-DNS-name>:5554/.well-known/openid-configuration.
Please find the error.log after today’s updates below.

7

Hi Jeff.
Updated servername to public DNS name, ssl to self-signed certificate, oidc_provider_metadata_url to https://<public-DNS-name>:5554/.well-known/openid-configuration. But still getting the 500 Internal Server Error.
New /var/log/httpd24/error.log

[Sun Apr 18 03:46:01.837706 2021] [auth_openidc:warn] [pid 2050] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCProviderMetadataURL SH
OULD be "https" for security reasons!
[Sun Apr 18 03:46:01.837715 2021] [auth_openidc:warn] [pid 2050] oidc_check_config_openid_openidc: the URL scheme (http) of the configured OIDCRedirectURI SHOULD be 
"https" for security reasons (moreover: some Providers may reject non-HTTPS URLs)
[Sun Apr 18 05:51:17.761541 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-2>:47904] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-o
penondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied), referer: ht
tp://<public-IP-address>:80/
[Sun Apr 18 05:51:17.761605 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-2>:47904] oidc_provider_static_config: could not retrieve metadata from url: ht
tp://<private-DNS-name>:5556/.well-known/openid-configuration, referer: http://<public-IP-address>:80/
[Sun Apr 18 05:51:18.220084 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-2>:47944] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-o
penondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied), referer: ht
tp://<public-IP-address>:80/?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=__HelloThinkPHP
[Sun Apr 18 05:51:18.220137 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-2>:47944] oidc_provider_static_config: could not retrieve metadata from url: ht
tp://<private-DNS-name>:5556/.well-known/openid-configuration, referer: http://<public-IP-address>:80/?s=/Index/\\th
ink\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=__HelloThinkPHP
[Sun Apr 18 07:53:36.254904 2021] [auth_openidc:error] [pid 21789] [client <client-IP-address-3>:48766] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-
openondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied)
[Sun Apr 18 07:53:36.254960 2021] [auth_openidc:error] [pid 21789] [client <client-IP-address-3>:48766] oidc_provider_static_config: could not retrieve metadata from url: h
ttp://<private-DNS-name>:5556/.well-known/openid-configuration
[Sun Apr 18 08:23:09.771382 2021] [auth_openidc:error] [pid 28888] [client <client-IP-address-4>:49258] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-op
enondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied), referer: htt
p://<public-IP-address>:80/
[Sun Apr 18 08:23:09.771445 2021] [auth_openidc:error] [pid 28888] [client <client-IP-address-4>:49258] oidc_provider_static_config: could not retrieve metadata from url: htt
p://<private-DNS-name>:5556/.well-known/openid-configuration, referer: http://<public-IP-address>:80/
[Sun Apr 18 19:20:12.350427 2021] [auth_openidc:error] [pid 28895] [client <client-IP-address-5>:51952] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm
-openondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied)
[Sun Apr 18 19:20:12.350496 2021] [auth_openidc:error] [pid 28895] [client <client-IP-address-5>:51952] oidc_provider_static_config: could not retrieve metadata from url: 
http://<private-DNS-name>:5556/.well-known/openid-configuration
[Sun Apr 18 20:42:09.711092 2021] [auth_openidc:error] [pid 28856] [client <client-IP-address-6>:43923] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-o
penondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied), referer: ht
tp://<public-IP-address>/
[Sun Apr 18 20:42:09.711151 2021] [auth_openidc:error] [pid 28856] [client <client-IP-address-6>:43923] oidc_provider_static_config: could not retrieve metadata from url: ht
tp://<private-DNS-name>:5556/.well-known/openid-configuration, referer: http://<public-IP-address>/
[Sun Apr 18 23:18:18.330359 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-7>:45066] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm-ope
nondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied), referer: http
://<public-IP-address>:80/?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=__HelloThinkPHP
[Sun Apr 18 23:18:18.330419 2021] [auth_openidc:error] [pid 28873] [client <client-IP-address-7>:45066] oidc_provider_static_config: could not retrieve metadata from url: http
://<private-DNS-name>:5556/.well-known/openid-configuration, referer: http://<public-IP-address>:80/?s=/Index/\\thin
k\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=__HelloThinkPHP
[Mon Apr 19 00:07:09.861265 2021] [auth_openidc:error] [pid 20784] [client <client-IP-address-1>:61924] oidc_util_http_call: curl_easy_perform() failed on: http://marketvm
-openondemand.5xl4sbrmsfxupnq3na4ffuukqe.ix.internal.cloudapp.net:5556/.well-known/openid-configuration (Failed to connect to <OOD-private-IP-address>: Permission denied)
[Mon Apr 19 00:07:09.861396 2021] [auth_openidc:error] [pid 20784] [client <client-IP-address-1>:61924] oidc_provider_static_config: could not retrieve metadata from url: 
http://<private-DNS-name>:5556/.well-known/openid-configuration
8

OK, one thing we’d need to triage is if this URL is accessable. The logs say permission denied, so I’m guessing it’s returning a 403. Do you have dex behind apache? Or the self signed certificate could be giving you trouble and that’s why it’s denying persmission (it won’t accept your certificates).

Not quite sure why it’s trying to access plain http instead of https though.

But I’d check to see if that URL to well-known actually works through curl.

curl "http://<the dex host>:5556/.well-known/openid-configuration"

Though this RHEL issue says you need to set this SE linux bool setsebool httpd_can_network_connect=on
https://bugzilla.redhat.com/show_bug.cgi?id=1386799

9

Don’t know what having dex behind apache, but ondemand-dex.x86_64 is installed. Does it mean any dex configuration on ondemand or apache?
Curl check returns the following.

$ curl http://<private-DNS-name>:5556/.well-known/openid-configuration
{
  "issuer": "https://<public-DNS-name>:5554",
  "authorization_endpoint": "https://<public-DNS-name>:5554/auth",
  "token_endpoint": "https://<public-DNS-name>:5554/token",
  "jwks_uri": "https://<public-DNS-name>:5554/keys",
  "userinfo_endpoint": "https://<public-DNS-name>:5554/userinfo",
  "response_types_supported": [
    "code"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "groups",
    "profile",
    "offline_access"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic"
  ],
  "claims_supported": [
    "aud",
    "email",
    "email_verified",
    "exp",
    "iat",
    "iss",
    "locale",
    "name",
    "sub"
  ]
}

For SE Linux bool, current httpd settings

$ getsebool -a | grep httpd
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> on
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> on
httpd_serve_cobbler_files --> off
httpd_setrlimit --> on
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off
10

I’m sure you’ve restarted apache after turning SELinux on or enabling those flags. I wonder if you have to reboot the entire machine for them to take affect?

The only thing this message comes back as is that apache cannot open a socket because it’s not allowed to.

(Failed to connect to <OOD-private-IP-address>: Permission denied)

You’ve clearly indicated that the right boolean is turned on. I wonder if there are messages in /var/log/audit/audit.log or /var/log/messages our journalctl that can help us traige this?

11

selinux is oh so fickle. I am bad and turn it off almost everywhere. That being said, I do not believe a restart is needed after installing the ondemand-selinux bits but one could test by running setenforce 0 and see if that helps. That effectively turn selinux off and hopefully let you know if that is an issue or not.

12

Hi Morgan.
setenforce 0 didn’t help.

13

There is something painfully obvious we’re missing here. You ran the curl command from the same host?

OK. Maybe it’s not selinux and in fact some firewall.

I want to get two things cleared up for sure. First if curl works from cli under the same conditions. Does curl from the machine that hosts apache to the same url? Does it work as the apache user?

Secondly, if you have https setup, and the servername is the public dns name, why’s it using the private dns name and plain http to get well known configs?

I’m guessing we should force it to only use https so that we can at least rule out any misalignments and so that you don’t have to open a plain http port.

I believe they should look in part like this. Specifying the servername and part of the dex config to indicate that it’s ssl only with some certificates.

servername: 'public_servername'
ssl:
  - 'SSLCertificateFile "/some/cert"'
  - 'SSLCertificateKeyFile "/some/key"'
dex:
  ssl: true
  https_port: "5554"
  tls_cert: /some/cert
  tls_key: /some/key
14

Now I found what ‘having dex behind apache’ means. Apache, ood, dex are all in one host.

the dex config was commented, now uncommented like above.

Still got the same 500 Internal Server Error

  1. curl well-known from my laptop using public DNS name
% curl http://<public-DNS-name>/.well-known/openid-configuration
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://<public-DNS-name>:443/.well-known/openid-configuration">here</a>.</p>
</body></html>

% curl https://<public-DNS-name>/.well-known/openid-configuration
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
  1. curl well-know from other machine on the same subnet with ood machine using private DNS name
$ curl http://<private-DNS-name>:5556/.well-known/openid-configuration
{
  "issuer": "https://<public-DNS-name>:5554",
  "authorization_endpoint": "https://<public-DNS-name>:5554/auth",
  "token_endpoint": "https://<public-DNS-name>:5554/token",
  "jwks_uri": "https://<public-DNS-name>:5554/keys",
  "userinfo_endpoint": "https://<public-DNS-name>:5554/userinfo",
  "response_types_supported": [
    "code"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "scopes_supported": [
    "openid",
    "email",
    "groups",
    "profile",
    "offline_access"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic"
  ],
  "claims_supported": [
    "aud",
    "email",
    "email_verified",
    "exp",
    "iat",
    "iss",
    "locale",
    "name",
    "sub"
  ]
}

$ curl https://<private-DNS-name>:5554/.well-known/openid-configuration
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
  1. curl well-know from other machine on the same subnet with ood machine using public DNS name
    same result with #1
15

Hey, sorry for the delay. Yea forget my question about dex being behind apache. I don’t believe that’s your setup.

Can you try from the same host? We need to replicate the same exact network route. There may be some iptable rule we need to implement, because even through it’s localhost, we’re not using it like that. So maybe there’s some firewall stopping us. This is the only guess I have. It does not seem to be an selinux issue given you turned it off and everything. I’m suspecting now it’s a networking thing, but we need to confirm that curl from the same host that apache runs works.

The only thing left after that is your self signed certificates. This is the next only thing I can think. If you’re in azure you should be able to setup letsencrypt certificates so you don’t have insecure self-signed certificates.

16

Decided to deploy OOD using scripts from a colleague and it worked.
Now I can login to the dashboard.
Next step is to setup Azure AD authentication using oidc.
I updated /etc/ood/config/ood_portal.yml as guided in OpenID Connect
Browse to the OOD dashboard redirects to Azure AD authentication and after authenticated, it redirects to /pun/sys/dashboard, but returns error

Error – can’t find user for <myEmail>
Run ‘nginx_stage --help’ to see a full list of available command line options.

What am I missing?

17

You can either use the user mapping script shim to convert email to local uid (assuming you can trust that the Azure AD provided email is trusted) (2. Setup User Mapping — Open OnDemand 1.8.12 documentation), or you can request a different OpenID Connect data field like “preferred_username” from the Azure AD response in the hopes that the result matches a local uid for your OOD environment.

Simply put, you need to map the result from AD to a local Linux user that exists in your OOD environment.

Another way to do it although it sounds funky is to add the same Azure AD as a user federation mechanism in Keycloak as read-only. This would defer the actual authentication using OpenID Connect to the upstream AD but then Keycloak would take the response (email currently) and map it back to the same AD where it would pull a username from which should then match an existing user in OOD, assuming they exist on the local system.

2 Likes
18

Hi.
I’ve faced with the same problem as yours.

Can I ask you for the deployment scripts you mentioned?

Thanks.