Dex Not Starting With SSL

Get Help
1

After doing a fresh install on RHEL 8.6, Dex doesn’t seem to be starting with SSL support. I see these errors:

[Thu May 25 14:44:09.041069 2023] [auth_openidc:error] [pid 168028:tid 140736280393472] [client 172.27.96.248:31820] oidc_util_http_call: curl_easy_perform() failed on: https://apps.talon.und.edu/dex/.well-known/openid-configuration (SSL certificate problem: unable to get local issuer certificate)
[Thu May 25 14:44:09.041108 2023] [auth_openidc:error] [pid 168028:tid 140736280393472] [client 172.27.96.248:31820] oidc_provider_static_config: could not retrieve metadata from url: https://apps.talon.und.edu/dex/.well-known/openid-configuration

I’m not sure what to do here. Is there another option I need to set in ood_portal.yml?

2

Are you using self signed certificates? You have to be sure that the certificates you generated for dex come from the same certificate authority that apache has. I’m not 100% sure what that means exactly, but self signed certificates always give dex problems. I’ll look for similar topics to link here.

3

This may help.

4

Jeff,

We are not using any self-signed certificates in the ood_portal config.

5

Googling this search string shows a few results. They seem to be related to the CA certificate and apache’s permission to find them. Stack overflow questions have some openssl utility commands to test this.

apache SSL certificate problem: unable to get local issuer certificate

Where did you get your certificates from and do you have the certificate authority in the right place? You may just need to issue update-ca-trust extract.

6

Jeff,

After adding the SSLCertificateChainFile to the ood_portal.yml file, I’m now getting a login page. But after logging in, I get the attached error.

ood_error
ood_error1019×697 25 KB

7

Check in /var/log/ondemand-nginx/$USER/error.log for the error message(s). There may also be a tmp html file in /tmp that you can download and view, though the error log may say the same thing.

8

This is the error log.

[ N 2023-05-26 10:20:44.9282 298503/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2023-05-26 10:20:44.9469 298506/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2023-05-26 10:20:44.9470 298506/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2023-05-26 10:20:45.0716 298506/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 298506
App 298650 output: /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:480:in `materialize': Could not find rails-5.2.8.1, jquery-rails-4.4.0, timecop-0.9.5, thor-0.19.1, dotenv-rails-2.7.6, font-awesome-sass-5.12.0, redcarpet-3.5.1, addressable-2.8.0, data-confirm-modal-1.6.3, turbolinks-5.2.1, nokogiri-1.12.5, ood_support-0.0.3, ood_appkit-1.1.5, ood_core-0.23.2, sinatra-2.2.0, sinatra-contrib-2.2.0, erubi-1.10.0, dalli-3.2.0, webpacker-5.4.3, actioncable-5.2.8.1, actionmailer-5.2.8.1, actionpack-5.2.8.1, actionview-5.2.8.1, activejob-5.2.8.1, activemodel-5.2.8.1, activerecord-5.2.8.1, activestorage-5.2.8.1, activesupport-5.2.8.1, railties-5.2.8.1, tilt-2.0.10, public_suffix-4.0.7, i18n-1.11.0, turbolinks-source-5.2.0, mini_portile2-2.6.1, racc-1.6.0, mustermann-1.1.1, rack-2.2.4, rack-protection-2.2.0, rack-proxy-0.7.2, semantic_range-3.0.0, nio4r-2.5.8, mail-2.7.1, rack-test-2.0.2, rails-html-sanitizer-1.4.3, globalid-1.0.0, arel-9.0.0, concurrent-ruby-1.1.10, minitest-5.15.0, tzinfo-1.2.9, loofah-2.18.0, thread_safe-0.3.6, rb-fsevent-0.11.1 in any of the sources (Bundler::GemNotFound)
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:185:in `specs'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:233:in `specs_for'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:18:in `setup'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:150:in `setup'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `block in <top (required)>'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:136:in `with_level'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:88:in `silence'
App 298650 output:      from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `<top (required)>'
App 298650 output:      from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
App 298650 output:      from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
App 298650 output:      from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
App 298650 output: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- bundler/setup (LoadError)
App 298650 output:      from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
[ E 2023-05-26 10:20:45.3695 298506/T3v age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/ood/apps/sys/dashboard: The application process exited prematurely.
  Error ID: 0803bd37
  Error details saved to: /tmp/passenger-error-JBVKIY.html

[ E 2023-05-26 10:20:45.3746 298506/T8 age/Cor/Con/CheckoutSession.cpp:283 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is 0803bd37. Please see earlier logs for details about the error.
9

That is very odd indeed. You are running 2.0? on what operating system? Did you install it from the packages (rpm or deb)?

10

I see you’re on RHEL/8. You’re installing 2.0.x on RHEL 8.6?

11

I’m running 3.0 on RHEL 8.6. It was installed from the yum repository.

12

It’s looking for rails-5.2.8.1 which is the version of rails for 2.0. OOD 3.0 uses rails-6.1. So it’s unclear to me why it’s searching for rails 5.2.

Can you do a spot check on this file?

[johrstrom ~()]  head -n 5 /var/www/ood/apps/sys/dashboard/Gemfile
source 'https://rubygems.org'


# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '6.1.7.3'
13

This is the output of that command:

[root@login1 ~]# head -n 5 /var/www/ood/apps/sys/dashboard/Gemfile
source 'https://rubygems.org'


# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '5.2.8.1'

This is the output of yum list installed | grep ondemand:

[root@login1 ~]# yum list installed | grep ondemand
cjose.x86_64                                   0.6.1-1.el8                                         @ondemand-web
mod_auth_openidc.x86_64                        2.4.5-1.el8                                         @ondemand-web
ondemand.x86_64                                3.0.1-1.el8                                         @ondemand-web
ondemand-apache.x86_64                         3.0.0-1.el8                                         @ondemand-web
ondemand-dex.x86_64                            2.32.0-1.el8                                        @ondemand-web
ondemand-gems-3.0.1-1.x86_64                   3.0.1-1.el8                                         @ondemand-web
ondemand-nginx.x86_64                          1.20.2-1.p6.0.14.ood3.0.0.el8                       @ondemand-web
ondemand-nodejs.x86_64                         3.0.0-1.el8                                         @ondemand-web
ondemand-passenger.x86_64                      6.0.14-1.ood3.0.0.el8                               @ondemand-web
ondemand-release-web.noarch                    3.0-1                                               @@commandline
ondemand-ruby.x86_64                           3.0.0-1.el8                                         @ondemand-web
ondemand-runtime.x86_64                        3.0.0-1.el8                                         @ondemand-web
14

OK that is really strange. You say this was a fresh install - no other OOD installation was here previously?

I just tried this - pulled the rpm down, unpacked it and checked that file and confirmed the RPM package has the correct file.

wget https://yum.osc.edu/ondemand/3.0/web/el8/x86_64/ondemand-3.0.0-1.el8.x86_64.rpm
rpm2cpio ondemand-3.0.0-1.el8.x86_64.rpm | cpio -idmv
head -n 5 var/www/ood/apps/sys/dashboard/Gemfile

Do you have a mount or something on top of that directory? Something on your system has overwritten the ondemand-3.0 RPM with the ondemand-2.0 package.

15

It was a fresh install but I copied over our previous configuration files so I didn’t have to set everything up from scratch. The two directories I copied were /etc/ood/config and /var/www/ood/apps/sys

16

Do not copy or modify this directory.

/etc/ood/config is fine, packages won’t update/overwrite these files, but they do overwrite files in /var/www/ood/apps.

17

Should I put my apps in /etc/ood/config/apps then?

18

Sorry - I forgot that’s where other apps are too.

You can add new directories into /var/www/ood/apps/sys, but you shouldn’t overwrite/change what the RPM provides.

So yes, you can copy your other applications into this directory, just don’t copy the applications (directories) that the RPM ships.

19

Should I just reinstall the ondemand package to restore those directories?

20

Yes that should take care of it.