Could not find concurrent-ruby-1.1.7 in any of the sources for dashboard

Hi all,

I am having an issue with a test user we have. The issue occurs when logging in. Other users can log in fine.

from /var/log/ondemand-nginx/fakeuser.m/error.log

App 15778 output: Error: The application encountered the following error: Could not find concurrent-ruby-1.1.7 in any of the sources (Bundler::GemNotFound)
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/spec_set.rb:91:in `block in materialize'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `map!'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `materialize'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup'
App 15778 output:     /opt/ood/ondemand/root/usr/share/gems/2.5/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>'
App 15778 output:     /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
App 15778 output:     /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
App 15778 output:     /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:363:in `activate_gem'
App 15778 output:     /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:221:in `block in run_load_path_setup_code'
App 15778 output:     /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:527:in `running_bundler'
App 15778 output:     /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:220:in `run_load_path_setup_code'
App 15778 output:     /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:86:in `load_app'
App 15778 output:     /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:116:in `block in <module:App>'
App 15778 output:     /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380:in `run_block_and_record_step_progress'
App 15778 output:     /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:115:in `<module:App>'
App 15778 output:     /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:28:in `<module:PhusionPassenger>'
App 15778 output:     /opt/rh/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb:27:in `<main>'
[ E 2021-01-27 13:17:55.0489 15943/T5r age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/ood/apps/sys/dashboard: The application encountered the following error: Could not find concurrent-ruby-1.1.7 in any of the sources (Bundler::GemNotFound)
  Error ID: 5cab7c61
  Error details saved to: /tmp/passenger-error-88PdTl.html

[ E 2021-01-27 13:17:55.0587 15943/T8 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-9] Cannot checkout session because a spawning error occurred. The identifier of the error is 5cab7c61. Please see earlier logs for details about the error.

Is there a procedure to rebundle? Or any other way to fix this?

/var/www/ood/apps/sys/dashboard/Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (5.2.4.4)
      actionpack (= 5.2.4.4)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
    actionmailer (5.2.4.4)
      actionpack (= 5.2.4.4)
      actionview (= 5.2.4.4)
      activejob (= 5.2.4.4)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (5.2.4.4)
      actionview (= 5.2.4.4)
      activesupport (= 5.2.4.4)
      rack (~> 2.0, >= 2.0.8)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.2)
    actionview (5.2.4.4)
      activesupport (= 5.2.4.4)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.3)
    activejob (5.2.4.4)
      activesupport (= 5.2.4.4)
      globalid (>= 0.3.6)
    activemodel (5.2.4.4)
      activesupport (= 5.2.4.4)
    activerecord (5.2.4.4)
      activemodel (= 5.2.4.4)
      activesupport (= 5.2.4.4)
      arel (>= 9.0)
    activestorage (5.2.4.4)
      actionpack (= 5.2.4.4)
      activerecord (= 5.2.4.4)
      marcel (~> 0.3.1)
    activesupport (5.2.4.4)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 0.7, < 2)
      minitest (~> 5.1)
      tzinfo (~> 1.1)
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    arel (9.0.0)
    autoprefixer-rails (8.6.5)
      execjs
    bootstrap-sass (3.4.1)
      autoprefixer-rails (>= 5.2.1)
      sassc (>= 2.0.0)
    bootstrap_form (2.7.0)
    browser (2.7.1)
    builder (3.2.4)
    byebug (11.1.3)
    climate_control (0.2.0)
    coffee-rails (4.2.2)
      coffee-script (>= 2.2.0)
      railties (>= 4.0.0)
    coffee-script (2.4.1)
      coffee-script-source
      execjs
    coffee-script-source (1.12.2)
    concurrent-ruby (1.1.7)
    crass (1.0.6)
    data-confirm-modal (1.6.3)
      railties (>= 3.0)
    dotenv (2.7.6)
    dotenv-rails (2.7.6)
      dotenv (= 2.7.6)
      railties (>= 3.2)
    dotiw (5.1.0)
      activesupport
      i18n
    erubi (1.9.0)
    execjs (2.7.0)
    ffi (1.13.1)
    font-awesome-sass (5.12.0)
      sassc (>= 1.11)
    globalid (0.4.2)
      activesupport (>= 4.2.0)
    i18n (1.8.5)
      concurrent-ruby (~> 1.0)
    jbuilder (2.10.1)
      activesupport (>= 5.0.0)
    jquery-datatables-rails (3.4.0)
      actionpack (>= 3.1)
      jquery-rails
      railties (>= 3.1)
      sass-rails
    jquery-rails (4.4.0)
      rails-dom-testing (>= 1, < 3)
      railties (>= 4.2.0)
      thor (>= 0.14, < 2.0)
    local_time (1.0.3)
      coffee-rails
    lograge (0.11.2)
      actionpack (>= 4)
      activesupport (>= 4)
      railties (>= 4)
      request_store (~> 1.0)
    loofah (2.7.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    marcel (0.3.3)
      mimemagic (~> 0.3.2)
    method_source (1.0.0)
    mimemagic (0.3.5)
    mini_mime (1.0.2)
    mini_portile2 (2.4.0)
    minitest (5.14.2)
    mocha (1.11.2)
    multi_json (1.15.0)
    mustermann (1.1.1)
      ruby2_keywords (~> 0.0.1)
    nio4r (2.5.4)
    nokogiri (1.10.10)
      mini_portile2 (~> 2.4.0)
    ood_appkit (1.1.5)
      addressable (~> 2.4)
      lograge (~> 0.3)
      ood_core (~> 0.1)
      rails (> 4.0.7, < 6.0)
      redcarpet (~> 3.2)
    ood_core (0.14.0)
      ffi (~> 1.9, >= 1.9.6)
      ood_support (~> 0.0.2)
    ood_support (0.0.3)
    pbs (2.2.1)
      ffi (~> 1.9, >= 1.9.6)
    public_suffix (4.0.6)
    rack (2.2.3)
    rack-protection (2.1.0)
      rack
    rack-test (1.1.0)
      rack (>= 1.0, < 3)
    rails (5.2.4.4)
      actioncable (= 5.2.4.4)
      actionmailer (= 5.2.4.4)
      actionpack (= 5.2.4.4)
      actionview (= 5.2.4.4)
      activejob (= 5.2.4.4)
      activemodel (= 5.2.4.4)
      activerecord (= 5.2.4.4)
      activestorage (= 5.2.4.4)
      activesupport (= 5.2.4.4)
      bundler (>= 1.3.0)
      railties (= 5.2.4.4)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.3.0)
      loofah (~> 2.3)
    rails_12factor (0.0.3)
      rails_serve_static_assets
      rails_stdout_logging
    rails_serve_static_assets (0.0.5)
    rails_stdout_logging (0.0.5)
    railties (5.2.4.4)
      actionpack (= 5.2.4.4)
      activesupport (= 5.2.4.4)
      method_source
      rake (>= 0.8.7)
      thor (>= 0.19.0, < 2.0)
    rake (13.0.1)
    rb-fsevent (0.10.4)
    rb-inotify (0.10.1)
      ffi (~> 1.0)
    rdoc (6.2.1)
    redcarpet (3.5.0)
    request_store (1.5.0)
      rack (>= 1.4)
    ruby2_keywords (0.0.2)
    sass (3.7.4)
      sass-listen (~> 4.0.0)
    sass-listen (4.0.0)
      rb-fsevent (~> 0.9, >= 0.9.4)
      rb-inotify (~> 0.9, >= 0.9.7)
    sass-rails (5.1.0)
      railties (>= 5.2.0)
      sass (~> 3.1)
      sprockets (>= 2.8, < 4.0)
      sprockets-rails (>= 2.0, < 4.0)
      tilt (>= 1.1, < 3)
    sassc (2.4.0)
      ffi (~> 1.9)
    sdoc (1.1.0)
      rdoc (>= 5.0)
    sinatra (2.1.0)
      mustermann (~> 1.0)
      rack (~> 2.2)
      rack-protection (= 2.1.0)
      tilt (~> 2.0)
    sinatra-contrib (2.1.0)
      multi_json
      mustermann (~> 1.0)
      rack-protection (= 2.1.0)
      sinatra (= 2.1.0)
      tilt (~> 2.0)
    sprockets (3.7.2)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.2.2)
      actionpack (>= 4.0)
      activesupport (>= 4.0)
      sprockets (>= 3.0.0)
    thor (0.19.1)
    thread_safe (0.3.6)
    tilt (2.0.10)
    timecop (0.9.1)
    turbolinks (5.2.1)
      turbolinks-source (~> 5.2)
    turbolinks-source (5.2.0)
    tzinfo (1.2.7)
      thread_safe (~> 0.1)
    uglifier (4.2.0)
      execjs (>= 0.3.0, < 3)
    websocket-driver (0.7.3)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)

PLATFORMS
  ruby

DEPENDENCIES
  addressable (~> 2.4)
  autoprefixer-rails (~> 8.4)
  bootstrap-sass (~> 3.4.1)
  bootstrap_form (~> 2.4)
  browser (~> 2.2)
  byebug
  climate_control (~> 0.2)
  coffee-rails (~> 4.2)
  data-confirm-modal (~> 1.2)
  dotenv-rails (~> 2.1)
  dotiw
  erubi
  font-awesome-sass (= 5.12.0)
  jbuilder (~> 2.0)
  jquery-datatables-rails (~> 3.4)
  jquery-rails
  local_time (~> 1.0.3)
  mocha (~> 1.1)
  ood_appkit (~> 1.1)
  ood_core (~> 0.11)
  ood_support (~> 0.0.2)
  pbs (~> 2.2.1)
  rails (= 5.2.4.4)
  rails_12factor
  redcarpet (~> 3.3)
  sass-rails (~> 5.0)
  sdoc
  sinatra
  sinatra-contrib
  thor (= 0.19.1)
  timecop (~> 0.9)
  turbolinks (~> 5.2.0)
  uglifier (>= 1.3.0)

BUNDLED WITH
   1.17.3

Kind regards and thanks!

Fixed by killing the nginx process for the affected user:

root     15970     1  0  2020 ?        00:00:00 nginx: master process (fakeuser.m) -c /var/lib/ondemand-nginx/config/puns/fakeuser.m.conf**
fakeuse+ 15971 15970  0  2020 ?        00:00:00 nginx: worker process

Logging in works fine after a new nginx process is started.

Kind regards.

Hi sorry for not replying sooner. This is likely due to an upgrade where the PUN was started before the upgrade occured and afterwards the gem it was depending on isn’t there anymore.

You took the right approach by just killing nginx, but we have this wrapper for convience (and to be sure the entire process tree gets killed. I believe our nginx stage runs an nginx command to SIGHUP or similar)

Here’s an example to get a list of all the PUNs that are running and to kill a specific one.

[root@69713d3b64d3 ~]# /opt/ood/nginx_stage/sbin/nginx_stage nginx_list
jeff
[root@69713d3b64d3 ~]# /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -u jeff